Research proposal / July 2026
Can motion embeddings survive bad poses?
A retrieval-first study of whether contextual metric learning can keep similar motion sequences near each other when estimated joints jitter, disappear, or arrive in broken frames.
- Foundation
- ICML 2023
- Transfer
- Images → motion
- Status
- Open hypothesis
System view / research question
Pose → context → retrieval01 / Planned corruptions
First, make the pose observation fail.
The detailed lab separates joint jitter, missing limbs, and frame loss so the planned stress test stays interpretable. The clean sequence remains visible as a fixed reference.
Interactive 01
Pose corruption lab
Planned protocol / simulation
Explanatory simulation of the proposed corruptions—not a model output or measured robustness result. Use the frame controls for a static comparison.
02 / Origin
A small keypoint error did not stay small.
The rowing pipeline exposed the failure chain. Motion blur, occlusion, oblique views, and tracking discontinuities displaced joints; monocular lifting added depth uncertainty; velocity and acceleration features amplified what remained.
That practical problem became a narrower research question: if the pose observation is unreliable, can the ordering of a motion embedding space remain useful?
Rowing case study03 / Retrieval intuition
A pair is fragile. A neighborhood adds context.
Cosine retrieval asks whether one query vector is close to one gallery vector. Contextual similarity also asks whether those examples agree about who else belongs nearby.
Shared and reciprocal neighbors create a local-consensus signal. The proposal tests whether that inductive bias degrades more slowly than direct pairwise objectives under pose corruption.
Interactive 02
Neighborhood explorer
Synthetic intuition demo
The points are deterministic teaching data. They show how a perturbed pairwise score and a neighborhood-derived score can order the same batch differently; they do not predict an experimental outcome.
04 / Foundation → extension
Same objective. Different failure.
The transfer changes the data and the corruption, not the central loss. A temporal encoder replaces the image network; the contextual construction still operates on a balanced batch of normalized embeddings.
What is established
The 2023 paper reports improved image retrieval, less overfitting, and greater robustness to label noise.What remains untested
Whether the same loss preserves pose-sequence retrieval when input coordinates—not labels—become unreliable.| Dimension | ICML 2023 foundation | Proposed extension |
|---|---|---|
| Domain | Still images | Temporal skeleton sequences |
| Observation | Image embedding | T × J × C joint coordinates → motion embedding |
| Stressor | Inconsistent or noisy labels | Jitter, occlusion, frame loss, pose-source shift |
| Task | Supervised image retrieval | Class-level pose-sequence retrieval |
| Evidence | Reported by Liao et al., ICML 2023 | Open hypothesis; no robustness result yet |
- 01
Sequence
x ∈ ℝᵀˣᴶˣᶜA tracked body across T frames, J joints, and C coordinates. - 02
Embedding
z = fθ(x) / ‖fθ(x)‖₂MotionBERT supplies the temporal encoder; the output is normalized. - 03
Geometry
S = ZZᵀ · D = 2 − 2SCosine similarity and unit-sphere squared distance describe every pair. - 04
Neighborhood
Nᵢⱼ = 𝟙[Dᵢⱼ ≤ dᵢ,ₖ + ε]Expanded top-k neighborhoods turn pairwise distances into local structure. - 05
Context
W₁ → R → W₂ → WShared agreement, reciprocal neighbors, query expansion, then symmetry. - 06
Objective
λLcontext + (1−λ)Lcontrast + γLregThe hybrid loss combines contextual fit, pairwise margins, and regularization.
Balanced batch. The neighborhood size k equals samples per represented class and must be even for the reciprocal k/2 step.
Discrete forward, useful backward. The comparison is binary in the forward pass; a straight-through estimator supplies a constant heuristic gradient during training.
Bounded cost. Context construction is cubic in batch size, so the experiment profiles moderate balanced batches before any full-encoder run.
Interactive 03
Contextual matrix microscope
3 classes × 4 samples / k = 4
This fixed 12-sequence batch is intentionally small. The browser mirrors the construction order for explanation only; training-scale contextual computation is cubic in batch size.
05 / Proposed experiment
A useful result either way.
The study isolates the loss before expanding scope. There are no completed robustness results yet; the deliverable is a controlled comparison whose null outcomes remain interpretable.
- Dataset
- NTU RGB+D 120, using its official cross-subject and cross-setup protocols.
- Encoder
- Pretrained MotionBERT with a normalized projection output; freeze first, fine-tune later.
- Retrieval
- A saved, disjoint query/gallery manifest with noisy queries against a fixed clean gallery.
- Corruptions
- Body-scaled Gaussian joint jitter, joint or limb masking, frame dropout, temporal jitter, and pose-source shift when available.
- Controls
- The same split, backbone, sampler, embedding dimension, augmentations, tuning budget, evaluator, and seeds for every loss.
- Metrics
- Recall@1/5/10, mAP, mAP@R or R-precision, retention, and clean-to-corrupted degradation.
Three go / no-go milestones
- 01
Protocol validation
Reproduce the starting pipeline, save the held-out query/gallery split, and require monotone corruption curves from frozen embeddings.
- 02
Loss isolation
Train matched projection heads with SupCon, contextual, and hybrid objectives; verify that batch composition,
k, and samples per class agree. - 03
Robustness claim
Fine-tune only after milestone two is stable, then add the corruption-aware MaskCLR control and the full matched ablation.
| Observed pattern | Scientific interpretation |
|---|---|
| Robust gain with similar clean retrieval | Evidence that contextual neighborhoods help under input corruption. |
| Clean gain, no robustness gain | Better ranking, but not support for the pose-noise hypothesis. |
| Gain only with corruption augmentation | Neighborhood learning and robust data exposure are complementary. |
| No gain over SupCon or MaskCLR | The encoder or corruption-aware training matters more than this objective. |
06 / Papers, code, artifacts
The evidence trail.
Primary papers and canonical repositories, grouped by the role they play in the proposal.
Current work
- A1
Robust pose-sequence retrieval prospectus
The eight-page July 2026 literature review, feasibility assessment, and proposed experiment.
- A2
Pose embedding repository
The implementation workspace and study artifacts for the proposed extension.
- A3
Rowing biomechanics origin
The case study where noisy tracking, uncertain 3D lift, and derivative amplification made the robustness question concrete.
Foundation
- F1
Contextual similarity optimization
Liao, Tsiligkaridis, and Kulis, ICML 2023. The image-retrieval objective and label-noise evidence this proposal transfers.
- F2
MotionBERT
The pretrained temporal motion encoder and public one-shot NTU pipeline chosen as the lowest-risk starting point.
- F3
NTU RGB+D 120
A large-scale skeleton action benchmark with standard cross-subject and cross-setup evaluation protocols.
Robustness controls
- R1
MaskCLR
The closest robustness-specific comparator and the source for joint noise, occlusion, frame masking, and pose-estimator-shift tests.
- R2
PoseConv3D
Independent evidence that estimated-pose representations can be tested for noise robustness and cross-dataset transfer.
- R3
Skeleton-DML
Direct precedent for deep metric learning and nearest-neighbor recognition on skeleton sequences.
Research question