Research 01 / 03
Funded research / August 2026–Present
Robust motion retrieval
I’m conducting funded undergraduate research at Boston University with Prof. Brian Kulis, testing whether contextual training helps recognize an action when its tracked joints jitter or disappear.
- Encoder
- Frozen MotionBERT
- Study
- 3 recipes × 3 seeds
- Status
- Active; final comparison pending
System view / research question
Pose → embedding → retrieval01 / Query corruption
First, make the pose observation fail.
Change one kind of damage at a time and compare it with the clean reference. These synthetic examples illustrate the stress tests; they do not model real camera occlusion.
Interactive 01
Pose corruption lab
Planned protocol / simulation
Explanatory simulation of the proposed corruptions—not a model output or measured robustness result. Use the frame controls for a static comparison.
02 / Origin
A small keypoint error did not stay small.
The rowing pipeline exposed the failure chain. Motion blur, occlusion, oblique views, and tracking discontinuities displaced joints; monocular lifting added depth uncertainty; velocity and acceleration features amplified what remained.
That practical problem became a narrower research question: if the pose observation is unreliable, can the ordering of a motion embedding space remain useful?
Rowing case study03 / Training intuition
Learning from nearby motions
Contextual training compares each pair of motions and their shared neighbors. Explore cosine and contextual scores within a synthetic training batch below. At evaluation, every trained head uses cosine retrieval against a clean gallery.
The experiment compares complete training recipes. Their margins and regularization differ too, so an improvement could not be attributed to neighborhoods alone.
Interactive 02
Neighborhood explorer
Synthetic intuition demo
The points are deterministic teaching data. They show how a perturbed pairwise score and a neighborhood-derived score can order the same batch differently; they do not predict an experimental outcome.
04 / Foundation → extension
Same objective. Different failure.
Frozen MotionBERT supplies motion features for a learned linear head. Contextual training constructs neighborhoods within a balanced batch of normalized embeddings. It uses clean training poses.
What is established
The 2023 paper reports improved image retrieval, less overfitting, and greater robustness to label noise.What remains untested
Whether the full contextual training recipe reduces retrieval degradation when pose coordinates become unreliable.| Dimension | ICML 2023 foundation | Proposed extension |
|---|---|---|
| Domain | Still images | Temporal skeleton sequences |
| Observation | Image embedding | T × J × C joint coordinates → motion embedding |
| Stressor | Inconsistent or noisy labels | Query-only jitter, missing joints, and missing frames |
| Task | Supervised image retrieval | One-shot retrieval on 20 unseen action classes |
| Evidence | Reported by Liao et al., ICML 2023 | Open hypothesis; no robustness result yet |
- 01
Sequence
x ∈ ℝᵀˣᴶˣᶜA tracked body across T frames, J joints, and C coordinates. - 02
Embedding
z = h(f(x)) / ‖h(f(x))‖₂Frozen MotionBERT supplies 8,704 features; a learned linear head yields 2,048 normalized values. - 03
Geometry
S = ZZᵀ · D = 2 − 2SCosine similarity and unit-sphere squared distance describe every pair. - 04
Neighborhood
Nᵢⱼ = 𝟙[Dᵢⱼ ≤ dᵢ,ₖ + ε]Expanded top-k neighborhoods turn pairwise distances into local structure. - 05
Context
W₁ → R → W₂ → WShared agreement, reciprocal neighbors, query expansion, then symmetry. - 06
Objective
λLcontext + (1−λ)Lcontrast + γLregThe hybrid loss combines contextual fit, pairwise margins, and regularization.
Balanced batch. The neighborhood size k equals samples per represented class and must be even for the reciprocal k/2 step.
Discrete forward, useful backward. The comparison is binary in the forward pass; a straight-through estimator supplies a constant heuristic gradient during training.
Bounded cost. Context construction is cubic in batch size. The study uses batches of eight actions × four examples and trains only the linear head.
Interactive 03
Contextual matrix microscope
3 classes × 4 samples / k = 4
This fixed 12-sequence batch is intentionally small. The browser mirrors the construction order for explanation only; training-scale contextual computation is cubic in batch size.
05 / Current experiment
A useful result either way.
Train contrastive, contextual, and supervised contrastive (SupCon) recipes on clean poses across three paired seeds. Select settings on clean development data, then evaluate all nine heads on unseen actions with identical query damage and a clean gallery.
Progress as of 22 October 2026: initial engineering pilots are complete. Runtime revalidation is in progress, and capacity approval remains pending before model selection and final training. The final robustness comparison remains pending. Follow the public research tracker for dated progress.
- Dataset
- NTU RGB+D 120: 80 training actions, 20 disjoint development actions, and 20 unseen test actions. Final heads train on all 100 auxiliary actions.
- Encoder
- Frozen pretrained MotionBERT, with only a biased linear 8,704-to-2,048 head trained. Outputs are L2-normalized for cosine retrieval.
- Retrieval
- One clean exemplar per unseen action. Primary queries exclude the exemplar and every synchronized view of that performance; an exact-official secondary query set is also reported.
- Corruptions
- Three separate, nested stress tests: jitter at 1%, 2.5%, and 5% of torso scale; 3, 6, or 8 missing joint trajectories; and 10, 25, or 40 masked frames out of 100.
- Matched recipes
- Contrastive, full contextual, and descriptive SupCon; paired seeds 7, 17, and 29; physical batches of 8 actions × 4 examples; exactly 20 epochs per run. Three pilots and 27 clean-development selection runs precede nine fresh final heads.
- Evaluation
- Lock all nine heads before novel opening. Nine heads × ten query conditions × two query definitions yield 180 cells, with top-1, mean reciprocal rank, top-5, and clean-to-corrupted top-1 drops.
Three go / no-go milestones
- 01
Protocol validation
Validate the frozen encoder, exact query/gallery identities, corruption inputs, and numerical runtime. Confirm capacity before beginning clean-development selection.
- 02
Matched training recipes
Compare contrastive, full contextual, and SupCon recipes with paired seeds, initialization, and physical batches. Train and lock all nine final heads before opening the unseen test actions.
- 03
Robustness claim
Evaluate clean and corrupted queries against the fixed clean gallery. Report every seed and condition, absolute accuracy, and clean-to-corrupted drops, including null or negative findings.
| Observed pattern | Scientific interpretation |
|---|---|
| Contextual-minus-contrastive degradation interval entirely below zero | Supports a smaller clean-to-corrupted top-1 accuracy drop for the full contextual recipe, conditional on the fixed seeds, actions, and gallery exemplars. Different margins and regularization prevent attribution to neighborhoods alone; absolute corrupted accuracy must also be reported. |
| No resolved contextual advantage | A useful null result. Report the uncertainty, every seed and condition, and absolute accuracy without changing the study after opening. |
| Contextual degrades more | A valid contrary result that limits the hypothesis. The comparison remains between complete recipes; SupCon is descriptive. |
06 / Papers, code, artifacts
The evidence trail.
Primary papers and canonical repositories, grouped by the role they play in the study.
Current work
- A1
Experimental plan (v3)
Training recipes, evaluation rules, and the September-December 2026 schedule.
- A2
Original July 2026 prospectus
The eight-page literature review and original proposal. The current v3 plan narrows its experimental scope.
- A3
Pose embedding repository
Code, protocols, and public study files.
- A4
Rowing biomechanics origin
The tracking errors that prompted this study.
Foundation
- F1
Contextual similarity optimization
Liao, Tsiligkaridis, and Kulis, ICML 2023. The image-retrieval recipe and label-noise evidence this study adapts to poses.
- F2
MotionBERT
The pretrained motion encoder and one-shot NTU retrieval pipeline used in this study.
- F3
NTU RGB+D 120
The skeleton action benchmark used here with its official one-shot novel actions and exemplars.
Related robustness research
- R1
MaskCLR
Related corruption-aware representation learning that informed the original proposal. A MaskCLR arm and pose-estimator shifts are outside the current December study.
- R2
PoseConv3D
Independent evidence that estimated-pose representations can be tested for noise robustness and cross-dataset transfer.
- R3
Skeleton-DML
Direct precedent for deep metric learning and nearest-neighbor recognition on skeleton sequences.
Research question