One phone video to a coaching scorecard
Eight stages, a parallel ball-tracking branch, three trained models, and a deterministic checker that stops the LLM from inventing anything. This page walks through what actually runs — architectures, numbers, and the code behind them.
The pipeline at a glance
A single 2D video becomes a 3D-grounded scorecard through eight processing stages, while a parallel branch tracks the ball itself and fits its measured flight.
Stage by stage
What each stage actually runs, and why it was chosen.
MediaPipe Lite finds 17 joints, every frame
Every frame is run through MediaPipe's Lite pose model, producing COCO-17 keypoints (nose, shoulders, elbows, wrists, hips, knees, ankles). Lite was picked over MediaPipe's own Heavy variant on measured event-timing accuracy, not intuition: on our benchmark, Lite scored PCE@5 0.090 vs Heavy's 0.052 — 73% better — while running 4× faster.
A Viterbi pass fixes left/right limb swaps before anything is lifted to 3D
Single-camera pose models occasionally flip left/right labels frame-to-frame — a wrist crossing the body's midline can get relabeled as the opposite wrist. Uncorrected, that flip corrupts every downstream rotation measurement (hip turn, X-factor) that depends on knowing which side is which. A per-joint Viterbi decode treats "which identity is correct this frame" as a hidden-state sequence and finds the lowest-cost path across the whole clip, instead of trusting each frame in isolation — killing swap chains, not just isolated flickers.
Scripts/smoothing.py →GolfPose's MixSTE2 transformer reconstructs the depth
A single camera can't see rotation about the vertical axis — the very thing that matters most for a golf swing (hip-shoulder separation, X-factor). The 2D keypoints are lifted into 3D by MixSTE2, a spatio-temporal transformer (embed dim 512, depth 8, 8 attention heads) from the GolfPose research paper, fine-tuned on golf-specific motion and pretrained with a 243-frame receptive field.
One-Euro filtering + a bone-length lock
Raw 3D output is noisy frame-to-frame. A three-pass cleanup fixes it: short low-confidence gaps are linearly interpolated, an adaptive One-Euro filter smooths slow phases heavily while staying snappy through impact, and a bone-length lock slides each joint along its parent→child direction so bone length matches the clip-wide median (real bones don't change length). Measured impact on a reference clip: mean acceleration dropped 0.0041 → 0.0020 (−51%).
Scripts/smoothing.py →A trained detector finds the 8 swing events
The 8 GolfDB swing events — address, toe_up, mid_backswing, top, mid_downswing,
impact, mid_follow_through, finish — are located by EventTCN, a
0.26M-parameter 1D convolutional network trained directly on 3D landmark trajectories.
Its 9-channel output (8 events + background) is decoded with a monotonic constraint so
event order can never invert. This was the single biggest accuracy win in the project.
PCE@5 = % of the 8 events recovered within ±5 frames of the labeled ground truth. The trained detector beats the published SwingNet baseline because it consumes clean 3D landmarks instead of raw 160px video frames.
Scripts/event_detector.py →15 biomechanical measures, scored against tour pros
At each detected event, 15 core indicators are computed directly from the 3D landmarks — the rotation metrics in particular are only possible because of the 3D lift; 2D can't see rotation about the vertical axis at all. Every indicator is reported as a value plus a pro band (the 25th–75th percentile of the GolfDB tour-pro distribution) and a confidence tier, never as a single "correct" target. A 16th indicator, hand speed through impact, is added when a real-time frame rate is known (phone uploads) — it's omitted for slow-motion GolfDB benchmark clips, where frame-based speed wouldn't mean anything.
Rotation
- Shoulder turn at topdegrees off address
- Hip turn at top / at impactdegrees off address
- X-factor at topshoulder turn − hip turn
Posture & head
- Spine tilt at address / impact+ posture loss between them
- Head sway (max)% of torso length
- Head lift (max)% of torso length
Arms, legs & tempo
- Lead / trail arm bend at topdegrees
- Lead knee flex, address / impactdegrees
- Hip lateral shiftweight-shift proxy, % of torso
- Tempo ratiobackswing : downswing frames
- Hand speed at impactbody-scales/sec — real-time uploads only
A subpixel tracker follows the ball; physics fills in the rest
Monocular ball tracking has no size cue — a golf ball is a 3–6 px dot with no reliable depth signal. Rather than a learned detector, a classical stabilized frame-diff + physics-gated linker follows the ball frame to frame: candidate tracks must originate near the impact-frame club/wrist region (from the pose stage), rise at a plausible launch angle, stay geometrically straight, and never back-extrapolate into a neighboring bay's ball already in flight. Each discriminator was added to kill one specific observed failure mode (flags, birds, other players' shots, club-arc ghosting).
The measured 2D track is then fit to a physics forward-model (projectile motion + drag/Magnus lift) with a hand-rolled Nelder-Mead optimizer — no scipy in the Lambda runtime — solving for ball speed, launch angle, and azimuth that best reproduce the observed pixel trajectory, anchored in scale by the golfer's pose-derived body height. Speed is only reported when enough subpixel points are recovered to pin the scale ambiguity that makes monocular ball speed otherwise unidentifiable.
Validated on two real swing videos: a 20-point track fit speed 150–160 mph, launch 24.0–26.5°, carry 237–257 yd (measured tier); an 8-point track landed in the partial tier, reporting direction confidently and carry as an envelope-based estimate. The chat coach cites whichever tier applies and phrases confidence accordingly — a measured carry is stated plainly, a simulated one is always labeled as an estimate.
Scripts/ball_track.py →An LLM writes the read — a verifier checks every claim
Claude turns the scorecard into a beginner-friendly explanation under a gated feature
config (f_strict_grounding, composite score 1.000 on a
held-out 200-example set). The model also emits structured claims
(indicator_keys + claim type), which a separate, deterministic
grounding verifier — plain code, not another LLM call — mechanically
checks against the scorecard before anything reaches the golfer.
Claim references an indicator that isn't in the scorecard at all.
Claim uses a measurement flagged low-confidence — those are off-limits.
Claim says "in range" but the value falls outside the tour-pro band.
Claim flags an issue for a value that's actually within the pro band.
The design is failure-tolerant on purpose: if the LLM call fails for any reason, the explanation step exits cleanly and leaves the scorecard intact rather than ever showing a fabricated read.
Scripts/coaching_explain.py →How it's deployed
A real upload pipeline on AWS, not just a cached demo: a golfer's own video goes in one end, a 3D replay and coaching chat come out the other.
Uploads go straight to S3 via a presigned POST — the browser never
touches compute directly. An EventBridge → SQS → Lambda chain then runs the same
pose + ball-track pipeline as the local demo inside a CPU-only container image (torch CPU,
~1–1.5 GB, cold-starts in 10–25s, warm ~150ms), writing queryable rows via the RDS Data API.
The coaching chat runs on Amazon Bedrock, not a direct model API call. Spend
stays bounded: the curated demo clips cache their explanation in S3 after the first call so
repeat views cost $0 in tokens, and uploads are capped by file size and reserved concurrency.
No GPU, no always-on server.
Go deeper
The full evaluation methodology, leaderboard, and source are in the repo.