scored 94.3% accuracy. I went back over the terminal output and the confusion matrix more than once. They agreed every time. The number went into my README and onto my CV.
It was also wrong. Not by a rounding error, either. The honest figure was 69%.
No one caught this for me. It only came to light because I decided to build tools that could verify my own results. Fixing it taught me more about production machine learning than building the original system did.
Plenty of articles explain data leakage in the abstract. This one is about what it looks like from the inside, with the numbers from before and after. If you evaluate models on sequential data of any kind (video, audio, wearable sensors, time series), there’s a decent chance the same bug is sitting in your pipeline right now.
Contents
The system
The project is a real-time fall detector for assistive monitoring. A webcam runs through MediaPipe Pose to get skeletal landmarks. From the landmarks I compute a few simple biomechanical features: torso angle, normalised hip and shoulder heights, and the geometry of the bounding box. A Random Forest classifies every frame, and a smoothing window over the last few predictions decides whether to raise the alarm. It all runs on CPU in real time. No video ever leaves the device. The only thing that leaves is the skeleton maths.
I trained it on the Le2i fall detection dataset (Charfi et al., 2013), which contains staged falls and daily activities recorded in home-like rooms. I extracted features per frame, labelled per frame, ran a standard train_test_split, and got 94.3% accuracy. As far as I was concerned, the project was done.
The leak
Here’s the line that lied to me:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)The problem is that this split is random over individual frames, and frames from the same video are not independent. Two frames captured 80 milliseconds apart look almost the same: same person, same pose, same lighting, almost the same feature vector. When you shuffle all the frames together and hold out 20% for testing, nearly every test frame has a close twin in the training set.
So the test doesn’t measure whether the model generalises to people and rooms it hasn’t seen. It measures whether the model can remember, and a Random Forest can remember almost everything. This is data leakage in its most ordinary form: the samples aren’t independent, so the random split hands the test set the answers in advance.
The fix is simple to state: split by the thing that makes your samples dependent. For video, that means splitting by video. I re-ran the same features and the same model with GroupKFold, grouping by video, so no recording could appear on both sides of a fold.
The grouped score came back at 69.1%. One fold, the same code split honestly, landed at 31%.
What stung wasn’t the 25-point drop. It was that 94.3% had felt completely earned: a good number sitting next to a reasonable-looking pipeline, and nobody thinks to look twice. I didn’t.
The labels
Once I had audited the split, I started auditing everything else. The labels turned out to be worse.
My training labels came from folder names. If a video sat in the falls/ folder, every frame of it was labelled fall. But a staged fall only lasts one to two seconds, and the rest of the clip is ordinary activity: walking around, sitting down, standing up. The dataset actually provides frame-level annotations that mark the exact start and end of every fall, so I cross-referenced my labels against them. The result was embarrassing.
In 76% of the frames carrying a fall label, the person was just walking around normally. Only 6% captured an actual fall, and the remaining 18% showed someone already lying on the floor afterwards. One 78-second video contributed 44% of the entire positive class, and 1.5 seconds of it was a fall.
My labels had changed the task without me realising. The model solved the easier problem they described, not the one I meant to set. That’s exactly the kind of failure leakage hides. A leaky evaluation can’t tell you that your labels are wrong, because it only measures memorisation, and a model can memorise wrong labels just as easily as right ones. If your labels come from folder names, file paths or any similar shortcut, they deserve the same suspicion as your split.
So I rebuilt the training set, starting from the ground truth this time.
A frame now counts as positive if it sits inside the annotated fall window or in the two seconds of lying that follow it. Everything before the fall is negative. The ambiguous frames late after a fall, where the person might still be lying down or might be getting up and there is no ground truth either way, I excluded completely instead of guessing.
I also used the whole dataset this time: 186 of its 190 videos, instead of my original 28. That meant doing work the first pipeline had skipped. Two of the scenes come with no annotation files at all, so I watched all 59 of those videos frame by frame and hand-annotated 29 fall windows myself. Around my estimated boundaries I excluded a safety buffer, so that my own imprecision could not contaminate the negative class. One annotation file was malformed, with its fall window buried in the middle of the file between bounding-box rows. The parser now handles that case, and the fix is documented in the repository.
A fall is movement, so the features needed movement in them too. I added velocity features such as hip descent rate and torso angular rate, computed over seconds rather than frames. That way the same model works on the dataset’s mixed frame rates and on a webcam with a variable frame rate, with no special cases.
Measure what the user experiences
After the rebuild, the honestly evaluated frame-level numbers recovered, with grouped out-of-fold accuracy back above 0.97. By then I no longer trusted frame accuracy as a headline, and not only because correct labels leave the classes lopsided enough to flatter it. A caregiver never sees a frame. They either get an alert or they don’t. For an alarm system, the questions that matter are simpler: when someone falls, does the alarm fire? How quickly? And how often does it fire when nobody falls?
So the evaluation now simulates the exact production alarm logic, a voting window over per-frame probabilities, on held-out videos, and reports event-level results:
- 121 of 126 falls detected (96.0%)
- 0.58 s median alarm latency from the start of the fall
- a false alarm on 3 to 7 of the 60 no-fall videos, depending on the model
Every miss came down to the same thing: the pose estimator lost the person mid-fall, whether from occlusion behind furniture, motion blur or footage too low-res to track. If the pose estimator can’t see the fall, the classifier never gets its turn. You only find that out by evaluating whole events, and it’s what’s pushing my next round of work toward perception rather than more classifier tuning.
The trade-off a single metric can’t express
Honest evaluation also surfaced a decision I had never noticed I was making. My most sensitive model treats lying on the floor after a fall as positive. That maximises detections, but it also means the alarm goes off when someone lies down on purpose. So I retrained on the fall motion alone. That model stays silent on a held-out video of a deliberate lie-down, with a peak probability of 0.17, but it detects fewer falls, 89.7%, because the pose estimator has to actually see the fall happen.
| Model | Falls detected | False-alarm videos | Deliberate lie-down |
| “person down” labels | 96.0% | 7/60 | alarms |
| “fall motion” labels | 89.7% | 3/60 | silent |
Neither model is wrong. In an elderly-care setting, you probably do want an alarm whenever someone is on the floor. If the goal is to show the system can tell a fall from a lie-down, the motion model wins. The behaviour is controlled by the label policy, not the architecture, and a single accuracy score would have hidden the whole trade-off. I ship both models and document the difference.
The system around it
I wanted the model to be something strangers could poke at, so I built the rest of the system around it. Detection runs on the edge device. When a fall is detected, the device sends roughly 100 bytes of JSON, and nothing else, to a FastAPI and PostgreSQL backend on AWS through an authenticated write API. The video itself never leaves the device, so privacy is built into the architecture instead of promised in a policy. On top of the backend there is a live monitoring dashboard, an acknowledgement workflow for caregivers, and a React Native mobile app that pushes the alert to a caregiver’s phone within seconds of the fall. All three repositories run their tests in CI on every push.
Then I fell over. Deliberately, onto the carpet, in front of the webcam. Every fall was planned and done carefully, and it still took three takes to get one worth keeping.
Each take ran from start to finish with the whole pipeline live. The one that survived is fifty-five seconds, unedited: the edge model detects the fall, the event crosses the internet, the cloud dashboard turns red on its own five seconds later, and the alert is acknowledged back to all clear. No cuts anywhere. If the video needed editing, it would prove nothing.
I only recorded that video because I finally trusted the system behind it. I would never have dared to film the 94.3% version.
The checklist
If you take one thing from my 25 wasted percentage points, take the first item below. The rest follow from it.
- Group your splits by whatever makes samples dependent: video, patient, subject, session, device. If switching to a random split makes your score jump, it’s the higher number that’s lying.
- Compare your labels with the real annotations instead of trusting how the files are organised. Then look inside your positive class and see what it really contains.
- Measure whole events, the way a user would: how many incidents it caught, how quickly it raised the alert, and how often it went off when nothing happened. Per-sample accuracy on its own hides too much.
- Keep a held-out behavioural test for the failure mode that scares you most. Mine is a deliberate lie-down that must not raise an alarm.
- Publish the correction. A corrected system people can actually believe is worth more than an impressive one they can’t.
Everything above is public: the detection code and evaluation harness, the platform, the mobile app, and the 55-second live demo.
References
Dataset. The Le2i (IMViA) Fall Detection Dataset, Université de Bourgogne (Charfi et al., 2013), is distributed openly with a request to cite the authors’ paper and carries no explicit licence. This article reproduces no images or video from the dataset. It reports only aggregate metrics computed from it, with attribution.
I. Charfi, J. Mitéran, J. Dubois, M. Atri, R. Tourki, “Optimised spatio-temporal descriptors for real-time fall detection: comparison of SVM and Adaboost based classification,” Journal of Electronic Imaging (JEI), Vol. 22, Issue 4, pp.17, October 2013.