Eight bugs that looked like the algorithm
Video2Any finds the slides in a recording by decoding frames in the browser, comparing each one against the last, and keeping the ones that changed enough. When it gets the answer wrong, the obvious suspect is the comparison. It usually isn’t.
Here are eight things we chased over one week. Two were in the detector. The rest were a stale guard, a lying test harness, a virtualised list, a race with an API call, a memory ceiling, and an error message that threw away the only useful part of itself.
The eight
1. The control that did nothing
Someone reported that a short video gave three slides on Dense, same as on Default. We spent a while reading threshold maths before checking the simplest thing: were the settings even being applied?
They were not. The first sweep is guarded by a ref so it runs once per file. Changing the density rebuilt the callback, but the guard sent the effect straight back round, and the only thing that ever re-ran was a button in an unrelated notice. The slides on screen after switching were the previous ones, at identical timestamps. Sparse, Default and Dense now give 3, 6 and 13 on that clip. Before, all three gave 6.
2. The test harness was wrong twice
To explain a bad result we rebuilt the pipeline offline in Node, fed it the real file, and got numbers that disagreed with the app in both directions. First the harness said one slide where the app produced six, then fifteen where the app produced six.
Two causes. It was sampling frames with ffmpeg at greyscale, while the browser hands the engine RGBA from a canvas, and the two downscalers disagree enough to move the calibrated threshold from 0.618 to 0.650. And it skipped the function that decides whether to use an activity mask, so it was not running the path the app runs. A harness that reproduces the algorithm but not the pipeline will happily give you a confident wrong answer.
3. The DOM only had what was on screen
A test asserted that a 32-minute video produced slides past the 30-minute mark. It read every thumbnail’s timestamp and found the last one at 27:00, which looked exactly like a truncated analysis.
The grid is virtualised. Only the visible rows exist as elements. Scrolling to the bottom first, the real last slide was at 32:00. The assertion had been measuring the viewport.
4. Every worker deduplicated alone
Long videos are swept in parallel chunks. Each worker drops repeated shots inside its own slice — a presenter returning to an earlier slide, a call cutting back to the same view — and none of them can see the others.
So a shot that recurs across a boundary shipped once per chunk it appeared in. On a 29-minute meeting that was three duplicate slides out of nine. It also meant the deck depended on how many chunks there were, which is to say on how many cores the viewer’s machine had. The check moved to the main thread, running as each slide arrives, so a duplicate is never put on screen and then taken away.
5. The better design made it worse
Each chunk calibrating its own threshold on its own slice is obviously wrong: one video, one distribution, one threshold. So we built that. Workers post the ratios they measured, the main thread pools them, calibrates once, and sends the answer back.
It narrowed the gap between four chunks and eight from three slides to one. It also made the deck worse. On that 29-minute meeting, where ffmpeg’s own scene detection counts seven real cuts, per-chunk calibration plus the deduplication above gives six and global calibration gives four. Pooling widens the distribution, which raises median plus two MADs, which drops real slides. We reverted it. A design can be more principled and still lose to the one you have, and the only way to know is to have something outside your own code to check against.
6. False because we hadn’t asked yet
A paying subscriber uploaded a long recording and got the first thirty minutes of it. Pressing Resample produced the whole thing, which is what made it look like a detection problem.
The editor starts before it knows who you are. The entitlement flag begins false — not because the answer was no, but because the request was still in flight — and the length cap read that false and applied the free tier’s ceiling. By the time Resample ran, the plan had arrived. The fix was to wait for it, and then to stop needing to: the worker already has the session when it serves the page, so it writes the plan into the document and the first render knows.
7. Every error was called “Error”
Transcription was failing more often than it succeeded — six failures to five successes — and every one of them was logged as the string “Error”. So were nineteen failed extractions.
The code recorded err.name, which is “Error” for every error anyone constructs by hand. The message, where the information always was, went nowhere. Once we recorded it, the transcription failures had a mechanism: decodeAudioData resamples to the audio context’s rate, so decoding a 55-minute lecture at the hardware’s 48 kHz stereo materialises about 630 MB of float samples before anything else happens, and two hours is 1.4 GB. Decoding on a context that is already at 16 kHz gives the model the same audio for about a sixth of the memory.
8. It was fast, and it still felt broken
A 92-minute meeting recording sweeps in 52 seconds. The first slide appears at 50 of them.
Nothing was slow. The threshold is calibrated from the whole distribution of frame differences, so no slide can be chosen until the video has been read end to end, and for those 50 seconds the only thing on screen was a percentage. A percentage that moves with nothing beside it reads as a stall. It now says which half of the work is running and how much of the video has been read, and the same 52 seconds stopped being a complaint.
What we’d tell ourselves a week ago
Check that the input reached the code before reasoning about the code. Half of these were settings that never arrived, answers that had not come back yet, or a test looking at the wrong thing.
And keep something outside your own pipeline to check against. ffmpeg’s scene detection is not what we are building and does not have to be — it just has to be independent. Without it we would have shipped the calibration rewrite, because it was the better design, and quietly lost three real slides out of seven.