0.2.0: what a webcam bubble does to slide detection

Version 0.2.0 of video-slide-extractor is on npm today, and it is a good one to ship: the failure it fixes is one we had already published ourselves, as a row in our own benchmark. It is the block-diff detector that decides, frame by frame, whether the picture in front of it is a new slide — the open core of what Video2Any runs in your browser. Zero dependencies, MIT, browser or Node.

What shipped in July was a snapshot of that core taken on 16 July: correct on a clean screen recording, and wrong on a good share of the recordings people actually have. Two things break it in the wild, the product learned both over the following month, and until today the package did not know either. That is the whole release.

Here is what the videos looked like, what the benchmark said before we changed anything, and what the numbers are now.

What the videos actually looked like

Video2Any has recorded extraction events since 16 July. Between then and today that is 411 extraction runs from 96 people, 354 of which completed, producing 291 exports — 188 PowerPoint files, 95 PDFs, 8 image archives. Almost all of it is a local file: 398 of the 411 runs were an upload from the machine the tab was running on. (Two days of our own automated testing are excluded from every number here.)

Of the 106 runs where the file told us how long it was, the average video was 53 minutes. These are not five-minute product demos. They are lectures, meetings and course recordings, and they are the videos the July detector had never been measured against.

The deck sizes are where it gets interesting. The average completed run produced 64 slides, which sounds reasonable and hides everything: 64 runs produced more than 100 slides, seven produced more than 300, and four came back with 900 — which happens to be exactly the number of samples the sweep is allowed to take. Every single sample was kept. Nothing was rejected. One of those four was a 30-minute video.

The output rate is not a constant either: at the same setting, runs under an hour averaged about 6.5 slides per minute and runs over an hour about 1.0. Some of that spread is real — a two-hour meeting genuinely has fewer slide changes per minute than a short demo — and from a count alone you cannot tell which side is right. The four ceiling runs you can. A detector that keeps everything is not detecting.

The benchmark had already said so

Before any of this was fixed, 0.2.0 had already added bench/ to the package: fixture videos rendered from a known slide sequence, so every transition timestamp is ground truth by construction rather than by hand-labeling. Each deck is rendered three ways — clean, crushed to crf 45 for compression noise, and with an animated webcam bubble in the corner.

On the MIT deck (46 slides, sampled every 2 seconds at 160×90) the clean variant scored an F1 of 0.966. The same deck with the webcam overlay scored 0.538: 125 captures for 46 slides, precision 0.368, and 63% of everything it returned was a duplicate of a slide it had already captured.

The reason is arithmetic, not bad luck. The default trigger is "2% of blocks changed". A webcam bubble, a cursor, a spinning logo — anything that never holds still — occupies more than 2% of the blocks on its own, so every sampled frame clears the threshold without the slide changing at all. The number of slides you get stops depending on the deck and starts depending on whether the presenter had their camera on.

We published that as the package’s honest failure row, and it is the shape of those four 900-slide runs. Fixing it is what 0.2.0 is.

What 0.2.0 adds

Three new exported functions, one optional field, and a type declaration for all of it.

• buildActivityMask

  Takes the sampled frames and finds the blocks that change in nearly every frame pair — the bubble, the cursor, the clock in the corner — and returns a mask that takes them out of scoring entirely. It returns null rather than a mask when the mask would cover most of the frame, and that case matters: where most of the picture is moving, the moving part is the subject, and masking it would mean detecting nothing at all.

• chooseThreshold

  One fixed threshold cannot fit both a deck and a talking head. A deck is still, so its real changes stand clear of the noise; camera footage moves constantly, where 0.02 makes every frame a new slide. This reads the distribution of frame-to-frame change and returns { changedRatio, mode } — and the mode is deliberate. bimodal, static, motion or default is the detector telling you which kind of footage it decided it was looking at, so when the answer is wrong you can see why rather than guess.

• analyzeSamples

  Runs both over the same frames and hands back { mask, choice }. This is what most callers actually want: you have samples, you want to know what to detect with, and you should not have to know that "what to detect with" is two separate decisions.

• frameDiff({ collect: true })

  Returns flags, one byte per block, marking where the frame changed — useful for drawing what the detector saw. Only under collect, because a key that is almost always absent is a better contract than a key that is almost always null.

The whole of it, in the order you would call it:

import { analyzeSamples, createSlideDetector } from 'video-slide-extractor';

// frames: RGBA samples, one every ~2s, downscaled to 160x90
const { mask, choice } = analyzeSamples(frames, 160, 90);
console.log(choice.mode);            // 'bimodal' | 'static' | 'motion' | 'default'

const detect = createSlideDetector(160, 90, {
  mask,
  changedRatio: choice.changedRatio
});

const kept = [];
frames.forEach((frame, i) => {
  if (detect(frame).keep) kept.push(i);   // sample indices to capture at full res
});

What it is worth, measured

Same fixtures, same protocol: MIT deck, 46 slides, overlay variant, one 160×90 sample every 2 s, detections matched to labeled transitions within ±2.5 s.

The mask alone takes the flood out: 125 captures down to 54, duplicates from 63% to 19%, F1 from 0.538 to 0.880. And it costs nothing on footage that does not need it — on the clean and noisy variants it returns null, so those rows are byte-for-byte what 0.1.1 produced.

The threshold is a trade, not a free win, and the honest way to say it is with the row that looks bad: on the clean MIT deck the calibrated threshold is more conservative than the old constant, and recall drops from 0.935 to 0.761. That is precisely why DEFAULTS did not move in this release. analyzeSamples is what you call when you do not know what kind of footage you were handed. When you do know, the constants are still there and still yours to set.

Both new rows were measured with the benchmark harness at the release commit. The published RESULTS.md still reports the method set from 0.1.x; adding the mask to it as a first-class method is the next thing to land in bench/.

What is deliberately not in it

Global duplicate collapse and transition smoothing are not here and are not coming. Both need the whole video at once — which shot came back later, which three captures were one dissolve — and that is a pipeline’s job. A detector that answers "did the picture change" should not also be holding the video.

Neither is slide density: how many slides a 90-minute lecture should produce is product policy, not detection. Video2Any has an opinion about it and a control for it. A library should not have either — it should tell you where the picture changed and let the answer stay yours.

Three related reads:

• Too many slides from one video — what Video2Any does about density
• Decoding video in parallel, in a tab — the pipeline this detector sits inside
• Zoom recording to PowerPoint — where the webcam bubble walks in the door

What is next

• A dwell gate. New content has to hold still for a couple of seconds before it counts as a slide. This is the thing that actually ends the 900-slide runs — the mask removes the overlay, but a fast dissolve or a scrubbed timeline still fires — and it runs in Video2Any today. It reaches the package once its interface is right: dwell means nothing without a sample rate, and the package deliberately does not know how often you sampled.
• A real overlay, not a synthetic one. The current overlay fixture is a rendered animation in a corner. A recorded webcam of an actual person is a harder, more honest test, and the mask should be measured against one.
• The failure neither method solves. Consecutive slides that differ by one added bullet line stay near the trigger floor at 160×90, so incremental builds get missed — visible in the benchmark even on a synthetic re-render. That is a resolution and scoring-policy problem, and docs/evaluation.md now makes the build-counting policy an explicit, reported parameter rather than something two evaluators can silently disagree about.
• Where calibration should happen. Video2Any splits a video across four workers, and each one calibrates its threshold on its own slice — so the same file can produce slightly different decks on machines with different core counts, which is not a property anyone wants. The obvious fix is to calibrate once, globally. We built it and measured it: on a 29-minute meeting where ffmpeg counts seven real cuts, per-chunk calibration found six and global found four. Global calibration is not automatically better, and the package will not pretend otherwise until there is a version that is.

Try it

It is one install and no dependencies, and it will run on whatever recording you have lying around:

npm i video-slide-extractor

If you build something with it, or you have footage that makes it fail, the repository is the place — a video that breaks the detector is worth more to us than a star. That is how the overlay fixture got written.

• Source, benchmark and evaluation protocol on GitHub
• video-slide-extractor on npm
• What is open and what stays in the app
• The detector with a file picker in front of it

Convert a videoBlog