Grounded video RAG benchmark: protocol, results and claim boundaries
This note freezes the public interpretation of a small comparison between BonFrame's grounded local editing path and a DeepSeek full-transcript baseline. It exists so that product copy cannot silently expand an eight-case result into a general model claim.
Question
Can a retrieval-based /cut path use substantially less generation context without falling more than five percentage points below a full-transcript cloud baseline on the same locked cases?
The task is source-grounded span selection, not free-form summarization. All successful outputs must resolve to real transcript sentences and satisfy the same duration and structural gates.
Paths
| Path | Evidence available to selector | Selector | Time coordinates |
|---|---|---|---|
| A8 | Retrieved ordered block directory | local Ministral 8B Q4 | stable transcript IDs |
| A3 | Retrieved ordered block directory | local Ministral 3B | stable transcript IDs |
| B | Full transcript | DeepSeek | resolved to allowed source spans |
| C | Same short block directory | DeepSeek | stable block IDs |
Path C checks whether a hosted model automatically performs better when it sees the same compressed evidence. It is diagnostic, not the primary baseline.
Locked data and repetitions
- Eight transcripts were held out from assembly-rule tuning.
- The product-protocol runs did not change assembly gates on those eight cases.
- Each case was repeated three times: 24 trials per path.
- Labels were swapped by
caseId#repeatbefore judging. - Mechanical gates checked duration and structural requirements.
- A separate GPT-5.1 judge evaluated grounding, coverage, exclusion and coherence.
- A trial passed only when both the mechanical and judged gates passed.
The predeclared non-inferiority margin was -5 percentage points. The token target required at least an 80% reduction in generation input relative to path B.
Earlier negative result
Before constrained local block selection was added, a rule-assembled retrieval path passed 2/8 cases while the full-transcript baseline passed 5/8. That run did not establish non-inferiority. It showed that retrieval alone can find relevant evidence while still producing a poor edit.
The subsequent system changed one responsibility: after retrieval built an ordered directory, a local model selected existing block_id values before frozen assembly. It did not generate timestamps.
Aggregate results
| Path | Mechanical gate | Combined pass | Difference from B | Mean generation input |
|---|---|---|---|---|
A8: product /cut + local 8B |
24/24 | 17/24 (70.8%) | +8.3pp | 860 |
A3: product /cut + local 3B |
24/24 | 14/24 (58.3%) | -4.2pp | 860 |
| B: DeepSeek full transcript | 24/24 | 15/24 (62.5%) | — | 4,600 |
| C in the 8B protocol | 24/24 | 11/24 (45.8%) | -16.7pp | short directory |
| C in the 3B protocol | 24/24 | 9/24 (37.5%) | -25.0pp | short directory |
The measured generation-input reduction for A8 and A3 relative to B was 81.3%.
Known failures
- The Docker case scored 0/3 for every path in the 8B protocol. In the 3B protocol, B passed only 1/3.
- Promotional or demonstration-heavy introductions could survive because they repeated the requested topic.
- Photosynthesis exposed incomplete coverage when short candidates ended before the full mechanism.
- Fair use showed that the same-evidence cloud selector could outperform 3B on an individual case even though its aggregate score was lower.
- Candidate and sentence boundaries sometimes mattered more than selector size.
Permitted interpretation
The result supports this statement:
On this locked eight-case, source-grounded span-selection task, BonFrame's local 8B path passed 17/24 trials versus 15/24 for a DeepSeek full-transcript baseline while using about 81% fewer generation input tokens. The local 3B path passed 14/24, within the predeclared -5pp non-inferiority margin.
It does not support:
- “8B beats cloud models”;
- “3B matches every cloud model”;
- “retrieval is always better than full context”;
- “the result has been confirmed by human editors”;
- “an 81% reduction applies to every video or prompt.”
Missing evidence
Before broad quality claims, the evaluation still needs:
- a larger locked holdout;
- human blind review;
- more domains and languages;
- adversarial intros, sponsorships and demonstrations;
- a second independently locked cloud baseline;
- published fixture metadata and aggregate machine-readable results that do not expose customer media.
The current result is sufficient to justify further engineering and to explain the architecture. It is not sufficient to rank general-purpose models.