A reproducible test protocol for wearable transcript quality

Status and purpose

This is a pre-registered methodology, not a product result. It defines the test before registered physical samples are accepted so the method cannot be rewritten around a favorable outcome.

The question is deliberately narrow: can speech captured by a wearable become a faithful, attributable transcript? The protocol does not score summaries, tasks, memories or recommendations because BEFORE.Band does not produce those objects.

1. Freeze the scenarios before testing

Use only fictional speech and fictional identities. Publish the fixture-set version, reference transcripts and scoring code before the first measured run. English, French and mixed-language material must remain separate reporting strata rather than being blended into one flattering average.

• Single-speaker speech in English and French at normal and fast pace.
• A consented two-speaker exchange with interruptions and short turns.
• Walking, indoor room noise and controlled street-noise conditions.
• A long recording that crosses upload or processing seams.
• Silence and non-speech audio to test invented words.
• Interrupted transfer, retry and recovery without duplicate transcript text.

2. Measure fidelity, not polish

Score the current clean transcript against the locked reference, while retaining the verbatim transcript version as evidence. Readability changes count as errors when they alter meaning, attribution, uncertainty, numbers, names or negation.

Every metric must publish its numerator, denominator and missing or failed runs. A single composite score is not enough to reveal where a transcript failed.

• Word error rate and character error rate, reported by language and scenario.
• Anonymous speaker-label consistency and speaker-attribution error.
• Timestamp coverage and absolute timestamp error.
• Duplicated or omitted words at chunk and retry seams.
• Words hallucinated during silence or non-speech audio.
• Meaning-changing edits introduced by conservative cleanup.
• End-to-end latency, processing failures and recovery success.

3. Preserve a complete run record

Tie each result to the registered device, firmware, capture settings, phone and app versions, transcription provider and model version, test-fixture version and scoring-code revision. Record the acoustic setup and failure history rather than keeping only successful runs.

Raw fictional audio remains in the controlled evaluation environment and is never exposed through MCP, logs or analytics. Public evidence contains reference text, transcript output, aggregate measures and redacted diagnostic metadata sufficient to reproduce the score.

4. Keep hardware and provider gates separate

Software fixtures can validate scoring and transcript version behavior, but they cannot establish microphone, storage, transfer or recovery performance. No physical-product result is valid until it comes from a registered sample under the published setup.

A supplier listing, demonstration clip or transcription-provider benchmark does not count as BEFORE.Band evidence. If the hardware or provider changes, report a new result rather than silently carrying the previous score forward.

5. Publish the failures with the score

The public report should contain a scenario-by-scenario table, aggregate and disaggregated metrics, confidence intervals where repeat runs permit them, every missing or failed run, known limitations and a dated change log.

A claim should link directly to the corresponding report version. Corrected transcripts remain visible as new versions rather than overwriting the evidence used for an earlier result.

• Device, firmware, pipeline and fixture identifiers.
• Results for every pre-registered scenario, including failures.
• Reference and output transcript versions for fictional fixtures.
• Limitations, deviations and unresolved failure modes.
• Date, protocol version and any superseded report link.