The easily underestimated part of long-video retrieval is not a single transcript or image description. It is making observations from different sources point to the same moment in the original media. This MVP is a personal interest experiment with a narrow goal: ingest a long video, create rebuildable local artifacts, return queries with time ranges and evidence references, and export a matching clip when a result needs human review.
Logical chunks and global time
Audio and frames can be analyzed in logical chunks so that a long input does not have to pass through every stage at once. Chunking is an execution detail, not the clock exposed by retrieval. Each chunk records its offset within the original video, and local timestamps are converted before an artifact is stored:
global time = chunk start offset + time within the chunk
Transcripts, scene boundaries, and export ranges therefore share one coordinate system even when they cross chunk boundaries. Chunk size and scheduling can change later without forcing the query layer to understand which worker or chunk produced a record.
Joining speech and visual observations
Speech transcription is useful for “what was said.” Visual observations add “what appeared” and “when the scene changed.” The two remain distinct artifact types, then meet through start and end times. A transcript record carries text and a time range. A visual record carries a concise observation, a sampled instant or covered interval, and a reference to its source artifact.
Joining does not mean flattening both streams into one long document. Retrieval selects candidates by relevance and temporal overlap, then groups nearby speech and visual evidence for presentation. This keeps each observation traceable and avoids presenting one sampled frame as an explanation of an entire spoken section.
JSONL artifacts and evidence references
Each ingestion produces inspectable artifacts: media metadata plus JSONL files for chunk records, transcripts, visual observations, and timeline entries. The query path reads the transcript, visual-observation, and timeline JSONL files and searches all three sources. Original media and derived data remain separate instead of being copied into one large analysis file.
Matching candidates keep their start time, end time, reason, score, and evidence references. Nearby candidates can be merged into one segment while retaining unique references and an export range. These references show which transcript, visual observation, or timeline event contributed to the segment. They support review but do not turn a model-generated description into an unquestionable fact.
Rebuilding the index
The derived index can be regenerated from artifacts already on disk. The current rebuild command reads transcript, visual-observation, and timeline JSONL files, converts their rows into three groups of index documents, writes index_documents.jsonl, and records the total and per-source document counts in index_manifest.json.
Artifact version fields, content-digest validation, rejection of invalid source rows, and a global timeline sort during rebuild are possible future hardening steps; they are not part of the current implementation. The rebuildable index remains a derived representation rather than the only record of what processing produced.
Exporting a clip from evidence
After a match, its evidence interval becomes a clip-export request. The exporter adds a small amount of context around the matched range and clamps the result to the media duration. A bundle may contain a video clip, audio, subtitles, representative frames, and a manifest describing the interval and evidence references. Optional interval-level visual review can refine the clip boundary while preserving the original match times for comparison.
Export is the entry point for human inspection, not a decorative extra. A prose summary alone makes a boundary hard to judge. Passing the same timeline coordinates to a player and exporter reconnects retrieval with material that can actually be watched.
Where the experiment fits
This structure suits personal media organization, lecture review, interview navigation, and finding moments in long screen recordings. Its usefulness depends on timestamp quality and is limited by transcription errors, sparse frame sampling, and imperfect visual descriptions. Fast action, small on-screen text, overlapping speech, and frame-level questions still need denser observation or manual review.
The current state verifies structure and processing order: logical chunks, global time conversion, replaceable analysis backends, index rebuilding, and evidence export. It is not presented as a large production run with heavyweight models, and it does not attempt archival governance, content moderation, or automatic factual judgment.
What I kept from the experiment
The most useful part was a shared clock and rebuildable artifacts, not the addition of one more model. When every observation retains its source identifier and time range, retrieval methods can change and mistakes still have a place to inspect. Organizing evidence before optimizing summaries makes a long-video tool feel more like a reviewable research aid.