Where every answer came from

Provenance is the answer to one question: where did this line come from? Not roughly — which file, on which path, read by which method, on which day, and was the file the same bytes then as now. The engine carries that answer from conversion to citation without dropping it.

When a file is converted, the readable copy opens with a small header the conversion layer writes before the text. It is plain and fixed, the same for a scanned invoice as for a spreadsheet.

Every file that goes in resolves to one of those statuses. A file the reader could not open is not skipped silently; it is written down as failed, with the reason, and its copy carries a one-line placeholder. Unique files are counted, never rows — a retry loop once made a manifest 3.6 times larger than the files it described, and the counting rule was fixed because of it.

A PDF is handled page by page. Digital text is taken first; a page that yields too little is treated as a scan, rasterised and read by a local OCR engine. For each such page, the engine that read it and its mean word confidence are recorded — on the page block and in the OCR manifest — so a low-confidence page shows as low-confidence rather than passing as clean text. In the archive OCR run, 7,996 documents and 35,633 pages were read this way, local-only, zero egress.

A cloud OCR provider exists in the code, off by default. Nothing leaves the machine unless it is switched on by hand and given credentials — and the tool says in plain words, before it starts, that low-confidence pages would go off-machine.

When the engine indexes a readable copy, it strips the header first, so none of it enters the index or can be quoted back as content. It then numbers the lines of the body — so a locator points at real text — and reads the source of record from the header. The citation you receive names the original file, not the converted copy.

A citation carries a document id, the source path, and a locator: line numbers for text, a page number for a PDF, a cell reference for a spreadsheet. If the page was read by OCR, the provenance carries that flag and the confidence figure with it. The quoted lines are verbatim from the copy; the copy is verbatim from the reader; the reader's status says how much to trust the read.

Every reader in the conversion layer is read-only. Nothing is written into a source file, ever; the readable copies go to a separate folder that can be deleted and regenerated. The fingerprint in the header is what makes that safe to say: if an original changes, its fingerprint no longer matches, and the copy is known to be stale.

Provenance can also break, and it has. At one point 1,097,899 recorded source paths were dead, pointing at files that could not be found. They were repaired to zero rather than left as citations to nowhere. A citation is only worth something if the path on it opens.

Everything above tells you where a line came from. None of it tells you who wrote it. The record knows the file, the path, the reader, the day and the bytes. It does not know whether the person who typed the line was the estimator or a summer student, whether the file was the draft or the signed version, whether it was later superseded by a file in another folder, or whether the figure was right when it was written.

That difference is the reason for the human gate. A citation to a superseded spreadsheet is a true citation to a wrong number. The engine can prove the line is in your records; it cannot vouch for it. Provenance tells you where to look. Only a person can say whether to believe what is there — which is why nothing enters the record until one does.