Six digital formats, scanned paper page by page —
and a written status for everything else

It reads plain text, Markdown, CSV, Excel workbooks, Word documents and PDFs that carry a text layer. A PDF without one is a scan, and a scan goes to a second lane that reads it page by page and writes down how sure it was about each page. Anything else is recorded, not lost.

The list is short on purpose. Each of these has a reader that was run over the founder's own archive of 56,687 files, and each comes out as a Markdown copy naming its source path, the method that read it and the status it was given.

A spreadsheet becomes rows of text: enough to find a figure and cite the row it sits on, not a working workbook. That is the right shape, because the engine never recalculates what it reads — ordinary code does the sums. A Word document keeps its paragraphs and tables, not its fonts or tracked changes. The copy is for finding and quoting; the original stays where it was, untouched.

What is not in the table matters as much. Legacy .doc and .xls — the older Office formats — have no reader in the product today, and 787 of them in the proving corpus are waiting for one. They are in the manifest with a status. They are not searchable.

The decision is made per file and the reading is done per page. A PDF with little or no text on its opening pages is a scan: it is tagged ocr-candidate and left for the second lane, which takes its worklist from the manifest rather than walking the folder again. An earlier version built the whole scan list in memory first, ran out of memory, restarted and never wrote a row. Driving it from the manifest fixed that, and lets the two lanes stop and resume independently.

For each page, a local OCR engine reads the image and records a mean confidence for what it read. Pages under the threshold are the hard tail; a stronger local engine can be turned on to re-read them. A cloud engine also exists and is off by default: switching it on needs both an explicit flag and credentials, and before a single page goes anywhere it prints a banner naming the provider and the exact number of pages that would leave the machine. Without that flag, nothing leaves.

In the proving run the OCR lane read 7,996 documents and 35,633 pages, local-only, with zero egress — both counts. The engine and confidence for every page is written into the derived document's provenance, so a poor page is recorded as poor, not smoothed over, and when an answer is later quoted from it the confidence travels with the citation.

A file the reader does not recognise, one that is corrupt, one that opens cleanly and is empty, one the listing names but the disk does not hold — each is an expected input, not an error. Each gets a status: empty, media-only, skipped-large, skipped:<ErrorType>, or one of the extract-failed statuses. The run does not stop; the row is written. Group the manifest by status and you see, in one list, everything that did not come through and why. The whole vocabulary is published.

Junk is filtered before any of this. The hidden lock stubs Office leaves beside an open document, desktop metadata files, thumbnail caches — these are recognised on the walk and never reach a reader. That sounds trivial. It was not: on one run, those stubs failing and being retried on every restart were the largest single cause of a manifest that reported three and a half times more work than had been done.