Every file resolves to a status.
None resolves to silence.
The manifest is one row per file: where it was, what it is, how it was read and what happened. Coverage, gaps and the worklists fall out of grouping by status. This page publishes the whole vocabulary — including the statuses that exist only because something went wrong — and the counting rule that came out of a manifest that once lied.
Counted, never assumed
A converter that cannot say what it did with every file will, sooner or later, drop your most important one without telling you. The manifest exists so that cannot happen quietly. Each file gets exactly one status, the status says what to do next, and the count is always of files — never of attempts.
One row per file
the shape of itThe manifest is a plain table. Its columns are path, ext,
route, status, bytes, chars,
sha1, depth, top_folder and ingested. The
sha1 is the file's fingerprint, and it makes a second run cheap: a file whose
fingerprint has not changed is not read again. A run that stops resumes from the manifest and
redoes nothing.
Both lanes read from it. The extraction lane writes a row for every file it meets; the OCR lane never walks the folder; it takes the rows tagged ocr-candidate as its
worklist. A problem in one lane never blocks the other, and the expensive lane runs only over
what needs it.
The status vocabulary
what each means, what you doThis is the whole list. Each status is a queryable worklist: group by this column and you have what is in, what is queued, what was set aside and why. Three exist because of a
specific failure on the proving run, kept for the same reason the failures are published — a
list with only ok on it would not be worth trusting.
| Status | What it means | Where the file stands | What a person does |
|---|---|---|---|
ok | Clean text extracted | In the corpus | Nothing. Ask questions of it. |
data | A spreadsheet, extracted as rows of text | In the corpus | Nothing. Figures can be cited by row. |
ocr-candidate | A PDF with little or no text layer — a scan | Queued | Run the OCR lane; this list is its worklist. |
media-only | An image or photograph; no text to extract | Not searchable | Nothing yet. Counted and kept. |
empty | Opened cleanly, contained no text | Noted | Glance at it if you expected text. |
skipped-large | Over the size guard | Held | A streamed re-read. |
skipped:<ErrorType> | Missing, moved or corrupt when the OCR lane reached it | Reconcile | Check the folder; the listing may name a file the disk lacks. |
extract-failed:doc-needs-relo | A legacy .doc the native single-shot converter could not read | Terminal | A bounded re-sweep with the heavier converter, never the main loop. |
extract-failed:crash-or-hang-quarantine | Hung the worker; killed from outside | Quarantined | An isolated retry in a bounded re-sweep; if it hangs again, open it. |
extract-failed:lo-timeout | The older converter timed out on it | Retired | Nothing. Kept so old manifests still read. |
Note what is not in the table: a status meaning "retrying". A failed extraction is terminal — it stays queryable, and is attempted again only inside a bounded re-sweep a person starts, never by the main loop. The next section is why.
Count files, never rows
the 3.6× lessonOn one run over the SIG tree the manifest reached 15,800 rows. Progress looked good. The unique files in it numbered about 4,400. The same files had been processed three and a half times over; the row count was measuring retries, not work.
Two bugs, compounding. The hidden lock stubs Office leaves beside an open document were picked up by the walk and failed, correctly, as unreadable. And resume treated every failed extraction as retryable, so every restart tried every failed file again, and each failure got another row. The fix, now the rule: the walk filters junk before it reaches a reader, and resume marks every path already in the manifest as done. Progress is measured in unique files. A row count can balloon while real coverage stands still.
A second lesson from the same period. A hard kill landed mid-append and left a truncated row; every restart read it, found no status, and died at the resume step until the reader was taught to skip a row with no path. The rule is now one writer per manifest and every row committed whole. A hard kill must never be able to poison resume.
What the manifest cannot know
the honesty clauseA manifest can only account for the files the walk was shown. It cannot know about a file the
storage layer hid, and it will faithfully record one the storage layer invented. On the proving run a storage layer reported files it did not hold — offloaded to a cloud drive,
still in the listing. The reader opened what was not there, failed or hung, and quarantined
them: 87% of the files in one region, none of them bad. The record puts it in
these words: "the storage layer will lie to you before the file formats do." The
skipped:<ErrorType> status exists so a file the disk does not hold is
reported as absent, not corrupt; 962 files in the proving corpus are stuck that way today and
reported that way.
The worklists still owed
Grouped by status, the proving manifests give an honest picture, and part of it is
unfinished. The SIG tree has 883 scans tagged ocr-candidate and 0 read. 787
legacy .doc and .xls files have no reader. Photographed documents
are media-only. Two Archive files are quarantined, waiting for a person. Nothing
filed after 2026-07-22 is in any corpus. Each is a count read from the manifest — the gaps sit beside the successes — and the record carries what
would close each one.
Where to go next
three pagesNever loses a file
Why a hang costs one file rather than the run, and the rest of the failure catalogue.
Open ▸ Plain wordsWhat it cannot read
The gaps in the table above, explained without jargon.
Open ▸ From row to citationProvenance
How each row's source path, method and status follow the text into an answer.
Open ▸