SA Strategic Agent Grounded on your own records
The conversion layer

Every file resolves to a status.
None resolves to silence.

The manifest is one row per file: where it was, what it is, how it was read and what happened. Coverage, gaps and the worklists fall out of grouping by status. This page publishes the whole vocabulary — including the statuses that exist only because something went wrong — and the counting rule that came out of a manifest that once lied.

Counted, never assumed

56,687Files in the Archive manifest, one status each
2Quarantined for a person to look at
3.6×How much larger a manifest once grew than the truth
0 of 883SIG scans queued for OCR that have been read

A converter that cannot say what it did with every file will, sooner or later, drop your most important one without telling you. The manifest exists so that cannot happen quietly. Each file gets exactly one status, the status says what to do next, and the count is always of files — never of attempts.

One row per file

the shape of it

The manifest is a plain table. Its columns are path, ext, route, status, bytes, chars, sha1, depth, top_folder and ingested. The sha1 is the file's fingerprint, and it makes a second run cheap: a file whose fingerprint has not changed is not read again. A run that stops resumes from the manifest and redoes nothing.

Both lanes read from it. The extraction lane writes a row for every file it meets; the OCR lane never walks the folder; it takes the rows tagged ocr-candidate as its worklist. A problem in one lane never blocks the other, and the expensive lane runs only over what needs it.

The status vocabulary

what each means, what you do

This is the whole list. Each status is a queryable worklist: group by this column and you have what is in, what is queued, what was set aside and why. Three exist because of a specific failure on the proving run, kept for the same reason the failures are published — a list with only ok on it would not be worth trusting.

StatusWhat it meansWhere the file standsWhat a person does
okClean text extractedIn the corpusNothing. Ask questions of it.
dataA spreadsheet, extracted as rows of textIn the corpusNothing. Figures can be cited by row.
ocr-candidateA PDF with little or no text layer — a scanQueuedRun the OCR lane; this list is its worklist.
media-onlyAn image or photograph; no text to extractNot searchableNothing yet. Counted and kept.
emptyOpened cleanly, contained no textNotedGlance at it if you expected text.
skipped-largeOver the size guardHeldA streamed re-read.
skipped:<ErrorType>Missing, moved or corrupt when the OCR lane reached itReconcileCheck the folder; the listing may name a file the disk lacks.
extract-failed:doc-needs-reloA legacy .doc the native single-shot converter could not readTerminalA bounded re-sweep with the heavier converter, never the main loop.
extract-failed:crash-or-hang-quarantineHung the worker; killed from outsideQuarantinedAn isolated retry in a bounded re-sweep; if it hangs again, open it.
extract-failed:lo-timeoutThe older converter timed out on itRetiredNothing. Kept so old manifests still read.

Note what is not in the table: a status meaning "retrying". A failed extraction is terminal — it stays queryable, and is attempted again only inside a bounded re-sweep a person starts, never by the main loop. The next section is why.

Count files, never rows

the 3.6× lesson

On one run over the SIG tree the manifest reached 15,800 rows. Progress looked good. The unique files in it numbered about 4,400. The same files had been processed three and a half times over; the row count was measuring retries, not work.

Two bugs, compounding. The hidden lock stubs Office leaves beside an open document were picked up by the walk and failed, correctly, as unreadable. And resume treated every failed extraction as retryable, so every restart tried every failed file again, and each failure got another row. The fix, now the rule: the walk filters junk before it reaches a reader, and resume marks every path already in the manifest as done. Progress is measured in unique files. A row count can balloon while real coverage stands still.

A second lesson from the same period. A hard kill landed mid-append and left a truncated row; every restart read it, found no status, and died at the resume step until the reader was taught to skip a row with no path. The rule is now one writer per manifest and every row committed whole. A hard kill must never be able to poison resume.

What the manifest cannot know

the honesty clause

A manifest can only account for the files the walk was shown. It cannot know about a file the storage layer hid, and it will faithfully record one the storage layer invented. On the proving run a storage layer reported files it did not hold — offloaded to a cloud drive, still in the listing. The reader opened what was not there, failed or hung, and quarantined them: 87% of the files in one region, none of them bad. The record puts it in these words: "the storage layer will lie to you before the file formats do." The skipped:<ErrorType> status exists so a file the disk does not hold is reported as absent, not corrupt; 962 files in the proving corpus are stuck that way today and reported that way.

The worklists still owed

Grouped by status, the proving manifests give an honest picture, and part of it is unfinished. The SIG tree has 883 scans tagged ocr-candidate and 0 read. 787 legacy .doc and .xls files have no reader. Photographed documents are media-only. Two Archive files are quarantined, waiting for a person. Nothing filed after 2026-07-22 is in any corpus. Each is a count read from the manifest — the gaps sit beside the successes — and the record carries what would close each one.

Where to go next

three pages
The property this serves

Never loses a file

Why a hang costs one file rather than the run, and the rest of the failure catalogue.

Open ▸
Plain words

What it cannot read

The gaps in the table above, explained without jargon.

Open ▸
From row to citation

Provenance

How each row's source path, method and status follow the text into an answer.

Open ▸