It never loses a file,
and it is never stopped by one

Every tool that reads documents works on the tidy majority. The difference is what happens on the rest: the file that hangs the reader, the listing that names a file the disk does not hold, the retry that looks like progress. This page is about the rest. The figures are from the engineering record; the approximate ones carry a tilde.

Never loses a file means never silently. It does not mean every file is read: 787 legacy Office files have no reader, 962 are stuck in a cloud drive, photographs are media-only, and two Archive files hung the worker and are quarantined for a person. Each has a row in the manifest with a status and a reason. The promise is that the list of what did not come through is as complete as the list of what did.

Never stopped by one means the run finishes. A corrupt file, a missing one, a file that locks the reader inside C code — each costs one file's timeout and one row. None halts the pipeline, and none is reported as anything other than what it was.

The first version read legacy .doc files through an office daemon. Two faults compounded: each file waited on the previous file's profile lock, and the timeout could never fire because a grandchild process held the output pipe open. Across the SIG tree that was ~578 stalls, the worst bottleneck. A native, single-shot converter that cannot hang took it to ~3, marked doc-needs-relo for a contained re-sweep. The lesson: prefer a single-shot tool, and assume any converter will hang.

Then a malformed PDF froze the extraction lane, and this was worse: the hang was inside the PDF library's C code, where no Python timeout can reach. The only defence is process isolation — run the risky parse in a short-lived child, give it no output pipe to deadlock on, put it in its own process group, and kill the group on a hard timeout. Every risky read now runs that way.

Before that, the backstop was an external watchdog that restarted the whole worker when the manifest stopped growing. It recovered, but every hang cost the full stall window plus a re-walk of the tree; on a hang-dense folder most of the wall-clock went to waiting. Unique throughput was ~3 files a minute. With each file in its own child and a short hard timeout it is ~2,250 a minute, with zero false quarantines — no good file marked bad because the worker was slow. Two trees of the same company's records differed enormously in hang density. The rule: you cannot predict the pathological few percent, so a hang has to be cheap, not merely survivable.

The most surprising failure was not a file format. A storage layer reported files it did not hold: offloaded to a cloud drive, still in the directory listing. The reader tried to open what was not there, failed or hung, and quarantined them — 87% of the files in one region, not one of them bad. The sentence that went into the record:

The storage layer will lie to you before the file formats do.

The rule now: a file the storage layer names but cannot deliver is reported as absent, with the error type in its status — never marked corrupt, never quietly passed over. The 962 cloud-stuck files in the proving corpus are reported that way today: a count a person can act on.

A retry loop once made a manifest 3.6 times larger than the truth: 15,800 rows over about 4,400 real files, because every failed file was tried again on every restart. The rule now: a failed extraction is terminal, retried only in a bounded re-sweep a person starts, and progress is counted in unique files, never rows. The manifest page tells it in full.

A hard kill once landed mid-row and left a truncated line that crashed every restart at the same place; now it is one writer, every row committed whole, and a reader that skips a row with no path. And the OCR lane once built its entire worklist in memory before reading a page, ran out of memory and looped; it now takes its worklist from the manifest.

Nothing on the source side is written to, renamed or moved; the readable copies go to a separate folder that can be deleted and rebuilt. Every copy carries the path of its original, the method that read it and its status, so a quoted line can always be traced to the file on your disk. A copy that has lost its original's address is a loss too, and is counted as one: when a move left 1,097,899 source paths pointing at nothing, they were repaired to zero.