| D-M3-15-1 |
2026-10-24 |
Only an attachment (purpose attachment) of a type an extractor supports gets a row and the job, created in the unit of work of the write that makes the file deliverable (recordFileChange, called by the scan pass and the stored original of files.thumbnail); the language is taken once from locale.default through identity.Directory and kept by re-extraction. |
Only attachments are searchable (14 2); the text of an avatar, a logo or a data export would be a copy nobody can reach. One place for every deliverable transition covers every current and future caller. |
| D-M3-15-2 |
2026-10-24 |
unsupported is written only by the job, for a type no extractor supports at run time; the trigger creates no row for an unsupported type. |
Rows exist only for files that may get text; a later version that supports more types creates their rows through the refresh walk. |
| D-M3-15-3 |
2026-10-24 |
Types: builtin-text for every text/* type except text/html and text/rtf, plus application/json; builtin-html for text/html; builtin-ooxml for DOCX, XLSX, PPTX; builtin-odf for ODT, ODS, ODP (the names of the detector). extractor stores <adapter>/<n>. A golden hash per adapter over fixed fixtures fails on any output change, and its update refuses an unchanged version. |
Every output change must raise the version, otherwise the refresh job would never re-extract what the old adapter produced. |
| D-M3-15-4 |
2026-10-24 |
No inbound port TextExtracts is built; 02 4.3 and the module README drop it. |
No other module reads file text; search reads it through SearchFiles (14 10), and a port without a consumer is dead code. |
| D-M3-15-5 |
2026-10-24 |
The closed set of failure codes is input_too_large, archive_invalid, archive_entries_exceeded, archive_size_exceeded, xml_doctype, xml_invalid, timeout, object_missing. A missing object of a deliverable file is failed object_missing with one Error log; any other object store or read error is transient and returns to River with the row still pending. files.extract_text has no River job timeout. |
A code without content (03); a transient store failure must not end a file failed; the use case bounds reading and parsing itself, and River's default timeout could cut the result write. |
| D-M3-15-6 |
2026-10-24 |
Text normalisation: invalid UTF-8 becomes U+FFFD, NUL and other C0 controls except tab, line feed and carriage return become a space, separators collapse, a cut never splits a rune. UTF-16 only behind a byte order mark, no charset guessing; HTML block elements write a line feed, inline elements nothing, no charset conversion. |
PostgreSQL text refuses NUL; guessing charsets would give wrong text that no test can hold. |
| D-M3-15-7 |
2026-10-24 |
OOXML and ODF are read whole into memory (at most search.extract.max_input_bytes) and opened with zip.NewReader; the entry count is checked first, then the declared sizes of the read parts against search.extract.max_uncompressed_bytes, then a shared counter of the bytes actually inflated; duplicate names, the encryption flag and methods other than store and deflate give archive_invalid. SEC-UP-5 and SEC-UP-7 of chapter 19 name the text extraction as the one in-memory exception. |
archive/zip needs random access and ranged reads through the store would cost a request per small read; the checks stop a zip bomb before a byte is inflated, and a lying header fails inside archive/zip. 14 5.2 prescribes archive/zip, which contradicted the unconditional SEC-UP-7. |
| D-M3-15-8 |
2026-10-24 |
Every XML document type declaration gives xml_doctype; other encodings than UTF-8 and undefined entities give xml_invalid; a run guard ends a part when one token grows beyond search.extract.max_text_bytes (counted from the token's start, not from the last < as the brief said), keeps the text before it and sets truncated; open start tags of more than search.extract.max_text_bytes together or a nesting deeper than search.extract.max_xml_depth (new key, default 256) give xml_invalid; elements match by local name; the parts are fixed names (word/document.xml, xl/sharedStrings.xml, ppt/slides/slide<N>.xml, content.xml). |
Office and ODF parts never need a DTD, which carries every external entity; encoding/xml buffers whole tokens, and comments, processing instructions and CDATA may hold a <, so only a count from the token's start bounds the memory of one token; the decoder keeps the name, attributes and one stack entry per namespace declaration of every open element, so a part of nothing but start tags (deep, or shallow with many declarations) would otherwise grow the heap with its inflated size; fixed names keep the parser surface small and give archive_invalid or empty text, never a wrong text. |
| D-M3-15-9 |
2026-10-24 |
When PostgreSQL refuses the search vector (SQLSTATE 54000), the text is halved at a rune boundary, marked truncated and written again in a new unit until it fits. Extraction writes no file.updated, no audit entry and no in-transaction event. |
Many distinct tokens can exceed the tsvector limit within the text limit; the empty text always fits. 12 2 does not list extraction among the changes that write file.updated. |
| D-M3-15-10 |
2026-10-24 |
The periodic job files.extract_text_refresh (queue maintenance, search.extract.refresh_interval, RunOnStart) walks deliverable attachments by id in batches of search.extract.refresh_batch_size and decides each with domain.RefreshDecision: reset a row of another version or a row that failed only for its size when it now fits (compare-and-set that keeps the text, so the file stays findable), create a missing row, re-enqueue a pending row, delete rows of deleted files; a full batch inserts the next run. |
A new binary brings new versions; a discarded job would otherwise leave a row pending forever. |
| D-M3-15-11 |
2026-10-24 |
The text goes when the file goes: markDeleted deletes the extract rows of the files it marks deleted in the same unit of work; the job and the walk catch up, and ON DELETE CASCADE removes the row with the files row. |
Content class data of a deleted file must not stay searchable or stored (14 5.3, 20). |
| D-M3-15-12 |
2026-10-24 |
SearchFiles unions two indexed branches (the file vector, the text vector) under one candidate limit, scores ts_rank over the joined vector and highlights name, transcript and text joined by line feeds in the language of the text (simple without one); the extract row's state is not filtered. |
No OR across tables, a file found both ways counts once, the weights rank name above transcript above text, and a stem match in the text is highlighted. |
| D-M3-15-13 |
2026-10-24 |
No metric of its own for extractions; the job wrapper and the job metrics cover the jobs. |
signals.md names none, and new signals need their own registration. |
| D-M3-15-14 |
2026-10-24 |
The zero id of ExtractRefresh.After is left out of the job arguments (omitzero). |
Job arguments must read back, and an id reads back only as a UUIDv7. |