Request
Would NARA consider supporting a small, well-defined subset of Markdown in transcription text submitted via POST /api/v2/transcriptions/ (and rendered in the Catalog UI)?
Today a transcription is a single plain-text blob. That works for prose, but it loses structure that is genuinely present in many documents:
- Tables — muster rolls, pension ledgers, census/tally sheets, manifests. As plain text these collapse into ambiguous runs of spaces; the row/column relationships (which are the whole point of the document) are lost.
- Emphasis — italics and bold that appear in the original (underlined headings, emphasized terms, ship names, Latin phrases).
- Basic structure — headings, lists, paragraph/line breaks that survive round-tripping.
A constrained Markdown subset — say tables, *italics*/_italics_, **bold**, headings, lists, and line breaks — would preserve far more of the document's meaning while staying plain-text, diffable, accessible, and safe (no raw HTML).
Why now / why it matters
High-quality OCR is increasingly Markdown-native. We're running Mistral OCR over NARA digital objects (it handles both typed text and handwriting) and it already emits Markdown — including reconstructed tables — as its natural output. We'd like to contribute that work back to the Catalog so it benefits everyone (we're building a PHP client for exactly this: https://github.com/survos/nara-php-api). Right now we have to flatten that structured output to plain text to submit it, discarding the table reconstruction that was the most valuable part.
Questions
- Is there any current or planned support for structured/marked-up transcription text?
- If we submit Markdown today, how is it stored and displayed — preserved verbatim, escaped, or stripped? Is there a max length or character-set constraint we should know about?
- Would a constrained, no-raw-HTML Markdown subset be acceptable from a moderation/safety standpoint? We're happy to follow whatever sanitization rules NARA prefers.
- If Markdown is out of scope, is there a recommended convention for representing tabular data in a transcription (e.g. TSV, pipe tables) that the Catalog will render or at least preserve?
Happy to provide sample documents (with and without tables) and example OCR output if useful. Thanks for maintaining the API!
Request
Would NARA consider supporting a small, well-defined subset of Markdown in transcription text submitted via
POST /api/v2/transcriptions/(and rendered in the Catalog UI)?Today a transcription is a single plain-text blob. That works for prose, but it loses structure that is genuinely present in many documents:
A constrained Markdown subset — say tables,
*italics*/_italics_,**bold**, headings, lists, and line breaks — would preserve far more of the document's meaning while staying plain-text, diffable, accessible, and safe (no raw HTML).Why now / why it matters
High-quality OCR is increasingly Markdown-native. We're running Mistral OCR over NARA digital objects (it handles both typed text and handwriting) and it already emits Markdown — including reconstructed tables — as its natural output. We'd like to contribute that work back to the Catalog so it benefits everyone (we're building a PHP client for exactly this: https://github.com/survos/nara-php-api). Right now we have to flatten that structured output to plain text to submit it, discarding the table reconstruction that was the most valuable part.
Questions
Happy to provide sample documents (with and without tables) and example OCR output if useful. Thanks for maintaining the API!