{"notes":[{"id":"text-spotting/nougat","category":"text-spotting","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Nougat","subtitle":"Going in Circles","author":"Z. Yuan","paperDate":"2023-08-25","publishedOn":"2026-04-14","world":"external/nougat","paperTitle":"Nougat: Neural Optical Understanding for Academic Documents","paperUrl":"https://arxiv.org/abs/2308.13418","bibliography":{"title":"Nougat: Neural Optical Understanding for Academic Documents","authors":["Lukas Blecher","Guillem Cucurull","Thomas Scialom","Robert Stojnic"],"url":"https://arxiv.org/abs/2308.13418","pdfUrl":"https://arxiv.org/pdf/2308.13418v1","codeUrl":"https://github.com/facebookresearch/nougat","preprint":{"id":"2308.13418","submittedOn":"2023-08-25","url":"https://arxiv.org/abs/2308.13418"},"publication":null,"citation":"@misc{textspottingnougat2023,\n  title = {{Nougat: Neural Optical Understanding for Academic Documents}},\n  author = {Lukas Blecher and Guillem Cucurull and Thomas Scialom and Robert Stojnic},\n  year = {2023},\n  eprint = {2308.13418},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2308.13418}\n}\n","citationPath":"/content/papers/citations/text-spotting/nougat.1d66a2aca861.bib"},"thumbnail":{"src":"/content/papers/assets/73888d5b579ab68d56551dc59d69ebcd0bf97704d288425f0341c9a9aa012138.png","width":1283,"height":288},"locale":"en","path":"/content/papers/en/text-spotting/nougat.7c3766703f2e.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"architecture-borrowed-from-donut","text":"Architecture: Borrowed from Donut","depth":3},{"id":"data-cutting-latex-source-into-pages","text":"Data: Cutting LaTeX Source into Pages","depth":3},{"id":"making-digital-pdfs-look-scanned","text":"Making Digital PDFs Look Scanned","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"results-on-the-arxiv-test-set","text":"Results on the arXiv Test Set","depth":3},{"id":"stuck-in-repetition","text":"Stuck in Repetition","depth":3},{"id":"detecting-repetition","text":"Detecting Repetition","depth":3},{"id":"what-users-ran-into","text":"What Users Ran Into","depth":3},{"id":"what-to-know-when-reading-this-paper","text":"What to Know When Reading This Paper","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"Feed a paper's PDF to an ordinary OCR engine, and most of the text comes out fine, but the math often falls apart into a heap of symbols.\n\nIt's easy to see why. In $x_i^2$, the subscript $i$ and the superscript $2$ are stacked to the right of $x$; a fraction puts its numerator above its denominator. Line-by-line OCR treats them as ordinary text, and the spatial relationships are lost. The PDF itself doesn't help much either: it records “which glyph is drawn where on the page,” and the original LaTeX structure of the math was already thrown away at compile time.\n\nNougat, introduced by Meta AI in August 2023, tries to solve this in one stroke: take an image of a paper's page and directly output markup text with LaTeX math and tables. This note follows arXiv v1. The code is open source under the MIT license, and the model weights are released under CC BY-NC.\n\n## Defining the Problem\n\nEach existing tool fell short somewhere:\n\n- OCR engines such as **Tesseract** are good at detecting and classifying individual characters or words, but they work line by line and can't make sense of spatial relationships such as superscripts and subscripts.\n- **GROBID** parses born-digital paper PDFs into XML and handles bibliographic data well; for math, it keeps only the Unicode characters found in the PDF.\n- **pdf2htmlEX** converts PDFs into HTML that looks the same, but it can't recover the meaning of the math either.\n\nThe authors wanted a model that looks only at the page image. Working from the image alone means old scanned papers and books can be handled too; outputting markup directly means the structure of math and tables can be preserved.\n\n## Solving the Problem\n\n### Architecture: Borrowed from Donut\n\n[![Nougat architecture: a document page image is encoded by a Swin Transformer into a latent representation z, and a Transformer decoder autoregressively outputs tokens; example output text is shown on the right](/content/papers/assets/73888d5b579ab68d56551dc59d69ebcd0bf97704d288425f0341c9a9aa012138.png)](/content/papers/assets/73888d5b579ab68d56551dc59d69ebcd0bf97704d288425f0341c9a9aa012138.png)\n\n*Figure 1 from the paper, excerpted from [**page 2 of the v1 PDF**](https://arxiv.org/pdf/2308.13418v1#page=2), CC BY-SA 4.0.*\n\nThe architecture is simple and borrows from Donut (2021):\n\n1. **Encoder**: the page's margins are cropped off, and it is resized and padded to a fixed 896×672 (an aspect ratio between US Letter and A4), and a Swin Transformer turns it into a sequence of patch vectors $z$.\n2. **Decoder**: a 10-layer mBART decoder reads $z$ through cross-attention and outputs tokens autoregressively, up to 4,096 of them. The tokenizer is borrowed from Galactica, a model built for scientific text.\n\nThe whole model has about 350 million parameters. The authors also tried a smaller 250-million-parameter version whose decoder has only 4 layers. Inference uses greedy decoding.\n\nThere is no text detection here and no OCR preprocessing of any kind. How the characters get read is left entirely for the network to learn.\n\n### Data: Cutting LaTeX Source into Pages\n\nThe biggest engineering effort went into the data. Training “page image → markup text” requires a large amount of paired data, and none existed at the time.\n\nThe authors collected the LaTeX source and compiled PDFs of 1,748,201 papers from arXiv, added a portion of open-access papers from PubMed Central, and, for pretraining, also used OCR text from the Industry Documents Library.\n\nThe processing pipeline:\n\n1. Convert LaTeX to HTML with LaTeXML, which along the way expands custom macros, normalizes whitespace, standardizes tables, and resolves references to the correct numbers.\n2. Convert the HTML into a lightweight markup language that keeps headings, bold and italic text, algorithms, inline and display LaTeX math, and LaTeX tables.\n3. Split the full source into pages and match each one to the corresponding page image of the PDF.\n\nThe third step is the hardest. LaTeX page breaks are decided by the compiler, and the source contains no page-break markers; figures and tables also often end up somewhere else in the PDF than in the source. The authors handle it in two passes:\n\n- **Coarse split**: extract the text of each PDF page with MuPDF and train a TF-IDF plus linear SVM classifier to predict which page each paragraph of the source belongs to. Ideally this produces a staircase-shaped sequence of page numbers; in practice it is noisy, so they use Gini impurity, as in decision trees, to find the best split points.\n- **Fine split**: near each predicted split point, fuzzy-match the source against the last few sentences of the previous PDF page and the first few sentences of the next to find the exact position. A page enters the dataset only if its two split points average a score of at least 0.9.\n\nIn the end, about 47% of the pages passed the filter.\n\n### Making Digital PDFs Look Scanned\n\nThe training data consists entirely of born-digital papers, yet the model has to handle scans. The authors used Albumentations to add a series of image perturbations: erosion, dilation, Gaussian noise, blur, binarization, compression, grid distortion, and elastic transforms.\n\nThey also added perturbations on the **text side**; we'll come back to what that does later.\n\n## Discussion\n\n### Results on the arXiv Test Set\n\n| Method | Category | Edit distance ↓ | BLEU ↑ | F1 ↑ |\n| --- | --- | ---: | ---: | ---: |\n| Embedded PDF text | All | 0.255 | 65.8 | 79.2 |\n| GROBID | All | 0.312 | 55.6 | 73.0 |\n| Nougat base | All | 0.071 | 89.1 | 93.1 |\n| Nougat base | Plain text | 0.058 | 91.2 | 95.7 |\n| Nougat base | Math | 0.128 | 56.9 | 76.5 |\n| Nougat base | Tables | 0.211 | 69.7 | 78.0 |\n\n*Table: Excerpted from Table 1 of the paper.*\n\nOverall, Nougat's edit distance is only about a quarter of GROBID's. Of the three content types, math is the hardest, with more than twice the edit distance of plain text.\n\nThe authors explain that math scores are inherently low: the same expression can be written many ways in LaTeX, such as the order of subscripts and superscripts, `\\frac` versus `\\over`, or `\\mathbf` versus `\\boldsymbol`. They render identically but produce different strings.\n\nThe small model (250 million parameters) scores almost the same as the large one.\n\n### Stuck in Repetition\n\nWhat makes this paper most worth reading is how candidly the authors record what failure looks like.\n\nThe model sometimes falls into repetition. In the simplest case, it outputs the same sentence or paragraph over and over; it may also alternate between two sentences, changing a few words each time, so that strict repetition detection misses it. In the references section, the model may even count up the numbering on its own and keep writing. Once it enters this state, it can't get back out.\n\nThis happens on about 1.5% of the pages in the test set, and more often on documents outside the training domain. The authors observe that repetition usually starts after the model has made a mistake.\n\nThat suggests a remedy: **let the model see its own mistakes during training**. During training, they randomly replace some correct tokens with arbitrary ones, and the replacement continues until a newly sampled random number exceeds 10%. The model thus learns to keep reading when the preceding text contains an error, instead of being carried off by the wrong token. The authors saw no drop in overall performance but a clear reduction in repetition; on out-of-domain documents, failed page conversions fell by 32%.\n\n### Detecting Repetition\n\nReducing repetition alone isn't enough. The decoder generates up to 4,096 tokens, and once it is stuck in a loop, it runs all the way to that limit for nothing. So the authors also designed a detection method.\n\n[![Two repetition-detection examples with three plots each: the maximum logit of each token, the sliding-window variance, and the variance from that position to the end. The top row falls into repetition after about token 1,000, and its variance drops sharply below the threshold; the bottom row is a normal sample whose variance stays high throughout](/content/papers/assets/1a40149c71166aacbd2c464be00126a4ae00240e18bac1afb5a04502192d89a2.png)](/content/papers/assets/1a40149c71166aacbd2c464be00126a4ae00240e18bac1afb5a04502192d89a2.png)\n\n*Figure 6 from the paper, excerpted from [**page 8 of the v1 PDF**](https://arxiv.org/pdf/2308.13418v1#page=8), CC BY-SA 4.0. The top row is a sample with repetition and the bottom row a normal sample; the dashed lines mark where repetition starts and the decision threshold.*\n\nStart with the top-left plot: the horizontal axis is the index of the generated token, and the vertical axis is the largest logit at each step. Before roughly token 1,000, the points are widely scattered; after that, they all crowd into a band between 17 and 18. The normal sample in the bottom row stays widely scattered throughout.\n\nIntuitively, with normal text the model's confidence rises and falls from step to step; once it is stuck in repetition, it grows more and more certain about the next token, and the maximum logit flattens out. The authors turn this into a two-step computation:\n\n1. Over the maximum-logit sequence $\\ell$, slide a window of size $B=15$ and compute the variance within each window, $\\mathrm{VarWin}_B$ (the middle plot).\n2. Then, from each position to the end of the sequence, compute the variance of those variances, $\\mathrm{VarEnd}_B$ (the right plot).\n\nIf $\\mathrm{VarEnd}_B$ drops below 6.75 and stays there until the end, the output is classified as repetitive.\n\nThree numbers give a feel for it (illustrative values, not data from the paper). In a normal stretch, the maximum logits might be 20, 25, and 18, with a variance of about 8.67; during repetition, they might be 17.0, 17.2, and 16.9, with a variance of only 0.016. With a gap of several hundredfold, the two are easy to tell apart.\n\nDuring inference, the “end” isn't visible yet, so the authors use only the most recent 200 tokens and half the threshold, checking as generation proceeds; once generation finishes, they run the check again over the full sequence.\n\n### What Users Ran Into\n\nIssue #11 on Nougat's GitHub is a report from August 30, 2023. A user tested a scan of the 1935 EPR paper from Physical Review: the first page came out fine, pages two through four all triggered a “Found repetitions” warning, and the output for them was just `[MISSING_PAGE_EMPTY:1]`. [facebookresearch/nougat#11](https://github.com/facebookresearch/nougat/issues/11)\n\nThat is repetition detection at work: a page judged to have failed is marked as missing instead of being output as a page of repeated text. For the user, a missing-page marker at least signals that this page needs separate handling.\n\nThe official README also states the language scope plainly: English papers work best, other Latin-script languages might work, and Chinese, Russian, Japanese, and the like won't.\n\n### What to Know When Reading This Paper\n\nThe authors list several points in their limitations section:\n\n1. **Repetition remains the biggest problem**, and the paper leaves it to future work.\n2. **Almost English only.** Special characters in other Latin-script languages are replaced by the closest English letter; non-Latin scripts fall into repetition immediately.\n3. **One page at a time.** Consistency across pages isn't guaranteed. The most obvious case is the references: the numbering sometimes skips, or picks up numbers the original doesn't have.\n4. **It is slow.** On a single A10G, processing 6 pages at a time takes 19.5 seconds per batch on average, whereas GROBID can process 10.6 PDFs per second.\n\n## Conclusion\n\nNougat recasts OCR for academic papers as a generation task: “look at a page, write a page of markup.”\n\n1. **A simple architecture**: a Swin Transformer encoder plus an mBART decoder, with no OCR preprocessing.\n2. **Data engineering takes center stage**: page-level training data cut from the LaTeX source of 1.75 million arXiv papers, using a classifier and fuzzy matching.\n3. **Math can be read end to end into LaTeX for the first time**, with a much lower overall edit distance than GROBID.\n\nIt also laid out the typical ailments of generative OCR early on: falling into repetition, replacing rare characters with common letters, and adding reference numbers that don't exist. The authors didn't stop at reporting scores; they also built countermeasures, injecting errors during training and detecting repetition during inference.\n\nOnce a page of text is handed to a language model to “write,” it will try its best to make the writing flow. When it can't go on, it's better to make it stop than to let it keep writing.\n","imageSizes":{"/content/papers/assets/73888d5b579ab68d56551dc59d69ebcd0bf97704d288425f0341c9a9aa012138.png":{"width":1283,"height":288},"/content/papers/assets/1a40149c71166aacbd2c464be00126a4ae00240e18bac1afb5a04502192d89a2.png":{"width":1233,"height":800}}}