{"notes":[{"id":"deepseek/deepseek-ocr","category":"deepseek","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"DeepSeek-OCR","subtitle":"A Picture Is Worth a Thousand Words","author":"Z. Yuan","paperDate":"2025-10-21","publishedOn":"2026-03-09","world":"external/deepseek-ocr","paperTitle":"DeepSeek-OCR: Contexts Optical Compression","paperUrl":"https://arxiv.org/abs/2510.18234","bibliography":{"title":"DeepSeek-OCR: Contexts Optical Compression","authors":["Haoran Wei","Yaofeng Sun","Yukun Li"],"url":"https://arxiv.org/abs/2510.18234","pdfUrl":"https://arxiv.org/pdf/2510.18234v1","codeUrl":"https://github.com/deepseek-ai/DeepSeek-OCR","preprint":{"id":"2510.18234","submittedOn":"2025-10-21","url":"https://arxiv.org/abs/2510.18234"},"publication":null,"citation":"@misc{deepseekdeepseekocr2025,\n  title = {{DeepSeek-OCR: Contexts Optical Compression}},\n  author = {Haoran Wei and Yaofeng Sun and Yukun Li},\n  year = {2025},\n  eprint = {2510.18234},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2510.18234}\n}\n","citationPath":"/content/papers/citations/deepseek/deepseek-ocr.66e8e5a7db8d.bib"},"thumbnail":{"src":"/content/papers/assets/6f53ecfe63f39804454a69ef4468c93e8018fa1afbe08d1bb52b7e8d52d194c3.png","width":1288,"height":350},"locale":"en","path":"/content/papers/en/deepseek/deepseek-ocr.37f4dfc7859e.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"overall-architecture","text":"Overall Architecture","depth":3},{"id":"walking-one-10241024-page-through","text":"Walking One 1024×1024 Page Through","depth":3},{"id":"the-decoder-recovering-text-from-a-compressed-representation","text":"The Decoder: Recovering Text from a Compressed Representation","depth":3},{"id":"data-and-training","text":"Data and Training","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"compression-experiments-100-pages-from-fox","text":"Compression Experiments: 100 Pages from Fox","depth":3},{"id":"real-documents-omnidocbench","text":"Real Documents: OmniDocBench","depth":3},{"id":"optical-compression-and-forgetting","text":"Optical Compression and Forgetting","depth":3},{"id":"discussion-after-release","text":"Discussion After Release","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-evaluation/linguistic-crutch","relation":"referenced","topic":""}]},{"id":"model-evaluation/linguistic-crutch","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Linguistic Crutch","subtitle":"The Price of Filling In the Blanks","author":"Z. Yuan","paperDate":"2026-01-07","publishedOn":"2026-02-21","paperTitle":"Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCR","paperUrl":"https://arxiv.org/abs/2601.03714","bibliography":{"title":"Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCR","authors":["Yunhao Liang","Ruixuan Ying","Bo Li","Hong Li","Kai Yan","Qingwen Li","Min Yang","Okamoto Satoshi","Zhe Cui","Shiwen Ni"],"url":"https://arxiv.org/abs/2601.03714","pdfUrl":"https://arxiv.org/pdf/2601.03714v2","codeUrl":null,"preprint":{"id":"2601.03714","submittedOn":"2026-01-07","url":"https://arxiv.org/abs/2601.03714"},"publication":null,"citation":"@misc{modelevaluationlinguisticcrutch2026,\n  title = {{Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCR}},\n  author = {Yunhao Liang and Ruixuan Ying and Bo Li and Hong Li and Kai Yan and Qingwen Li and Min Yang and Okamoto Satoshi and Zhe Cui and Shiwen Ni},\n  year = {2026},\n  eprint = {2601.03714},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2601.03714}\n}\n","citationPath":"/content/papers/citations/model-evaluation/linguistic-crutch.af1f73a098b5.bib"},"thumbnail":{"src":"/content/papers/assets/3dfc834dee089d76acab6a1ce767b06c592ef84c6b48c2a5f79a7723f9cc3f09.png","width":644,"height":300},"locale":"en","path":"/content/papers/en/model-evaluation/linguistic-crutch.41a291e10181.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"keeping-the-image-conditions-identical","text":"Keeping the Image Conditions Identical","depth":3},{"id":"three-kinds-of-disruption","text":"Three Kinds of Disruption","depth":3},{"id":"two-more-tests","text":"Two More Tests","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"when-sentences-stop-making-sense","text":"When Sentences Stop Making Sense","depth":3},{"id":"when-words-stop-looking-like-words","text":"When Words Stop Looking Like Words","depth":3},{"id":"what-about-other-models","text":"What About Other Models?","depth":3},{"id":"question-answering-and-long-context","text":"Question Answering and Long Context","depth":3},{"id":"what-to-know-when-reading-this-paper","text":"What to Know When Reading This Paper","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"deepseek/deepseek-ocr","relation":"citing","topic":""}]}],"text":"Judging by the name, this is an OCR model. Read the abstract, though, and you find that the authors are really asking something else.\n\nWhen an LLM processes long text, the cost of attention grows with the square of the length. If a page of a document comes to a thousand text tokens, could you take a picture of it, give it to the model as a hundred, or even sixty-four, vision tokens, and have the model read the original text back?\n\nIf so, images become a way to compress text. The authors call this idea **contexts optical compression**, and OCR is the testing ground where they check it: the input is an image, the answer is the original text, and a simple comparison tells you whether it's right.\n\nThis is a technical report that DeepSeek-AI uploaded to arXiv on October 21, 2025; this note follows v1. The weights and code were open-sourced at the same time.\n\n## Defining the Problem\n\nIn the related-work section, the authors put the question bluntly: for a document of a thousand words, what is the minimum number of vision tokens needed to decode it back?\n\nAnswering that requires an encoder whose number of vision tokens can be adjusted. The authors survey three common approaches in the open-source VLMs of the time, each with drawbacks:\n\n1. **Dual-tower**, as in Vary: an extra SAM encoder runs in parallel to handle high resolution. Parameters and memory stay reasonably under control, but images need two separate preprocessing paths, and pipeline parallelism is hard to apply during training.\n2. **Tile-based**, as in InternVL2.0: a large image is cut into small tiles that are encoded separately. It can handle extremely high resolutions, but the native resolution is usually below 512×512, so a large image gets chopped into many fragments and the number of vision tokens shoots up.\n3. **Adaptive resolution**, as in Qwen2-VL: patches are cut directly from the original image. It's flexible, but activation memory for large images is very high, and once the vision-token sequence gets long, both prefill and generation slow down.\n\nThe authors wanted an encoder that meets five conditions: it handles high resolution, its memory doesn't blow up at high resolution, it produces few vision tokens, it supports multiple resolutions, and its parameter count is moderate. No existing option met all five, so they built their own, called **DeepEncoder**.\n\n## Solving the Problem\n\n### Overall Architecture\n\n[![DeepSeek-OCR architecture: the input page is split into 16×16 patches and passed through DeepEncoder, made up of SAM (local attention), a 16× downsampling convolution, and CLIP (global attention); the result, together with the prompt, goes to the DeepSeek-3B MoE decoder, which outputs text](/content/papers/assets/6f53ecfe63f39804454a69ef4468c93e8018fa1afbe08d1bb52b7e8d52d194c3.png)](/content/papers/assets/6f53ecfe63f39804454a69ef4468c93e8018fa1afbe08d1bb52b7e8d52d194c3.png)\n\n*Figure 3 from the paper, excerpted from [**page 5 of the v1 PDF**](https://arxiv.org/pdf/2510.18234v1#page=5), CC BY 4.0. DeepEncoder is inside the dashed box on the left; the decoder is on the right.*\n\nLet's go through it once from left to right:\n\n- **SAM** (80 million parameters, ViTDet architecture) handles perception, mainly with local window attention.\n- **Conv 16x** is two convolutional layers that cut the number of tokens to one sixteenth.\n- **CLIP** (ViT, 300 million parameters) handles global attention. Its original patch embedding layer is removed, since its input is already the tokens output by the previous stage.\n- **DeepSeek-3B** is the decoder, a 3-billion-parameter MoE that activates about 570 million parameters per inference.\n\nDeepEncoder as a whole has about 380 million parameters. The logic of this arrangement: **while there are many tokens, do only cheap local attention, and save the expensive global attention for after compression**.\n\n### Walking One 1024×1024 Page Through\n\nWe'll use the paper's own example, a 1024×1024 page in Base mode:\n\n1. SAM works in 16×16-pixel patches, giving 1024/16 × 1024/16 = 64×64 = **4,096** patch tokens. This stage relies mainly on window attention and has only 80 million parameters, so memory holds up.\n2. The convolution module is two layers with kernel 3, stride 2, and padding 1, raising the channels from 256 to 1024. Each layer halves both height and width: 64×64 → 32×32 → 16×16, leaving 4,096 / 16 = **256** tokens.\n3. These 256 tokens go into CLIP for global attention. Pairwise attention over 256 tokens, compared with 4,096, means a score table 256 times smaller.\n4. The decoder receives these 256 vision tokens along with the prompt and outputs text autoregressively.\n\nSo “how many tokens a page becomes” is determined entirely by the input resolution:\n\n| Mode | Resolution | Vision tokens | Processing |\n| --- | --- | ---: | --- |\n| Tiny | 512×512 | 64 | Direct resize |\n| Small | 640×640 | 100 | Direct resize |\n| Base | 1024×1024 | 256 | Aspect ratio kept, padded |\n| Large | 1280×1280 | 400 | Aspect ratio kept, padded |\n| Gundam | n local 640×640 tiles + 1024×1024 global view | n×100 + 256 | Tiled, n from 2 to 9 |\n\n*Table: Compiled from §3.2.2 and Figure 4 of the paper. There is also a Gundam-master mode (1024×1024 local + 1280×1280 global), obtained by continuing to train the already-trained model.*\n\n\n> [!TIP]\n> \n> **How Many Tokens Are Still Valid After Padding?**\n> \n> Base and Large modes pad the image to a square to keep its aspect ratio, and tokens in the padded area carry no content. The paper gives a formula for the valid tokens:\n> \n> $$\n> N_{valid}=\\left\\lceil N_{actual}\\times\\left[1-\\frac{\\max(w,h)-\\min(w,h)}{\\max(w,h)}\\right]\\right\\rceil\n> $$\n> \n> Try it on a portrait A4 page, whose width-to-height ratio is about 1:1.414. The term in brackets is $1-(1.414-1)/1.414\\approx0.707$. Base mode gives $\\lceil256\\times0.707\\rceil=182$, and Large mode gives 283.\n> \n> In the OmniDocBench table later on, Base is listed as 256(182) and Large as 400(285). The table reports averages over the whole test set, where page proportions vary; here we only use A4 to check the order of magnitude.\n\n\n### The Decoder: Recovering Text from a Compressed Representation\n\nThe authors describe the decoder's job as a mapping: from $n$ compressed vision tokens $\\mathbf{Z}\\in\\mathbb{R}^{n\\times d_{latent}}$, reconstruct the representation of $N$ text tokens.\n\nThe decoder is DeepSeekMoE, which at inference activates 6 of its 64 routed experts plus 2 shared experts. In the authors' words, it has the expressive power of a 3-billion-parameter model at an inference cost close to that of a small 500-million-parameter one.\n\nIt's worth pausing here. When $N$ is more than ten times $n$, the decoder can't get by on “translating” each vision token one at a time. It has to use the language knowledge it has learned to expand the compressed information into full sentences. That is why the method works, and it is also what later research would question; we'll come back to it in the discussion.\n\n### Data and Training\n\nDocument data is the centerpiece. The authors collected about 30 million PDF pages from the web, covering about 100 languages, roughly 25 million of them in Chinese and English. The annotations come in two kinds:\n\n- **Coarse annotations**: text extracted directly from the PDFs with fitz, mainly to teach the model to recognize characters in many languages.\n- **Fine annotations**: 2 million pages each of Chinese and English, labeled with interleaved “coordinates + category + text” using layout models such as PP-DocLayout and OCR models such as MinerU and GOT-OCR2.0. For minority languages, they used fitz to generate small images, trained a GOT-OCR2.0 on them, and then used it to annotate, producing 600,000 samples.\n\nThere are also 3 million Word documents, 10 million scene-text samples each in Chinese and English labeled with PaddleOCR, and what the authors call “OCR 2.0” data: 10 million charts (labeled as HTML tables), 5 million chemical formulas (SMILES), and 1 million plane-geometry samples.\n\nThe overall mix is 70% OCR data, 20% general vision data, and 10% text-only data.\n\nTraining has two stages. First, DeepEncoder is trained on its own with a small language model and a next-token prediction objective; then SAM and the compressor are frozen, and the whole model is trained together with the decoder. The main training run used 20 nodes with eight A100-40G GPUs each, processing about 70 billion tokens of multimodal data per day.\n\n## Discussion\n\n### Compression Experiments: 100 Pages from Fox\n\n[![Left: precision and compression ratio for 64 and 100 vision tokens at different text lengths on the Fox benchmark; right: average number of vision tokens versus overall edit distance for each model on OmniDocBench](/content/papers/assets/45775893f2592575b0f63ed52ce871bcfdd8aacf7f1772a97ee461c2c1e86e52.png)](/content/papers/assets/45775893f2592575b0f63ed52ce871bcfdd8aacf7f1772a97ee461c2c1e86e52.png)\n\n*Figure 1 from the paper, excerpted from [**page 1 of the v1 PDF**](https://arxiv.org/pdf/2510.18234v1#page=1), CC BY 4.0. In (a), bars are precision (left axis) and dashed lines are compression ratio (right axis); in (b), further right on the horizontal axis means fewer vision tokens, and higher on the vertical axis means lower edit distance.*\n\nStart with the left plot. The horizontal axis is the number of text tokens in each page's ground truth; the purple bars are 64 vision tokens (Tiny), the blue bars 100 (Small), and the dashed lines the corresponding compression ratios.\n\nThe authors selected every English page in the Fox benchmark with between 600 and 1,300 text tokens, which came to exactly 100 pages, and tested them with “Free OCR,” a prompt that doesn't ask for layout. A few rows from Table 2:\n\n| Text tokens | 64 vision tokens | 100 vision tokens | Pages |\n| --- | --- | --- | ---: |\n| 600–700 | 96.5% (10.5×) | 98.5% (6.7×) | 7 |\n| 800–900 | 83.8% (13.2×) | 96.8% (8.5×) | 28 |\n| 1000–1100 | 79.3% (16.5×) | 91.5% (10.6×) | 11 |\n| 1200–1300 | 59.1% (19.7×) | 87.1% (12.6×) | 4 |\n\n*Table: Excerpted from Table 2 of the paper. Compression ratios are in parentheses.*\n\nTwo things to keep in mind when reading this table:\n\n1. **The page counts are uneven.** The abstract's “about 60% at 20× compression” comes from the last row, which has only 4 pages; the ranges in the middle, from 700 to 900, have 28 pages each.\n2. **Format differences pull the scores down.** The model's output format doesn't fully match Fox's ground truth, and the authors say actual performance should be somewhat higher than the table shows.\n\nFor the decline beyond 10× compression, the authors offer two possible reasons: longer documents have more complex layouts, and at 512×512 and 640×640, long text is already blurry. The latter, they suggest, could be treated as a kind of “forgetting mechanism.”\n\n### Real Documents: OmniDocBench\n\nThe right plot and Table 3 switch to OmniDocBench, a real-world document parsing benchmark. The paper doesn't state the version; the table scores English and Chinese separately, consistent with the format before v1.5. Every metric is an edit distance, so lower is better. The authors' emphasis is token efficiency:\n\n- **Small mode** uses only 100 vision tokens and reaches an overall English edit distance of 0.221, lower than GOT-OCR2.0 with 256 tokens (0.287).\n- **Gundam mode** averages 795 tokens and scores 0.127 in English and 0.181 in Chinese, both better than MinerU2.0, which averages 6,790 tokens (0.133 and 0.238).\n\nTable 4 breaks the results down by document type, showing how many tokens each kind of document needs. Slides already reach an edit distance of just 0.116 in Tiny mode; books and reports already do well in Small mode. Newspapers are another matter entirely: 0.94 in Tiny, falling to 0.122 only in Gundam. The authors' explanation is that a newspaper page has four to five thousand text tokens, far beyond the roughly 10× compression range of the other modes.\n\n### Optical Compression and Forgetting\n\nIn the discussion section, the authors sketch a further idea: earlier content in a multi-turn conversation could be rendered as images and compressed; the older the conversation, the smaller the image, with fewer tokens and blurrier text, like human memory slowly fading.\n\nThe authors themselves call this a very preliminary exploration. The conclusion is clear: **OCR alone is not enough to fully validate true contexts optical compression**; future work includes pretraining on interleaved digital and optical text, needle-in-a-haystack tests, and other evaluations.\n\n### Discussion After Release\n\nThe paper drew plenty of discussion after it came out. In a post on October 20, 2025, Andrej Karpathy said it was a good OCR model, but what interested him more was whether pixels might make better LLM inputs than text. His reasons included information compression, keeping formatting such as bold and color, and getting rid of the tokenizer. [Karpathy, 2025-10-20](https://x.com/karpathy/status/1980397031542989305)\n\nAnother line of response was to check how much of those high scores actually came from the image. In January 2026, “Visual Merit or Linguistic Crutch?” replaced Fox's paragraphs with random letters: Tiny mode fell from 88% to about 20%, while on the same data PaddleOCR-v5, which recognizes text line by line, dropped only from 94% to 90%.\n\nBack to that mapping: when each vision token has to account for a dozen or more text tokens on average, the decoder's language knowledge has to fill in a great deal. With normal English, that knowledge is what makes the compression work; when the content follows no pattern, it can't fill the gaps back in.\n\n\n> [!TIP]\n> \n> For the full setup of the semantic perturbation experiments, see:\n> \n> - **[[26.01] Linguistic Crutch: The Price of Filling In the Blanks](/blog/papers/model-evaluation/linguistic-crutch)**\n\n\n## Conclusion\n\nDeepSeek-OCR turns “how many vision tokens does a page need?” into a measurable question and gives a practical answer:\n\n1. **DeepEncoder's arrangement works.** Local attention first, then compression, then global attention lets a 1024×1024 page produce only 256 vision tokens while keeping memory in check.\n2. **At about 10× compression, English pages can mostly be read back.** On Fox, precision is about 97% within 10×; at higher compression ratios it starts to drop, and the figures around 20× rest on only a handful of pages.\n3. **As an OCR tool, its token efficiency is excellent.** Small mode already beats GOT-OCR2.0, and Gundam mode beats MinerU2.0 with less than an eighth of the tokens. The authors also say a single A100-40G can produce more than 200,000 pages of training data a day.\n\nAs for “compressing an LLM's long context with images,” the paper takes only the first step. The OCR scores show that the decoder can recover the text, but they don't tell us how much of the recovered text came from the image. Three months later, someone took up that question.\n","imageSizes":{"/content/papers/assets/6f53ecfe63f39804454a69ef4468c93e8018fa1afbe08d1bb52b7e8d52d194c3.png":{"width":1288,"height":350},"/content/papers/assets/45775893f2592575b0f63ed52ce871bcfdd8aacf7f1772a97ee461c2c1e86e52.png":{"width":1272,"height":558}}}