{"notes":[{"id":"image-generation/vision-banana","category":"image-generation","legacyPath":"image-generation/2604-vision-banana","legacySlug":"2604-vision-banana","legacyPaths":[],"title":"Vision Banana","subtitle":"If it can generate images, surely it can understand them","author":"nbswords","paperDate":"2026-04-22","publishedOn":"2026-05-04","paperTitle":"Image Generators are Generalist Vision Learners","paperUrl":"https://arxiv.org/abs/2604.20329v1","bibliography":{"title":"Image Generators are Generalist Vision Learners","authors":["Valentin Gabeur","Shangbang Long","Songyou Peng","Paul Voigtlaender","Shuyang Sun","Yanan Bao","Karen Truong","Zhicheng Wang","Wenlei Zhou","Jonathan T. Barron","Kyle Genova","Nithish Kannen","Sherry Ben","Yandong Li","Mandy Guo","Suhas Yogin","Yiming Gu","Huizhong Chen","Oliver Wang","Saining Xie","Howard Zhou","Kaiming He","Thomas Funkhouser","Jean-Baptiste Alayrac","Radu Soricut"],"url":"https://arxiv.org/abs/2604.20329v1","pdfUrl":"https://arxiv.org/pdf/2604.20329v1","codeUrl":null,"preprint":{"id":"2604.20329","submittedOn":"2026-04-22","url":"https://arxiv.org/abs/2604.20329v1"},"publication":null,"citation":"@misc{imagegenerationvisionbanana2026,\n  title = {{Image Generators are Generalist Vision Learners}},\n  author = {Valentin Gabeur and Shangbang Long and Songyou Peng and Paul Voigtlaender and Shuyang Sun and Yanan Bao and Karen Truong and Zhicheng Wang and Wenlei Zhou and Jonathan T. Barron and Kyle Genova and Nithish Kannen and Sherry Ben and Yandong Li and Mandy Guo and Suhas Yogin and Yiming Gu and Huizhong Chen and Oliver Wang and Saining Xie and Howard Zhou and Kaiming He and Thomas Funkhouser and Jean-Baptiste Alayrac and Radu Soricut},\n  year = {2026},\n  eprint = {2604.20329},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2604.20329v1}\n}\n","citationPath":"/content/papers/citations/image-generation/vision-banana.8b134972f59d.bib"},"thumbnail":{"src":"/content/papers/assets/f406fe7c1fe33b3cb7b26b45ea0f0feb7cc4d00468788ffa491c6db2144c8da1.jpg","width":1612,"height":798},"locale":"en","path":"/content/papers/en/image-generation/vision-banana.053bd824d6c0.md","headings":[{"id":"intro","text":"Intro","depth":2},{"id":"method","text":"Method","depth":2},{"id":"why-instruction-tuning","text":"Why Instruction Tuning?","depth":3},{"id":"instruction-tuning-recipe","text":"Instruction Tuning Recipe","depth":3},{"id":"2d-semantic-understanding","text":"2D Semantic Understanding","depth":2},{"id":"semantic-segmentation","text":"Semantic Segmentation","depth":3},{"id":"instance-segmentation","text":"Instance Segmentation","depth":3},{"id":"referring-expression-segmentation","text":"Referring Expression Segmentation","depth":3},{"id":"3d-understanding","text":"3D Understanding","depth":2},{"id":"metric-depth-estimation","text":"Metric Depth Estimation","depth":3},{"id":"surface-normal-estimation","text":"Surface Normal Estimation","depth":3},{"id":"retaining-generation-ability","text":"Retaining Generation Ability","depth":2},{"id":"conclusion","text":"Conclusion","depth":2},{"id":"discussion","text":"Discussion","depth":2},{"id":"my-thoughts","text":"My Thoughts","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"In the LLM world, we are used to seeing models trained with generative pretraining acquire downstream abilities in language understanding, reasoning, and coding. Those abilities were not trained as separate tasks; they emerged from the objective of predicting the next token.\n\nCould the same story apply to vision? In other words, when a model is trained to **generate** high-quality images, does it also learn to **understand** images along the way?\n\n## Intro\n\n- Earlier work had already observed scaling behavior in generative vision pretraining, but its results had continued to lag behind non-generative approaches such as contrastive learning and masked autoencoding.\n- This paper's claim is that, with suitable instruction tuning, a state-of-the-art image generator (Nano Banana Pro here) can become a generalist vision model, reaching or exceeding the level of specialist models on various 2D and 3D understanding tasks.\n    - The instruction-tuned model is called **Vision Banana**.\n    - The key design is to express the outputs of every vision task as an **RGB image**. This lets the model use the abilities its image generator already has, without changing its architecture or adding a new head.\n\nThe next two figures show Vision Banana's task overview and benchmark results across these areas. Apart from instance segmentation and image editing, it reaches the best reported results in these comparisons. Let's look more closely at how an image generator handles these vision tasks.\n\n\n\n![benchmark overview 1](/content/papers/assets/f406fe7c1fe33b3cb7b26b45ea0f0feb7cc4d00468788ffa491c6db2144c8da1.jpg)\n\n\n\n\n\n![benchmark overview 2](/content/papers/assets/22bcb4cdd0aacc5ab2c488e46d8477d0bf91af9dfa656ee8af26b5f1bd112aed.jpg)\n\n\n\n## Method\n\n### Why Instruction Tuning?\n\nEarlier research had already found that modern image generators **can** produce pictures that look like segmentation masks or depth maps, even in a zero-shot setting. The trouble is that those pictures **cannot be decoded precisely into actual vision outputs**, so they cannot be compared quantitatively with other models on benchmarks.\n\nFor example, ask the model for a \"depth map\" and it may give you something that looks like one. But if the relationship between color and distance is unclear, there is no way to evaluate it numerically.\n\nThe purpose of instruction tuning, then, is to **teach the model to output a reversible visualization in the format specified by the prompt**. For example:\n\n> \"Segment the skateboard category in pure yellow (\\<255, 255, 0\\>)\"\n\nAfter the model generates the image, we can cluster pixels near the color (255, 255, 0) to obtain the skateboard mask.\n\n### Instruction Tuning Recipe\n\n- Mix vision-task data into Nano Banana Pro's original training mixture at a **very low ratio** and perform a lightweight fine-tune.\n    - That low ratio matters: it helps the model retain its original generation ability, or what the paper repeatedly calls its \"generative prior.\"\n- Sources of the training data:\n    - 2D tasks: internal models annotate images crawled from the web.\n    - 3D tasks: rendering engines produce synthetic data.\n- **No training split from an evaluation benchmark is used** in this instruction tuning. The paper therefore calls the evaluation zero-shot transfer with respect to those benchmarks.\n\n## 2D Semantic Understanding\n\n### Semantic Segmentation\n\nThis is the familiar task of assigning a class to every pixel. Vision Banana's approach is straightforward: specify a color for each class in the prompt, and let the model generate the mask image:\n\n- `\"Generate a semantic segmentation visualization image, using this color mapping: {\"cat\": \"red\", \"lock\": \"pink\", \"exit sign\": \"light purple\", \"background\": \"yellow\"}.\"`\n\nThe prompt is remarkably flexible. The color can be a hex code (`#80C000`), an RGB tuple (`<255, 165, 0>`), or even a description in natural language (`\"light purple\"`); the model can understand all three.\n\nOn Cityscapes, it gets **0.699 mIoU, beating SAM 3's 0.652** by 4.7 percentage points. It is the best among the compared zero-shot transfer models, and it is only about 14 percentage points behind the closed-set SegMan (mIoU = 0.842). Those models do not all share the same training setting.\n\n\n\n![cityscapes mIoU](/content/papers/assets/00f79e12737d98326e1de8237ec710c05d40e51bf6006f306f33a9f4e60ef505.jpg)\n\n\n\nThe next figure shows an example of Vision Banana doing semantic segmentation.\n\n\n\n![semantic segmentation](/content/papers/assets/31cb9cce12f904494488747a77bd5ecab11a14c97bf29f20bdb0242746c2b844.jpg)\n\n\n\n### Instance Segmentation\n\nInstance segmentation is trickier, because **we do not know how many instances there are in advance**. We therefore cannot assign a color to every instance ahead of time in the prompt.\n\nTheir solution is **per-class inference**: segment one class at a time and let the model dynamically assign a different color to each instance.\n\n```\n\"Generate an instance segmentation visualization of this image.\nEach piece of garlic is colored differently.\"\n```\n\nAt evaluation time, they use thresholding to cluster pixels with similar colors.\n\nOn SA-Co/Gold, the reported $pmF_1$ is 0.540, close to DINO-X's 0.552 but still below SAM 3's 0.661. Vision Banana's figure uses a random subset of 500 queries, and the training settings are different. The authors acknowledge that instance segmentation remains a harder task.\n\nThe next figure shows an example of Vision Banana doing instance segmentation.\n\n\n\n![instance segmentation](/content/papers/assets/4832b4701f8fcc839d764d08b813e1a0e673cdf61b87f55b14487863cfb1cb5c.jpg)\n\n\n\n### Referring Expression Segmentation\n\nThis task is even more interesting because the model must understand a free-form text query: \"the man in a pink T-shirt,\" \"the cat that is stretching,\" or \"the toaster being used as a game controller.\"\n\nThis is precisely where the generative model has an advantage: **the multimodal intelligence inherited from pretraining lets it reason about *what* to segment**.\n\nOn ReasonSeg, Vision Banana paired with Gemini 2.5 Pro reaches **0.793 gIoU, beating SAM 3 Agent paired with Gemini 2.5 Pro at 0.770** under the paper's zero-shot transfer setting.\n\nInterestingly, **Vision Banana can also handle referring expressions within semantic and instance segmentation**, even without specific training on free-form queries for those tasks. For example, it understands what \"patterns on the wall\" and \"crescent-shaped croissants\" refer to. That suggests cross-task transfer.\n\nThe following figure shows examples of Vision Banana understanding and interpreting natural language. It can:\n\n1. Describe an object's appearance (\"man in pink t shirt\").\n2. Describe an object's actions (\"stretching\" and \"cleaning\").\n3. Understand an unusual use of an object (a toaster as a game controller).\n4. Understand multilingual text content (Chinese and English text on a menu).\n\n\n\n![referring segmentation](/content/papers/assets/493de66c40da0f5a6f43f0be7fb353aee5280d637a41cd60fbbc65a9702e4594.jpg)\n\n\n\n## 3D Understanding\n\n### Metric Depth Estimation\n\nThis is the part of the paper I find most interesting. Depth estimation differs from segmentation in a fundamental way: depth is an **unbounded continuous value** $d \\in [0, \\infty)$, whereas RGB color is a **bounded three-dimensional space** $[0, 1]^3$. How can the former be encoded as the latter?\n\nThey design a **reversible (bijective)** mapping between depth and a specified path through RGB space:\n\n**Step 1:** Use [Barron's (2025) power transform](https://arxiv.org/abs/2502.10647) to compress unbounded depth into a bounded value:\n\n$$f(d, \\lambda, c) = 1 - (1 - d/\\lambda c)^{\\lambda+1}$$\n\nHere $\\lambda = -3$ and $c = 10/3$. The transform gives **higher resolution to the near field than to the far field**, because nearby objects matter more for robotics or AR/VR.\n\n**Step 2:** Use that normalized distance in $[0, 1)$ for **piecewise-linear interpolation** along the edges of the RGB cube. The path resembles the first iteration of a 3D Hilbert curve, running from black (0,0,0) to white (1,1,1).\n\n\n\n![RGB cube path](/content/papers/assets/08afd0a73c5bdba98d28a7e5133ceae3e657ceebb43c4cde8baf77923f0fb5cb.jpg)\n\n\n\nThe mapping along that path is **reversible**, so they:\n\n- turn ground-truth depth into an RGB image with the mapping during training, to use as the training target;\n- project a generated RGB image **onto the nearest cube edge** during inference, then invert the mapping to recover metric depth.\n\nAcross six benchmarks, the average $\\delta_1$ accuracy is 0.882. On the **four datasets shared with Depth Anything v3**, Vision Banana scores **0.929 versus 0.918**, and its depth fine-tuning uses **no camera intrinsics** or **real-world depth training data**—the depth data are synthetic.\n\n\n\n![depth benchmark](/content/papers/assets/68251ed3e2158a9654608fd193ca185bab05b2bd5c2325a3f642f67ee0bbf3f4.jpg)\n\n\n\nThe next figure shows Vision Banana doing metric depth estimation.\n\n\n\n![metric depth example](/content/papers/assets/720d7e158df7595a62d4fb276c6ebf256721cb37dee2afaff436a9e4445e70b2.jpg)\n\n\n\nThe paper also has an amusing \"vibe test\": one of the authors took a photograph at Kinkaku-ji. Vision Banana estimated the distance to a green-marked point as 13.71 meters; a measurement with Google Maps gave 12.87 meters, for an AbsRel error of about 0.065.\n\n\n\n![vibe test](/content/papers/assets/2b59ea22a1f87d3b5ddf9cebec79936c2d0fc3c194c42acbf10f8c67aea928a3.jpg)\n\n\n\n### Surface Normal Estimation\n\nA surface normal is a unit vector $(x, y, z) \\in [-1, 1]^3$, so its relationship to RGB is much more natural. There is no need for a mapping as elaborate as the one for depth. The model uses camera-space normals directly:\n\n- Facing Left $(-1, 0, 0)$: Pinkish Red\n- Facing Up $(0, 1, 0)$: Light Green\n- Facing Camera $(0, 0, 1)$: Light Blue / Purple\n\nWhen averaged across the three indoor datasets, the reported mean and median angular errors are the lowest among the compared specialist models, including Lotus-2, StableNormal, and DSINE.\n\n\n\n![normal benchmark](/content/papers/assets/22978dc6783a6e858b15e61be795dd3230725a9a8657c8ac1f714766b00f6928.jpg)\n\n\n\nThe next figure shows an example of Vision Banana doing surface normal estimation.\n\n\n\n![normal example](/content/papers/assets/025ad8948111c8235d1cad3b2d49ea9406a006d63918b5ebe83174b1f285042b.jpg)\n\n\n\n## Retaining Generation Ability\n\nDoes instruction tuning damage the model's original image-generation ability?\n\n- Text-to-image (GenAI-Bench): **53.5%** win rate against Nano Banana Pro.\n- Image editing (ImgEdit): **47.8%** win rate.\n\nIn other words, Vision Banana and the base model are **roughly tied** on these generation tasks, supporting the claim that instruction tuning was lightweight.\n\n## Conclusion\n\n1. **Image generators are themselves generalist vision learners:** Strong visual-understanding abilities **emerge naturally** during generative pretraining. Instruction tuning merely \"unlocks\" them. This echoes how LLMs changed the field of NLP.\n2. **Image generation can serve as a universal interface for vision:** Just as text generation serves as a universal interface for NLP tasks, vision outputs (masks, depth, normals) can all be expressed as RGB images. Specify the visualization scheme in the prompt to perform different tasks.\n3. **Generative models are naturally equipped to handle ambiguity:** A single input can have several reasonable outputs. In segmentation, for example, there can be multiple reasonable ways to decide \"which region counts as one instance.\" Discriminative models need rather hacky losses to avoid collapsing to a blurry mean (the SAM family, for example, produces multiple masks but computes the loss for only one). A generative model already models the whole distribution, so it can handle that ambiguity naturally. That is the paper's motivation, rather than a guarantee that every generated sample is correct.\n4. **Future work:**\n    - Broaden the variety of instruction-tuned tasks, perhaps unlocking more cross-task generalization.\n    - Move from monocular input to multi-view and video input.\n    - Integrate with LLMs for cross-modality reasoning.\n    - Reduce inference cost, because an image generator is far heavier than a lightweight specialist model.\n\n## Discussion\n\nTraditionally, people have built a dedicated architecture for each vision task: a mask decoder for segmentation, a regression head for depth, a special loss function for normals. Every task looks different, and models cannot share as much. Vision Banana instead shapes every vision output as an RGB image. The **image generator** itself becomes a universal interface for vision tasks. This feels like the way GPT unified NLP tasks as text generation, and I think it has real potential.\n\nThe metric-depth design in particular—the power transform plus piecewise-linear interpolation along the edges of the RGB cube—is, to me, the cleverest part of the paper. It addresses a major obstacle to using a generation model for vision tasks: what it generates could not previously be decoded precisely back into metric values for a benchmark. Without this invertible mapping, even an attractive depth visualization would remain only qualitative.\n\nThe way generative models handle ambiguity is also worth thinking about. To avoid collapsing to a blurry mean, discriminative models have to devise all sorts of hacky losses; the SAM family's practice of outputting multiple masks but computing the loss for just one is a typical example. A generative model already models the entire distribution, so this problem can be handled more naturally.\n\n## My Thoughts\n\nAfter reading this paper, I feel that Vision Banana's contribution lies more in **reframing the problem** than in a new algorithm or architectural innovation.\n\nIt invents no new loss, no new module, and no new architecture. It simply rethinks the **output format of vision tasks**: with a well-designed reversible RGB encoding and a strong image-generator base model, it can beat various specialists without a specialized architecture.\n\nI think this direction could well become a mainstream path for vision foundation models over the next few years. The visual-understanding abilities emerging from generative pretraining should receive more attention. Paired with a universal output format like RGB, perhaps we really will see an omni model that can perform every vision task someday.\n\nOf course, inference cost remains a practical issue. A model at Vision Banana's scale probably will not be deployed directly on a phone or edge device today, but that is a problem for future work.\n","imageSizes":{"/content/papers/assets/f406fe7c1fe33b3cb7b26b45ea0f0feb7cc4d00468788ffa491c6db2144c8da1.jpg":{"width":1612,"height":798},"/content/papers/assets/22bcb4cdd0aacc5ab2c488e46d8477d0bf91af9dfa656ee8af26b5f1bd112aed.jpg":{"width":1589,"height":780},"/content/papers/assets/00f79e12737d98326e1de8237ec710c05d40e51bf6006f306f33a9f4e60ef505.jpg":{"width":1619,"height":788},"/content/papers/assets/31cb9cce12f904494488747a77bd5ecab11a14c97bf29f20bdb0242746c2b844.jpg":{"width":1356,"height":864},"/content/papers/assets/4832b4701f8fcc839d764d08b813e1a0e673cdf61b87f55b14487863cfb1cb5c.jpg":{"width":1357,"height":799},"/content/papers/assets/493de66c40da0f5a6f43f0be7fb353aee5280d637a41cd60fbbc65a9702e4594.jpg":{"width":1166,"height":861},"/content/papers/assets/08afd0a73c5bdba98d28a7e5133ceae3e657ceebb43c4cde8baf77923f0fb5cb.jpg":{"width":1244,"height":410},"/content/papers/assets/68251ed3e2158a9654608fd193ca185bab05b2bd5c2325a3f642f67ee0bbf3f4.jpg":{"width":1393,"height":746},"/content/papers/assets/720d7e158df7595a62d4fb276c6ebf256721cb37dee2afaff436a9e4445e70b2.jpg":{"width":1183,"height":744},"/content/papers/assets/2b59ea22a1f87d3b5ddf9cebec79936c2d0fc3c194c42acbf10f8c67aea928a3.jpg":{"width":1234,"height":762},"/content/papers/assets/22978dc6783a6e858b15e61be795dd3230725a9a8657c8ac1f714766b00f6928.jpg":{"width":1204,"height":380},"/content/papers/assets/025ad8948111c8235d1cad3b2d49ea9406a006d63918b5ebe83174b1f285042b.jpg":{"width":986,"height":825}}}
