Back to paper library
arXiv 2026Image generationPaper notes

Vision Banana

If it can generate images, surely it can understand them

Original paper

Image Generators are Generalist Vision Learners

Paper authors

Valentin Gabeur · Shangbang Long · Songyou Peng …

Show all 25 authors

Valentin Gabeur · Shangbang Long · Songyou Peng · Paul Voigtlaender · Shuyang Sun · Yanan Bao · Karen Truong · Zhicheng Wang · Wenlei Zhou · Jonathan T. Barron · Kyle Genova · Nithish Kannen · Sherry Ben · Yandong Li · Mandy Guo · Suhas Yogin · Yiming Gu · Huizhong Chen · Oliver Wang · Saining Xie · Howard Zhou · Kaiming He · Thomas Funkhouser · Jean-Baptiste Alayrac · Radu Soricut

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2604.20329
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{imagegenerationvisionbanana2026,
  title = {{Image Generators are Generalist Vision Learners}},
  author = {Valentin Gabeur and Shangbang Long and Songyou Peng and Paul Voigtlaender and Shuyang Sun and Yanan Bao and Karen Truong and Zhicheng Wang and Wenlei Zhou and Jonathan T. Barron and Kyle Genova and Nithish Kannen and Sherry Ben and Yandong Li and Mandy Guo and Suhas Yogin and Yiming Gu and Huizhong Chen and Oliver Wang and Saining Xie and Howard Zhou and Kaiming He and Thomas Funkhouser and Jean-Baptiste Alayrac and Radu Soricut},
  year = {2026},
  eprint = {2604.20329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.20329v1}
}
On this page

In the LLM world, we are used to seeing models trained with generative pretraining acquire downstream abilities in language understanding, reasoning, and coding. Those abilities were not trained as separate tasks; they emerged from the objective of predicting the next token.

Could the same story apply to vision? In other words, when a model is trained to generate high-quality images, does it also learn to understand images along the way?

Intro

  • Earlier work had already observed scaling behavior in generative vision pretraining, but its results had continued to lag behind non-generative approaches such as contrastive learning and masked autoencoding.
  • This paper's claim is that, with suitable instruction tuning, a state-of-the-art image generator (Nano Banana Pro here) can become a generalist vision model, reaching or exceeding the level of specialist models on various 2D and 3D understanding tasks.
    • The instruction-tuned model is called Vision Banana.
    • The key design is to express the outputs of every vision task as an RGB image. This lets the model use the abilities its image generator already has, without changing its architecture or adding a new head.

The next two figures show Vision Banana's task overview and benchmark results across these areas. Apart from instance segmentation and image editing, it reaches the best reported results in these comparisons. Let's look more closely at how an image generator handles these vision tasks.

Method

Why Instruction Tuning?

Earlier research had already found that modern image generators can produce pictures that look like segmentation masks or depth maps, even in a zero-shot setting. The trouble is that those pictures cannot be decoded precisely into actual vision outputs, so they cannot be compared quantitatively with other models on benchmarks.

For example, ask the model for a "depth map" and it may give you something that looks like one. But if the relationship between color and distance is unclear, there is no way to evaluate it numerically.

The purpose of instruction tuning, then, is to teach the model to output a reversible visualization in the format specified by the prompt. For example:

"Segment the skateboard category in pure yellow (<255, 255, 0>)"

After the model generates the image, we can cluster pixels near the color (255, 255, 0) to obtain the skateboard mask.

Instruction Tuning Recipe

  • Mix vision-task data into Nano Banana Pro's original training mixture at a very low ratio and perform a lightweight fine-tune.
    • That low ratio matters: it helps the model retain its original generation ability, or what the paper repeatedly calls its "generative prior."
  • Sources of the training data:
    • 2D tasks: internal models annotate images crawled from the web.
    • 3D tasks: rendering engines produce synthetic data.
  • No training split from an evaluation benchmark is used in this instruction tuning. The paper therefore calls the evaluation zero-shot transfer with respect to those benchmarks.

2D Semantic Understanding

Semantic Segmentation

This is the familiar task of assigning a class to every pixel. Vision Banana's approach is straightforward: specify a color for each class in the prompt, and let the model generate the mask image:

  • "Generate a semantic segmentation visualization image, using this color mapping: {"cat": "red", "lock": "pink", "exit sign": "light purple", "background": "yellow"}."

The prompt is remarkably flexible. The color can be a hex code (#80C000), an RGB tuple (<255, 165, 0>), or even a description in natural language ("light purple"); the model can understand all three.

On Cityscapes, it gets 0.699 mIoU, beating SAM 3's 0.652 by 4.7 percentage points. It is the best among the compared zero-shot transfer models, and it is only about 14 percentage points behind the closed-set SegMan (mIoU = 0.842). Those models do not all share the same training setting.

The next figure shows an example of Vision Banana doing semantic segmentation.

Instance Segmentation

Instance segmentation is trickier, because we do not know how many instances there are in advance. We therefore cannot assign a color to every instance ahead of time in the prompt.

Their solution is per-class inference: segment one class at a time and let the model dynamically assign a different color to each instance.

Code
"Generate an instance segmentation visualization of this image.
Each piece of garlic is colored differently."

At evaluation time, they use thresholding to cluster pixels with similar colors.

On SA-Co/Gold, the reported pmF1pmF_1 is 0.540, close to DINO-X's 0.552 but still below SAM 3's 0.661. Vision Banana's figure uses a random subset of 500 queries, and the training settings are different. The authors acknowledge that instance segmentation remains a harder task.

The next figure shows an example of Vision Banana doing instance segmentation.

Referring Expression Segmentation

This task is even more interesting because the model must understand a free-form text query: "the man in a pink T-shirt," "the cat that is stretching," or "the toaster being used as a game controller."

This is precisely where the generative model has an advantage: the multimodal intelligence inherited from pretraining lets it reason about what to segment.

On ReasonSeg, Vision Banana paired with Gemini 2.5 Pro reaches 0.793 gIoU, beating SAM 3 Agent paired with Gemini 2.5 Pro at 0.770 under the paper's zero-shot transfer setting.

Interestingly, Vision Banana can also handle referring expressions within semantic and instance segmentation, even without specific training on free-form queries for those tasks. For example, it understands what "patterns on the wall" and "crescent-shaped croissants" refer to. That suggests cross-task transfer.

The following figure shows examples of Vision Banana understanding and interpreting natural language. It can:

  1. Describe an object's appearance ("man in pink t shirt").
  2. Describe an object's actions ("stretching" and "cleaning").
  3. Understand an unusual use of an object (a toaster as a game controller).
  4. Understand multilingual text content (Chinese and English text on a menu).

3D Understanding

Metric Depth Estimation

This is the part of the paper I find most interesting. Depth estimation differs from segmentation in a fundamental way: depth is an unbounded continuous value d∈[0,∞)d \in [0, \infty), whereas RGB color is a bounded three-dimensional space [0,1]3[0, 1]^3. How can the former be encoded as the latter?

They design a reversible (bijective) mapping between depth and a specified path through RGB space:

Step 1: Use Barron's (2025) power transform to compress unbounded depth into a bounded value:

f(d,λ,c)=1−(1−d/λc)λ+1f(d, \lambda, c) = 1 - (1 - d/\lambda c)^{\lambda+1}

Here λ=−3\lambda = -3 and c=10/3c = 10/3. The transform gives higher resolution to the near field than to the far field, because nearby objects matter more for robotics or AR/VR.

Step 2: Use that normalized distance in [0,1)[0, 1) for piecewise-linear interpolation along the edges of the RGB cube. The path resembles the first iteration of a 3D Hilbert curve, running from black (0,0,0) to white (1,1,1).

The mapping along that path is reversible, so they:

  • turn ground-truth depth into an RGB image with the mapping during training, to use as the training target;
  • project a generated RGB image onto the nearest cube edge during inference, then invert the mapping to recover metric depth.

Across six benchmarks, the average δ1\delta_1 accuracy is 0.882. On the four datasets shared with Depth Anything v3, Vision Banana scores 0.929 versus 0.918, and its depth fine-tuning uses no camera intrinsics or real-world depth training data—the depth data are synthetic.

The next figure shows Vision Banana doing metric depth estimation.

The paper also has an amusing "vibe test": one of the authors took a photograph at Kinkaku-ji. Vision Banana estimated the distance to a green-marked point as 13.71 meters; a measurement with Google Maps gave 12.87 meters, for an AbsRel error of about 0.065.

Surface Normal Estimation

A surface normal is a unit vector (x,y,z)∈[−1,1]3(x, y, z) \in [-1, 1]^3, so its relationship to RGB is much more natural. There is no need for a mapping as elaborate as the one for depth. The model uses camera-space normals directly:

  • Facing Left (−1,0,0)(-1, 0, 0): Pinkish Red
  • Facing Up (0,1,0)(0, 1, 0): Light Green
  • Facing Camera (0,0,1)(0, 0, 1): Light Blue / Purple

When averaged across the three indoor datasets, the reported mean and median angular errors are the lowest among the compared specialist models, including Lotus-2, StableNormal, and DSINE.

The next figure shows an example of Vision Banana doing surface normal estimation.

Retaining Generation Ability

Does instruction tuning damage the model's original image-generation ability?

  • Text-to-image (GenAI-Bench): 53.5% win rate against Nano Banana Pro.
  • Image editing (ImgEdit): 47.8% win rate.

In other words, Vision Banana and the base model are roughly tied on these generation tasks, supporting the claim that instruction tuning was lightweight.

Conclusion

  1. Image generators are themselves generalist vision learners: Strong visual-understanding abilities emerge naturally during generative pretraining. Instruction tuning merely "unlocks" them. This echoes how LLMs changed the field of NLP.
  2. Image generation can serve as a universal interface for vision: Just as text generation serves as a universal interface for NLP tasks, vision outputs (masks, depth, normals) can all be expressed as RGB images. Specify the visualization scheme in the prompt to perform different tasks.
  3. Generative models are naturally equipped to handle ambiguity: A single input can have several reasonable outputs. In segmentation, for example, there can be multiple reasonable ways to decide "which region counts as one instance." Discriminative models need rather hacky losses to avoid collapsing to a blurry mean (the SAM family, for example, produces multiple masks but computes the loss for only one). A generative model already models the whole distribution, so it can handle that ambiguity naturally. That is the paper's motivation, rather than a guarantee that every generated sample is correct.
  4. Future work:
    • Broaden the variety of instruction-tuned tasks, perhaps unlocking more cross-task generalization.
    • Move from monocular input to multi-view and video input.
    • Integrate with LLMs for cross-modality reasoning.
    • Reduce inference cost, because an image generator is far heavier than a lightweight specialist model.

Discussion

Traditionally, people have built a dedicated architecture for each vision task: a mask decoder for segmentation, a regression head for depth, a special loss function for normals. Every task looks different, and models cannot share as much. Vision Banana instead shapes every vision output as an RGB image. The image generator itself becomes a universal interface for vision tasks. This feels like the way GPT unified NLP tasks as text generation, and I think it has real potential.

The metric-depth design in particular—the power transform plus piecewise-linear interpolation along the edges of the RGB cube—is, to me, the cleverest part of the paper. It addresses a major obstacle to using a generation model for vision tasks: what it generates could not previously be decoded precisely back into metric values for a benchmark. Without this invertible mapping, even an attractive depth visualization would remain only qualitative.

The way generative models handle ambiguity is also worth thinking about. To avoid collapsing to a blurry mean, discriminative models have to devise all sorts of hacky losses; the SAM family's practice of outputting multiple masks but computing the loss for just one is a typical example. A generative model already models the entire distribution, so this problem can be handled more naturally.

My Thoughts

After reading this paper, I feel that Vision Banana's contribution lies more in reframing the problem than in a new algorithm or architectural innovation.

It invents no new loss, no new module, and no new architecture. It simply rethinks the output format of vision tasks: with a well-designed reversible RGB encoding and a strong image-generator base model, it can beat various specialists without a specialized architecture.

I think this direction could well become a mainstream path for vision foundation models over the next few years. The visual-understanding abilities emerging from generative pretraining should receive more attention. Paired with a universal output format like RGB, perhaps we really will see an omni model that can perform every vision task someday.

Of course, inference cost remains a practical issue. A model at Vision Banana's scale probably will not be deployed directly on a phone or edge device today, but that is a problem for future work.