STORM: End-to-End Referring Multi-Object Tracking in Videos
- Publication
- CVPR Findings 2026Conference
- First submitted to arXiv
- arXiv:2604.10527
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingstorm2026,
title = {{STORM: End-to-End Referring Multi-Object Tracking in Videos}},
author = {Zijia Lu and Jingru Yi and Jue Wang and Yuxiao Chen and Junwen Chen and Xinyu Li and Davide Modolo},
year = {2026},
eprint = {2604.10527},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2604.10527}
}
Ordinary multi-object tracking follows every person, or every car, in the frame. In practice, we often care about only a few of them: "the person in sunglasses holding a laptop," "the two kids on the left," "the skier and the skis beneath him."
Specifying what to track with a sentence is a task called referring multi-object tracking (RMOT). The model first has to understand the description, find the objects that match it, and then follow them all the way, one more layer than ordinary tracking.
Zijia Lu, Davide Modolo and colleagues at Amazon argue that understanding descriptions is exactly what large language models are good at. They built STORM, a multimodal large language model that reads in a video and a description and directly writes out, as text, each object's box in each frame.
Defining the problem
Existing approaches are pieced together
Past RMOT methods mostly add a text encoder to a specialized detector, so that detection results are filtered by the description. This can handle "the red car," but not descriptions involving relationships, like "the two performers to the right of the painting."
Methods that use multimodal large language models for grounding can understand complex language, but most handle only single images. To use them on video, you have to attach a separate tracker or detector, with the language model only responsible for understanding the text and not directly involved in "where this box goes in the next frame." The authors point out that, so far, no method handles RMOT end to end.
Data that is scarce and messy
The bigger problem is data. Multimodal large language models need large amounts of high-quality training data, but RMOT annotation is extremely expensive: every frame needs boxes, and the boxes have to line up with a precise description.
Existing datasets each have their flaws. Refer-KITTI has only traffic scenes and two categories; LaSOT's descriptions are often not specific enough, so a video may contain several animals of the same kind while the description doesn't say which one; Elysium is large, but quite a few of its boxes don't match their descriptions; LaMOT is small, and many of its descriptions are just category names.
Solving the problem
Writing out every box as text

STORM's architecture is a standard LLaVA-style design: a vision encoder turns each frame into visual tokens, a two-layer MLP projects them into the text space, and they go together with the description into a language model with 8 billion parameters. The vision encoder is NaViT, which can handle any resolution and aspect ratio without forcing the frame down to a fixed size, an advantage for tasks that need precise localization.
The model's output is simply a stretch of text in the format "Object 1: Frame 1: [x1, y1, x2, y2], Frame 2: …; Object 2: …", one section per object, writing out its box in each frame in order. Coordinates are absolute pixel positions in the frame. When an object is temporarily blocked or leaves the frame, an empty box is output. Training is ordinary next-token prediction with a cross-entropy loss. The whole process has no detector, no tracker and no Hungarian algorithm; identity consistency is maintained by the language model itself: the boxes written under "Object 1" are the same object.
When a video is too long, it's cut into several segments, and the boxes last predicted in one segment serve as the prompt for the next, linking the tracks together.
Learning the task in pieces
RMOT data is too scarce, so the authors' solution is to break it into two data-rich subtasks: grounding by description and tracking.
- Stage one. With large amounts of image grounding data, the model learns to find objects in a single image from a description; with single-object tracking data, it learns to follow one object frame by frame; and a little referring single-object tracking data joins the two abilities together.
- Stage two. Fine-tuning on STORM-Bench, which the authors built themselves. During training, the model is made to write out its reasoning first: find all the described objects in the first frame, then track them in order. This explicitly teaches the model the order of the two subtasks.
The authors call this approach task-composition learning (TCL).
A bottom-up dataset

Earlier large datasets were mostly produced top-down: first write a description for the whole video, then use Grounding-DINO to find the nouns in the description, then use a tracker to chain the boxes together. Every step depends on an expert model, and an error at any step stays in the data.
STORM-Bench does it the other way round. The authors exploit an asymmetry: finding objects from a description is hard, but given an object, writing a description of it is relatively easy. The data is built on VidOR, a dataset that already has ground-truth boxes for everyday objects.
- Write a description for each object. Draw a red box around the target object in the video and ask Qwen2.5-VL to describe it. Then ask another model to verify whether the description corresponds to this one object only, with three different inputs: the original video with the red box, the video blurred outside the box, and the object cropped out on its own. Only descriptions that pass all three are kept.
- Compose multi-object descriptions. Hand the descriptions of every object in the video to a language model good at reasoning; for a few randomly chosen objects, it finds the attributes they share or that set them apart, and writes a single description, such as "the instruments on the shelf" or "the girl and the cat running behind her." A multimodal model then checks it once more against the video and boxes.
The result is 15,000 training videos and 714 test videos, with 200,000 descriptions and 73,700 object tracks across 80 categories. Each description refers to 3 objects on average and averages 12.2 English words, far longer than LaMOT's 3.8 and Elysium's 2.1.
Discussion
Referring multi-object tracking
| Method | HOTA | AssA | IDF1 | ID sw. |
|---|---|---|---|---|
| Grounding DINO + OC-SORT | 31.7 | 56.5 | 27.3 | 2,953 |
| Qwen2.5-VL + OC-SORT | 37.9 | 62.6 | 38.6 | 2,767 |
| VisionLLMv2 + OC-SORT | 45.3 | 58.9 | 44.6 | 12,097 |
| LaMOT | 46.7 | 56.1 | 48.8 | 7,972 |
| STORM (STORM-Bench only) | 34.8 | 56.8 | 36.4 | 2,832 |
| STORM | 66.7 | 82.6 | 78.3 | 183 |
Compiled from Table 4 of the paper, STORM-Bench test set.
With no end-to-end method to compare against directly, the authors have several models capable of per-frame grounding each produce boxes, then chain them into tracks with OC-SORT; LaMOT uses its official weights, fine-tuned on STORM-Bench.
STORM's HOTA is 66.7, 20 points above the best, LaMOT; it has only 183 ID switches against LaMOT's 7,972. This isn't an entirely fair comparison: STORM-Bench is the authors' own dataset, and its annotations come from the same source as STORM's training data. But a gap of more than forty times in ID switches is hard to explain by that alone. Methods that ground each frame and then chain the results can pick the wrong person in any frame; STORM writes the whole track under the same "Object 1," so identities don't swap easily in the first place.
In the same table, STORM trained only on STORM-Bench gets a HOTA of just 34.8, lower even than Qwen2.5-VL plus OC-SORT; add the stage-one grounding and tracking pretraining, and it jumps to 66.7. The architecture alone isn't enough; learning the task in pieces is the main reason.

This figure shows how flexible the descriptions can be. Same video, same model, and the description can refer to an object's position (the chairs on the left), relations between objects (the two performers to the right of the painting), possession (the two instruments of the player on the left), or mix different categories (the player and the painting).
What learning in pieces buys
| Image grounding | SOT | Referring SOT | Referring SOT score |
|---|---|---|---|
| 1M | 0 | 100K | 77.4 |
| 1M | 0 | 1M | 84.1 |
| 1M | 900K | 100K | 83.0 |
Compiled from Table 5 of the paper; data amounts are numbers of training samples, and the score is the test result for referring single-object tracking (SOT).
Referring single-object tracking data is more expensive to annotate than ordinary single-object tracking data. This table shows that 900,000 samples of ordinary tracking data plus 100,000 referring samples come close to the score of 1 million referring samples. Cheap subtask data can stand in for expensive full-task data.
The amount of RMOT data itself was also tested: 5,000 samples give HOTA 26.4, 10,000 give 53.4, 12,000 give 63.3 and 15,000 give 66.7. Gains start to level off after about 12,000, so the authors stopped the dataset at 15,000.
Image grounding and single-object tracking
Although STORM was designed for video, it's still competitive at image grounding. On RefCOCO's validation set and testA, RefCOCO+'s testA and RefCOCOg's test set, it's the highest in the table; in the remaining columns other models are slightly ahead, for example on RefCOCO's testB, where MiniGPT-v2 gets 85.3 and STORM 83.8.
On Elysium's single-object tracking, given the first-frame box, STORM's AUC is 89.8 against Elysium's 88.7; when the target is specified with a short description, it's 84.1 against 83.3. After the authors rewrote the descriptions to be longer and more specific, the two have nearly identical AUC (87.4 vs. 87.5), with STORM slightly ahead on precision.
Speed
Writing out coordinates one by one as text comes at the cost of speed. In the appendix, when tracking a single object, the speed is around 3.3 frames per second whether the video is 32 frames or 256. But with more targets it slows down:
| Targets | Boxes to write | Time | Per box |
|---|---|---|---|
| 1 | 64 | 19.7 s | 0.31 s |
| 8 | 512 | 143.3 s | 0.28 s |
Times compiled from Table 7 in the paper's appendix (64-frame video); per-box times computed for this note.
The time spent is almost proportional to the number of boxes to write, about 0.3 seconds per box. Tracking 8 targets, a 64-frame video takes over two minutes, which works out to less than half a frame per second.
The authors also mention in the appendix that with very long videos, or more than ten targets, the output text sequence gets too long and the model struggles. The remaining errors are mainly in localization: when the description is vague or the scene is crowded, the boxes are off.
Conclusion
STORM hands referring multi-object tracking entirely to a multimodal large language model: read the video, read the description, and write out each object's box in each frame as text. Identity is maintained by the structure of the output, so ID switches are astonishingly rare.
What really makes it work is the training. With too little data for the full task, it first uses large amounts of image grounding and single-object tracking data to learn the two abilities separately, then uses a small amount of full-task data to join them up. The self-built STORM-Bench takes an "object first, description after" approach, sidestepping the errors that pile up layer by layer in top-down annotation.
It's just that every box has to be written out as text, one at a time, so the more things there are to track, the longer the writing takes.