Back to paper library
arXiv 2026Object trackingPaper notes

PLANET

Dimensional Strike

Original paper

Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking

Paper authors

Orcun Cetintas · Guillem Brasó · Tim Meinhardt · Laura Leal-Taixé

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2609.00924
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingplanet2026,
  title = {{Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking}},
  author = {Orcun Cetintas and Guillem Brasó and Tim Meinhardt and Laura Leal-Taixé},
  year = {2026},
  eprint = {2609.00924},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.00924}
}
On this page

A camera turns the three-dimensional world into a flat image. Two people standing one behind the other overlap in the image; a person walking toward the camera just has a box that grows a little. Depth, front and back, the true direction of movement: all of it gets squashed in the projection.

All a tracker sees is this plane. Whether it compares appearance, compares box overlap or predicts how boxes move, everything happens on the image plane, so it inherits all of these ambiguities too.

In the past year or two, geometric foundation models like VGGT and Depth Anything 3 have become able to estimate a whole scene's 3D structure and camera parameters straight from an ordinary video. The team behind SUSHI and NOOUGAT (Orcun Cetintas, Guillem Brasó, Tim Meinhardt, Laura Leal-Taixé) wanted to see whether putting this estimated 3D information into a tracker would let it see more clearly.

Defining the problem

Using 3D information for tracking has actually been done before. Some methods estimate how visible pedestrians are in order to reason about occlusion, some convert tracks into world coordinates, and some use 3D human body models. Others add depth constraints to the Kalman filter, or predict tracks on a bird's-eye view.

The authors point out that in these approaches the 3D information is either tied to a specific category, for example usable only for pedestrians, or added only at the association stage, as an extra rule or cost, after the object representation has already formed. The object representation inside the tracker is still only 2D.

The other problem is time. End-to-end query-based trackers usually see only 2 to 10 frames during training, and how far back they can look at inference is limited too; MOTIP, on which PLANET is based, can look back 30 frames, which in an ordinary video is only a second or two. Anyone blocked for longer than that falls outside its memory.

What PLANET sets out to do is bring 3D information in at the moment queries form, and make that information usable over a longer stretch of time.

Solving the problem

First, turn the datasets into 3D

Original Figure 1. Lifting a 2D tracking dataset into 3D. The original dataset has only RGB video and 2D annotations; a visual geometry model produces point maps and camera parameters, from which a 3D label is built for each 2D box. The resulting data contains RGB, 2D boxes, IDs, point maps, camera parameters and 3D labels.
Original Figure 1, Cetintas et al., from PDF page 3, CC BY 4.0.

Datasets like DanceTrack, SportsMOT and BFT have only 2D boxes. The first step is to turn them into 3D.

The authors process each whole video with the long-sequence version of Depth Anything 3: depth and camera parameters are estimated over overlapping time windows, and loop closure keeps drift from building up over long stretches. Once the estimated depth and camera parameters are back-projected, every pixel of every frame corresponds to a point in world coordinates; this is called a point map.

The point map covers the whole scene, with no indication yet of which points belong to which person. For each annotated 2D box, the authors take the average position of the points inside the box as that object's 3D center:

  1. Boxes that aren't blocked at all are used directly.
  2. For a blocked box, which mixes foreground and background, the authors combine depth from neighboring frames, depth in this frame and the box's position in the image to judge conservatively which part is actually the object.
  3. Consistency is checked along each track from frame to frame. Anything still uncertain is marked invalid and not used for training.

These 3D centers give position only, with no object size or orientation; they're estimated "pseudo-labels."

Letting queries see 3D as they form

Original Figure 2. The PLANET architecture. At the top, RGB video passes through the backbone to get 2D features, paired with 2D positional encodings; at the bottom, point maps pass through a position head and a feature head to get geometric position embeddings and geometric features (green, geometric query lifting). Both go together into the Transformer to produce world-grounded queries. Top right is detection: 2D reference points are anchored to 3D reference points, and 2D and 3D prediction heads output boxes and 3D positions respectively (blue, anchored 3D localization). Bottom right is ID prediction: the dual-resolution memory (red) contains a sparse long-term memory and a dense short-term memory.
Original Figure 2, Cetintas et al., from PDF page 4, CC BY 4.0.

PLANET builds on MOTIP. What sets MOTIP apart is that detection and identity prediction share the same set of queries: each query has to output a box and also predict which ID that box belongs to. So as long as the queries contain 3D information, both detection and association benefit.

The authors bring 3D into the queries in two ways.

Geometric query lifting. The point map passes through two small learnable networks. One produces geometric features that are added directly to the backbone's image features; the other produces geometric position embeddings, which are concatenated with the original 2D positional encodings and then fused. When the Transformer updates the queries, both the features and the positions it sees carry the scene's 3D structure.

Anchored 3D localization. Each query in Deformable DETR has a 2D reference point that guides box prediction. The authors lift this 2D reference point through the point map into a corresponding 3D reference point, then have a small 3D prediction head predict a residual from the query to correct it into the object's 3D center. This prediction is supervised by the 3D pseudo-labels built earlier, as an auxiliary task.

The purpose of the auxiliary task is to force the queries to carry the object's position in 3D space with them. If two people overlap in the image but one is in front and the other behind, their queries should carry different 3D positions.

Why "anchor" to the 2D reference point instead of learning a separate 3D reference point? Because the backbone and detector have already learned good localization from large amounts of 2D data. Starting from the 2D reference point, 3D localization only needs to learn a correction, not learn everything from scratch.

Dual-resolution memory: seeing further with the same slots

3D information can help recognize a blocked person again, provided the tracker still remembers that person. PLANET's third design stretches how far back it can look without increasing memory capacity.

The trick is to split the memory in two. One half goes to the most recent consecutive frames, keeping short-term detail; the other half goes to earlier history, taking only one frame every few frames. During training the sampling interval is also varied at random, so the model gets used to seeing a past with uneven gaps.

The paper doesn't give the exact capacity or interval, only that the visible range extends from MOTIP's 30 frames to about 200 frames. Let's get a feel for it with a set of assumed numbers:

ApproachSlotsRecent framesEarlier samplesReach
All consecutive3030—30 frames
Dual-resolution301515, 1 every 12 frames15 + 15 × 12 = 195 frames

Worked example for this note; the capacity of 30 matches MOTIP's 30 frames, and the sampling interval of 12 is an assumed value.

What the memory stores is the detector's final output queries, which already carry 3D information, so this extended memory preserves the 3D cues as well. The change needs no extra training time.

Discussion

What each design contributes

SettingHOTAAssADetAIDF1
MOTIP baseline63.053.275.066.6
+ geometric query lifting64.054.375.967.6
+ anchored 3D localization64.454.976.167.8
+ dual-resolution memory65.656.976.170.3

Compiled from Table 1 of the paper; each setting is the average of 5 training runs.

Geometric query lifting improves both detection and association, adding about 1 point each to DetA and AssA. Anchored 3D localization mainly helps association, adding 0.6 AssA and only 0.2 DetA. Dual-resolution memory works entirely on association: DetA doesn't move, while AssA gains 2 points and IDF1 2.5.

The authors also check how memory and 3D interact. Adding the same dual-resolution memory, a 2D-only model gains 1.5 IDF1, and a model with 3D gains 2.5. 3D cues are more stable, so the longer they're remembered, the more useful they are.

How should 3D go in?

ApproachHOTAAssAIDF1
MOTIP baseline63.053.266.6
Fuse Depth Anything 3 features directly63.153.166.6
World-grounded queries64.454.967.8

Compiled from Table 4 of the paper.

The most intuitive approach is to plug the 3D foundation model's features straight into the tracker. It barely helps: HOTA rises by only 0.1, and AssA and IDF1 don't improve. Those features were trained to estimate geometry, and the tracker doesn't know what to do with them.

Another comparison concerns 3D localization. With the same 3D supervision, if the 3D prediction head uses a separately learned 3D reference point, HOTA actually drops from 64.0 to 63.8; anchored to the 2D reference point, it rises to 64.4.

The 3D information itself is useful, but the tracker can only use it when it has a clear place in query formation and a clear task in the training objective.

Three datasets

DatasetMOTIPPLANETAssA: MOTIP → PLANET
DanceTrack69.672.160.4 → 63.9
SportsMOT72.673.763.2 → 63.2
BFT70.572.671.8 → 73.9

HOTA. DanceTrack compiled from Table 5 of the paper, SportsMOT from Table 6 of the paper, BFT from Table 7 of the paper, all test sets.

It comes first on all three datasets, using only each dataset's own training data and no extra data like CrowdHuman. Some methods in the comparison tables do use extra data.

DanceTrack sees the biggest gain, with association AssA up 3.5 and IDF1 going from 74.7 to 78.0. Dancers dressed alike and constantly crossing paths are exactly the kind of scene that's easiest to confuse on a flat image. On SportsMOT, AssA is 63.2, the same as MOTIP, and the gain is all in detection: DetA goes from 83.5 to 86.0 and MOTA from 92.4 to 95.2.

NOOUGAT, from the same authors a year earlier, has an offline HOTA of 68.4 on DanceTrack and isn't in this paper's comparison tables. The two take different routes: NOOUGAT uses an off-the-shelf detector and focuses on learning association; PLANET is end to end, with detection and association both benefiting from 3D.

What's left unsaid

This paper has only just gone up on arXiv and has no venue yet; the code and the lifted 3D datasets are said to be released only after the paper is accepted.

The paper has no limitations section, and there are a few things readers may want to know. At inference, PLANET needs a point map for every frame, which means running Depth Anything 3 first. The paper reports speeds of 12 frames per second in FP32 and 20 in FP16, comparable to MOTIP, but doesn't say whether that includes the time for this step. The quality of the 3D pseudo-labels also depends on the geometry model; the paper explains how uncertain labels are filtered out but doesn't evaluate how accurate the labels are. Nor does the paper say clearly which split of which dataset the ablations use.

Conclusion

Since a scene's 3D structure can now be estimated from ordinary video, PLANET tries putting it into the tracker.

The approach has three key points. First, lift existing 2D datasets into 3D, getting point maps and a 3D center for each box. Next, bring 3D information in as the queries form, and use an auxiliary task anchored to the 2D reference point to force the queries to remember the object's 3D position. Finally, use dual-resolution memory to stretch how far back it can look to about 200 frames with the same capacity.

The most interesting comparison in the ablations is that plugging in a 3D foundation model's features directly barely helps. 3D information only pays off when it's placed where the tracker will actually use it.