NOOUGAT: Towards Unified Online and Offline Multi-Object Tracking
- Publication
- International Journal of Computer Vision 2026Journal
- First submitted to arXiv
- arXiv:2509.02111
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multiobjecttrackingnoougat2025,
title = {{NOOUGAT: Towards Unified Online and Offline Multi-Object Tracking}},
author = {Benjamin Missaoui and Orcun Cetintas and Guillem Brasó and Tim Meinhardt and Laura Leal-Taixé},
year = {2025},
eprint = {2509.02111},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2509.02111}
}
Trackers have always split into two camps. Online trackers process one frame at a time and can only look at the past, which is what self-driving cars and surveillance cameras need; offline trackers can see the whole video and check back and forth, which suits annotating data and analyzing matches after the fact.
The two camps' methods look completely different, too. On the online side is the SORT and ByteTrack line, where association relies on hand-written rules: match high-score boxes first, then low-score boxes, and decide how much weight appearance and motion each get. On the offline side are graph neural networks like MPNTrack and SUSHI, where association is learned and the results are good, but only fixed-length clips can be processed.
Real applications, though, don't come with just two latency requirements, "right now" and "no limit." Remote surgery can wait two hundred milliseconds; a drone captures only one or two images a second, so it can accumulate dozens of frames and process them together.
The authors of this paper include three of SUSHI's authors and Tim Meinhardt, first author of TrackFormer; the first author is Benjamin Missaoui, and the affiliation listed on the paper is NVIDIA for all of them. They wanted to build a tracker that can wait as long as you like: wait one frame and it's online tracking, wait two hundred-odd frames and it's offline tracking, with any length in between. They named it NOOUGAT, which sounds just like nougat.
Defining the problem

The authors point out each camp's shortcomings.
Online trackers rely on rules for association, and the rules' parameters have to be tuned per dataset. HybridSORT's appearance weight is 4.0 on DanceTrack, 1.3 on MOT17 and 4.6 on MOT20, and it changes further depending on detection confidence. These rules also struggle with long occlusions; most methods give up on a track once it has been broken for 30 frames.
Offline trackers' graph networks are strong, but they assume a video can be processed in one go. When a video is too long, it has to be cut into overlapping clips, each processed separately and then stitched together in the overlaps with the Hungarian algorithm or linear programming. That step falls back on rules, and the overlapping parts get computed several times.
A quick calculation shows how much is repeated. For a 1,024-frame video:
| Method | Clip length | Stride | Clips | Frames processed |
|---|---|---|---|---|
| MPNTrack | 15 | 1 | 1,010 | 15,150 |
| SUSHI | 512 | 256 | 3 | 1,536 |
| NOOUGAT | 256 | 256 | 4 | 1,024 |
Worked example for this note; the MPNTrack and SUSHI settings are taken from Section 4.1 of the paper.
MPNTrack computes each frame almost 15 times, SUSHI 1.5 times, and NOOUGAT just once.
Solving the problem
Cut into non-overlapping segments, joined one by one

NOOUGAT cuts the video into non-overlapping sub-clips of frames each. Each segment is first processed with SUSHI's GNN hierarchy to get the tracklets within that segment.
Next, a new ALT layer (autoregressive long-term tracking) joins "the tracks so far" with "the tracklets of the newly arrived segment":
The joined result is then joined with the next segment, and so it rolls on.
ALT itself is also a graph network layer. Each past track is a node, each new tracklet is a node too, and each past track is connected to its 10 nearest new tracklets. As in each level of SUSHI, the network judges whether each edge links the same person, then rounds the result into matches.
Stitching thus moves from hand-written rules to a learned graph network. ALT also shares weights with the GNN hierarchy, and the authors found this made training more than twice as fast, converging in about 40,000 iterations instead of 90,000.
One parameter, from online to offline
is NOOUGAT's most important parameter.
With , each segment is a single frame, the GNN hierarchy has nothing to do, and ALT joins the past tracks directly to this frame's detections: a frame-by-frame online tracker. In this case the authors also restrict the graph to a bipartite graph: edges exist only between past tracks and new detections, and decisions already made in the past can't be changed, in keeping with the rules of online tracking.
The larger is, the longer the stretch of time the GNN hierarchy can see within each segment and the more confident its judgments, at the cost of waiting for more frames before producing output. The offline setting uses and allows past associations to be corrected.
For 30 FPS video, every extra frame of waiting adds 33 milliseconds of latency. The authors match this against several applications:
| Application | Acceptable latency | Matching |
|---|---|---|
| Autonomous driving | Under 100 ms | 1–2 |
| Surveillance | About 125 ms | 3–4 |
| Remote surgery | 200 ms best, 300 ms max | 6–9 |
| Aerial tracking | Capture is only 1–2 FPS | 15–30 |
| Annotation, sports analysis | No limit | Any |
Compiled from Section 5.4 of the paper.

With the same model architecture, waiting a little longer buys a little more accuracy. As goes from 1 to 8, IDF1 rises by about 8 points; after that the gains slow, reaching about 77 at 256 frames.
Features and training
The input features follow SUSHI's geometry, motion and appearance, plus two things online trackers commonly use. One is OC-SORT's velocity direction consistency, which originally compares a track with a detection box; the authors extend it to compare two tracklets: the direction of travel over the last 12 boxes of the earlier one, and the direction inferred backward from the first 12 boxes of the later one. The smaller the angle between them, the more likely they're the same person. The other is detection confidence, added to the nodes' initial features.
The difficulty in training ALT is that the "past tracks" it has to handle at inference come in endless variety: long and short, broken and continuous. During training, the authors randomly cut a stretch of past ground-truth tracks as input, with random length and starting point, then randomly delete some tracks and boxes; they also deliberately leave a gap of a few frames between the past and the incoming segment to simulate brief occlusions.
Discussion
Test sets
| Dataset | Best online rival | Rival AssA | Online AssA | Offline AssA |
|---|---|---|---|---|
| DanceTrack | Hybrid-SORT | 52.6 | 54.9 | 58.7 |
| SportsMOT | DiffMOT | 65.1 | 74.3 | 83.0 |
| MOT17 | DiffMOT | 64.6 | 65.3 | 68.5 |
| MOT20 | BoT-SORT | 62.9 | 67.5 | 70.7 |
DanceTrack compiled from Table 1 of the paper, SportsMOT from Table 2 of the paper, MOT17 from Table 3 of the paper, MOT20 from Table 4 of the paper; all test sets. The rival is the online method in each table with the highest AssA among those using the same detections as NOOUGAT.
The methods compared all use ByteTrack's YOLOX detections. The online version has the highest association score on all four datasets, and the offline version adds another 3 to 9 points. SportsMOT improves the most: players often run out of view and come back, which is exactly where long-term association shines.
On MOT17 and MOT20, the offline version doesn't necessarily have the highest HOTA. On MOT17, CoNo-Link's HOTA is 67.1 and NOOUGAT's is 66.9; the authors' explanation is that CoNo-Link has stronger detections, with MOTA 2 points higher, while NOOUGAT wins on AssA and IDF1.
How much better is learned than hand-written?
The authors run two direct comparisons.
- Matching. In the online setting, ALT is replaced by the Hungarian algorithm, with a cost that combines appearance, motion and velocity direction using HybridSORT's weights, plus a grid search around them. On the DanceTrack validation set, HOTA drops from 64.6 to 62.1 and IDF1 from 69.0 to 65.8.
- Stitching. In the offline setting, SUSHI is retrained with the same features and ReID as NOOUGAT, keeping its overlapping clips and Hungarian stitching. SUSHI's IDF1 is 72.0; NOOUGAT's is 76.6.
The second comparison is especially convincing, because the two graph networks are nearly identical and the only difference is whether the stitching is rule-based or learned.
How long can a break be and still be rejoined?

In SportsMOT, occlusions longer than 30 frames are the bulk of the cases. In the distribution in the lower half, occlusions over 32 frames add up to 303, far more than the 66 shorter than 32 frames.
SORT-family trackers usually drop a track after 30 frames. With NOOUGAT, the longer tracks are retained, the higher IDF1 goes, reaching about 92 at 256 frames of retention. There's no rule in between designed specially for long occlusions; it's all learned by ALT on its own.
What each feature does
With geometric features alone, HOTA is 59.1; with appearance alone, it's actually only 57.9; geometry plus motion gives 60.4; everything together gives 64.6. DanceTrack's dancers dress alike, so appearance on its own isn't very reliable, but combined with other cues it helps resolve ambiguous cases.
Speed and scale
On a V100, with detections and ReID precomputed, the online version runs at only 12 FPS while the offline version manages 340 FPS. The reason is that the online version has to build a new graph every frame, and graph construction alone takes nearly half the time; the offline version builds only 8 graphs per 256-frame segment. The authors admit the online implementation still has a lot of room for optimization. The paper also estimates the speed under ideal conditions (oracle runtime), which already reaches 37 FPS at .
On the most crowded sequence, MOT20-05 (1,154 people, 610,000 detection boxes), the offline version still runs at 74 FPS. On the DanceTrack validation set, both the online and the offline () versions use less than 6 GB of GPU memory.
The online model has only 27,000 trainable parameters; MeMOTR has 51 million. NOOUGAT handles only association and leaves detection to an off-the-shelf detector, so it isn't at a disadvantage even on datasets like MOT17 with very little training data.
Other domains
The authors also test on three more datasets. On BEE24, a bee-tracking dataset with videos up to five thousand frames long, NOOUGAT's HOTA is 4.5 higher than the next-best method. VETRA, aerial footage of vehicles, has only one frame per second, and cars move a long way between adjacent frames; NOOUGAT beats BoT-SORT by 17.2 HOTA, but still trails Deep SR-SORT, which was designed specifically for this dataset and uses priors on vehicle speed and size (82.2 vs. 68.0).
With MOT17's public detections, the advantage is much smaller: the offline version's IDF1 is 72.1, SUSHI's 71.5. The authors suspect that when the detection boxes are too noisy, ALT has a harder time learning.
Conclusion
What NOOUGAT does is turn SUSHI's offline graph network into a tracker that accepts any latency. The video is cut into non-overlapping segments, each segment is processed by a graph network, and the segments are joined by another graph network layer. Stitching in offline methods and matching in online methods are both replaced by learned association.
Its value is that just by adjusting , you can pick a point between latency and accuracy. Wait one frame, and it's an online tracker leading on association scores; wait two hundred-odd frames, and it's an offline tracker; and every in between is the same model.