Back to paper library
arXiv 2025IJCV 2026Object trackingPaper notes

NOOUGAT

As Much as You Like

Original paper

NOOUGAT: Towards Unified Online and Offline Multi-Object Tracking

Paper authors

Benjamin Missaoui · Orcun Cetintas · Guillem Brasó · Tim Meinhardt · Laura Leal-Taixé

First submitted to arXiv
arXiv:2509.02111
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multiobjecttrackingnoougat2025,
  title = {{NOOUGAT: Towards Unified Online and Offline Multi-Object Tracking}},
  author = {Benjamin Missaoui and Orcun Cetintas and Guillem Brasó and Tim Meinhardt and Laura Leal-Taixé},
  year = {2025},
  eprint = {2509.02111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.02111}
}
On this page

Trackers have always split into two camps. Online trackers process one frame at a time and can only look at the past, which is what self-driving cars and surveillance cameras need; offline trackers can see the whole video and check back and forth, which suits annotating data and analyzing matches after the fact.

The two camps' methods look completely different, too. On the online side is the SORT and ByteTrack line, where association relies on hand-written rules: match high-score boxes first, then low-score boxes, and decide how much weight appearance and motion each get. On the offline side are graph neural networks like MPNTrack and SUSHI, where association is learned and the results are good, but only fixed-length clips can be processed.

Real applications, though, don't come with just two latency requirements, "right now" and "no limit." Remote surgery can wait two hundred milliseconds; a drone captures only one or two images a second, so it can accumulate dozens of frames and process them together.

The authors of this paper include three of SUSHI's authors and Tim Meinhardt, first author of TrackFormer; the first author is Benjamin Missaoui, and the affiliation listed on the paper is NVIDIA for all of them. They wanted to build a tracker that can wait as long as you like: wait one frame and it's online tracking, wait two hundred-odd frames and it's offline tracking, with any length in between. They named it NOOUGAT, which sounds just like nougat.

Defining the problem

Original Fig. 1. Three kinds of tracker. (a) Online tracker: past tracks and this frame's detections are matched with hand-written rules. (b) Offline tracker: the video is cut into overlapping clips, each yielding local tracks, which are then stitched together with hand-written rules. (c) This paper: past tracks and an incoming segment of tracklets are matched and stitched in a learned way, with online and offline both using the same architecture.
Original Fig. 1, Missaoui et al., from PDF page 2, arXiv license, scholarly quotation.

The authors point out each camp's shortcomings.

Online trackers rely on rules for association, and the rules' parameters have to be tuned per dataset. HybridSORT's appearance weight is 4.0 on DanceTrack, 1.3 on MOT17 and 4.6 on MOT20, and it changes further depending on detection confidence. These rules also struggle with long occlusions; most methods give up on a track once it has been broken for 30 frames.

Offline trackers' graph networks are strong, but they assume a video can be processed in one go. When a video is too long, it has to be cut into overlapping clips, each processed separately and then stitched together in the overlaps with the Hungarian algorithm or linear programming. That step falls back on rules, and the overlapping parts get computed several times.

A quick calculation shows how much is repeated. For a 1,024-frame video:

MethodClip lengthStrideClipsFrames processed
MPNTrack1511,01015,150
SUSHI51225631,536
NOOUGAT25625641,024

Worked example for this note; the MPNTrack and SUSHI settings are taken from Section 4.1 of the paper.

MPNTrack computes each frame almost 15 times, SUSHI 1.5 times, and NOOUGAT just once.

Solving the problem

Cut into non-overlapping segments, joined one by one

Original Fig. 3. The NOOUGAT architecture. On the right, the newly arrived frames pass through the GNN hierarchy (graph networks stacked level by level) to get this segment's tracklets. On the left are the past tracks already worked out. The ALT layer joins the two to get global tracks. The ALT layer and the GNN hierarchy share weights.
Original Fig. 3, Missaoui et al., from PDF page 5, arXiv license, scholarly quotation.

NOOUGAT cuts the video into non-overlapping sub-clips of TT frames each. Each segment is first processed with SUSHI's GNN hierarchy to get the tracklets within that segment.

Next, a new ALT layer (autoregressive long-term tracking) joins "the tracks so far" with "the tracklets of the newly arrived segment":

Talti=ALT(Talti−1, Thicli)\mathcal{T}^{i}_{\text{alt}} = \text{ALT}\left(\mathcal{T}^{i-1}_{\text{alt}},\ \mathcal{T}^{i}_{\text{hicl}}\right)

The joined result is then joined with the next segment, and so it rolls on.

ALT itself is also a graph network layer. Each past track is a node, each new tracklet is a node too, and each past track is connected to its 10 nearest new tracklets. As in each level of SUSHI, the network judges whether each edge links the same person, then rounds the result into matches.

Stitching thus moves from hand-written rules to a learned graph network. ALT also shares weights with the GNN hierarchy, and the authors found this made training more than twice as fast, converging in about 40,000 iterations instead of 90,000.

One parameter, from online to offline

TT is NOOUGAT's most important parameter.

With T=1T = 1, each segment is a single frame, the GNN hierarchy has nothing to do, and ALT joins the past tracks directly to this frame's detections: a frame-by-frame online tracker. In this case the authors also restrict the graph to a bipartite graph: edges exist only between past tracks and new detections, and decisions already made in the past can't be changed, in keeping with the rules of online tracking.

The larger TT is, the longer the stretch of time the GNN hierarchy can see within each segment and the more confident its judgments, at the cost of waiting for more frames before producing output. The offline setting uses T=256T = 256 and allows past associations to be corrected.

For 30 FPS video, every extra frame of waiting adds 33 milliseconds of latency. The authors match this against several applications:

ApplicationAcceptable latencyMatching TT
Autonomous drivingUnder 100 ms1–2
SurveillanceAbout 125 ms3–4
Remote surgery200 ms best, 300 ms max6–9
Aerial trackingCapture is only 1–2 FPS15–30
Annotation, sports analysisNo limitAny

Compiled from Section 5.4 of the paper.

Original Fig. 2. The effect of the number of incoming frames T on IDF1, on the DanceTrack validation set. As T grows from 1 to 256, IDF1 rises steadily from about 64 to about 77; the small icons along the way stand for self-driving cars, surveillance cameras, surgical robots, drones, motion capture, sports analysis and offline processing.
Original Fig. 2, Missaoui et al., from PDF page 3, arXiv license, scholarly quotation.

With the same model architecture, waiting a little longer buys a little more accuracy. As TT goes from 1 to 8, IDF1 rises by about 8 points; after that the gains slow, reaching about 77 at 256 frames.

Features and training

The input features follow SUSHI's geometry, motion and appearance, plus two things online trackers commonly use. One is OC-SORT's velocity direction consistency, which originally compares a track with a detection box; the authors extend it to compare two tracklets: the direction of travel over the last 12 boxes of the earlier one, and the direction inferred backward from the first 12 boxes of the later one. The smaller the angle between them, the more likely they're the same person. The other is detection confidence, added to the nodes' initial features.

The difficulty in training ALT is that the "past tracks" it has to handle at inference come in endless variety: long and short, broken and continuous. During training, the authors randomly cut a stretch of past ground-truth tracks as input, with random length and starting point, then randomly delete some tracks and boxes; they also deliberately leave a gap of a few frames between the past and the incoming segment to simulate brief occlusions.

Discussion

Test sets

DatasetBest online rivalRival AssAOnline AssAOffline AssA
DanceTrackHybrid-SORT52.654.958.7
SportsMOTDiffMOT65.174.383.0
MOT17DiffMOT64.665.368.5
MOT20BoT-SORT62.967.570.7

DanceTrack compiled from Table 1 of the paper, SportsMOT from Table 2 of the paper, MOT17 from Table 3 of the paper, MOT20 from Table 4 of the paper; all test sets. The rival is the online method in each table with the highest AssA among those using the same detections as NOOUGAT.

The methods compared all use ByteTrack's YOLOX detections. The online version has the highest association score on all four datasets, and the offline version adds another 3 to 9 points. SportsMOT improves the most: players often run out of view and come back, which is exactly where long-term association shines.

On MOT17 and MOT20, the offline version doesn't necessarily have the highest HOTA. On MOT17, CoNo-Link's HOTA is 67.1 and NOOUGAT's is 66.9; the authors' explanation is that CoNo-Link has stronger detections, with MOTA 2 points higher, while NOOUGAT wins on AssA and IDF1.

How much better is learned than hand-written?

The authors run two direct comparisons.

  1. Matching. In the online setting, ALT is replaced by the Hungarian algorithm, with a cost that combines appearance, motion and velocity direction using HybridSORT's weights, plus a grid search around them. On the DanceTrack validation set, HOTA drops from 64.6 to 62.1 and IDF1 from 69.0 to 65.8.
  2. Stitching. In the offline setting, SUSHI is retrained with the same features and ReID as NOOUGAT, keeping its overlapping clips and Hungarian stitching. SUSHI's IDF1 is 72.0; NOOUGAT's is 76.6.

The second comparison is especially convincing, because the two graph networks are nearly identical and the only difference is whether the stitching is rule-based or learned.

How long can a break be and still be rejoined?

Original Fig. 5. The effect of track retention time on IDF1, on the SportsMOT validation set. The upper half shows the NOOUGAT online model (orange line) and the upper bound with perfect association (red line), with retention from 2 to 256 frames; the dashed line marks the 30 frames commonly used by SORT-family methods. The lower half is the distribution of occlusion lengths: 105 occlusions of 32 to 64 frames, 120 of 64 to 128 frames, and 78 of 128 to 256 frames.
Original Fig. 5, Missaoui et al., from PDF page 11, arXiv license, scholarly quotation.

In SportsMOT, occlusions longer than 30 frames are the bulk of the cases. In the distribution in the lower half, occlusions over 32 frames add up to 303, far more than the 66 shorter than 32 frames.

SORT-family trackers usually drop a track after 30 frames. With NOOUGAT, the longer tracks are retained, the higher IDF1 goes, reaching about 92 at 256 frames of retention. There's no rule in between designed specially for long occlusions; it's all learned by ALT on its own.

What each feature does

With geometric features alone, HOTA is 59.1; with appearance alone, it's actually only 57.9; geometry plus motion gives 60.4; everything together gives 64.6. DanceTrack's dancers dress alike, so appearance on its own isn't very reliable, but combined with other cues it helps resolve ambiguous cases.

Speed and scale

On a V100, with detections and ReID precomputed, the online version runs at only 12 FPS while the offline version manages 340 FPS. The reason is that the online version has to build a new graph every frame, and graph construction alone takes nearly half the time; the offline version builds only 8 graphs per 256-frame segment. The authors admit the online implementation still has a lot of room for optimization. The paper also estimates the speed under ideal conditions (oracle runtime), which already reaches 37 FPS at T=4T = 4.

On the most crowded sequence, MOT20-05 (1,154 people, 610,000 detection boxes), the offline version still runs at 74 FPS. On the DanceTrack validation set, both the online and the offline (T=256T = 256) versions use less than 6 GB of GPU memory.

The online model has only 27,000 trainable parameters; MeMOTR has 51 million. NOOUGAT handles only association and leaves detection to an off-the-shelf detector, so it isn't at a disadvantage even on datasets like MOT17 with very little training data.

Other domains

The authors also test on three more datasets. On BEE24, a bee-tracking dataset with videos up to five thousand frames long, NOOUGAT's HOTA is 4.5 higher than the next-best method. VETRA, aerial footage of vehicles, has only one frame per second, and cars move a long way between adjacent frames; NOOUGAT beats BoT-SORT by 17.2 HOTA, but still trails Deep SR-SORT, which was designed specifically for this dataset and uses priors on vehicle speed and size (82.2 vs. 68.0).

With MOT17's public detections, the advantage is much smaller: the offline version's IDF1 is 72.1, SUSHI's 71.5. The authors suspect that when the detection boxes are too noisy, ALT has a harder time learning.

Conclusion

What NOOUGAT does is turn SUSHI's offline graph network into a tracker that accepts any latency. The video is cut into non-overlapping segments, each segment is processed by a graph network, and the segments are joined by another graph network layer. Stitching in offline methods and matching in online methods are both replaced by learned association.

Its value is that just by adjusting TT, you can pick a point between latency and accuracy. Wait one frame, and it's an online tracker leading on association scores; wait two hundred-odd frames, and it's an offline tracker; and every TT in between is the same model.