2nd Workshop on Tracking and Its Many Guises: Tracking Any Object in Open-World

Over the course of its rich history, object tracking has been tackled under many disguises: multi-object tracking, single-object tracking, video object segmentation, video instance segmentation, and more. A significant majority of these disguises are closed-world benchmarks where the methods are expected to detect and track a predefined set of frequently observed classes. However, an intelligent system should go beyond a closed-world setting: It should be able to detect and track objects that have never been seen before. To this end, we have introduced TAO-OW: Tracking Any Object in an Open World benchmark for measuring tracking performance in an open-world setting where all objects must be tracked. Methods are specifically evaluated on how well they can track object classes that were missing from the training set (unknown objects), as well as objects which were in the training set (known objects).

Competition

We are excited to announce two Multi-Object Tracking (MOT) competitions: the Long-Tail Challenge and the Open-World Challenge. With these challenges, we aim to advance multi-object tracking and segmentation research in challenging few-shot and open-world conditions.

We base our challenges on TAO (Tracking Any Object) dataset and BURST (Benchmark for Unifying Object Recognition, Segmentation, and Tracking) video segmentation labels. We provide 2,914 videos with pixel-precise labels for 16,089 unique object tracks (600,000 per-frame masks) spanning 482 object classes!

In the Long-Tail Challenge, we focus on tracking and classifying all objects within the TAO/BURST object class vocabulary. In the Open-World Challenge, we investigate multi-object tracking and segmentation in a setting where labels for only a subset of target classes are available during model training. All objects need to be tracked but not classified.

In summary, the Long-Tail and Open-World Challenges offer a unique opportunity for researchers to investigate how far we can get with object tracking in long-tailed and open-world regimes and advance the field.

The submission deadline for both challenges is June 5th, 2023. Participants can submit their results through the MOTChallenge platform. Winners will be invited to present their work at our workshop.

Important Information

• Challenge closes: June 5th, 2023
• Abstract submission deadline: June 6th, 2023
• Technical report deadline: June 10th, 2023
• Each participant of the challenge should send an abstract (max. 1200 characters) to Idil Esen Zulfikar (zulfikar@vision.rwth-aachen.de) by June 6th to publish a short paper or to present their method in the workshop.

General Challenge Information/Rules

• Deadline for challenge submission is June 5th, 2023 23:59 CEST.
• We are using MOTChallenge platform for results submission. Leaderboards are available at TAO Long-Tail and TAO Open-World .
• The rules of the MOTChallenge platform must be respected. See MOTChallenge rules and instructions.
• We will announce the (i) best-performing method in each track and (ii) the most innovative approach. We will invite the authors of top-performing and most-innovative methods to present their work at our workshop.
• For any questions, please contact Idil Esen Zulfikar (zulfikar@vision.rwth-aachen.de) or Ali Athar (athar@vision.rwth-aachen.de).

Important Links

• Data: TAO Dataset with TAO Pixel-Precise Segmentation Mask Annotations.
• Challenge Page: Please see the challenge page for more details.
• Baselines: Open-World Tracking Baseline (OWTB) (see paper) code can be found here, and STCN baseline (see paper) code can be found here.
• Pre-computed Object Detections: Please download pre-computed object proposals for all sequences from here for the TAO Open-World challenge, and pre-computed object proposals for all sequences from here for the TAO Long-Tail challenge.

Challenge 1: Long-Tail Tracking Challenge (Long-Tail)

In the Long-Tail Tracking Challenge we ask to track and classify all objects specified in the TAO/BURST object class vocabulary. Models can leverage labeled data for all 482 semantic classes during training. The challenge emphasizes the long-tail distribution of object classes, with a few classes occurring frequently and the majority occurring rarely. Participants are expected to develop methods that can handle long-tail distribution and are robust to highly imbalanced datasets.

The challenge’s goal is to advance the state-of-the-art in multi-object tracking and segmentation. Participants are encouraged to use creative and innovative approaches to achieve the highest possible performance on this challenging dataset.

Important:

• In addition to the benchmark provided on the MOTChallenge platform, the challenge organizers will evaluate submissions to identify the best-performing method and the most innovative approach. Winners will be invited to present their work at a workshop.
• To ensure fairness and prevent overfitting, participants must submit their code for the challenge organizers to review. The code will be used to verify that no training or tuning was performed on the test set, and that the results truly represent the performance of the submitted methods.
• A benchmark with the current results is available here.

Challenge 2: Open-World Tracking Challenge (Open-World)

The Open-World Challenge focuses on multi-object tracking and segmentation in a setting where only a limited number of labeled classes are available during training (see Opening up Open-World Tracking Paper). This is a challenging problem, as methods need to track all objects, including those not presented as labeled instances, during the model training.

Unlike the Long-Tail Challenge, in the Open-World Challenge, we (i) limit the number of labeled classes used for model training (i.e., only labels for classes within COCO class vocabulary can be used), and (ii) do not require classifying tracked object instances.

Important:

• This challenge’s rules limit the use of labeled data to encourage participants to develop methods that can learn from few examples and generalize to unseen classes. Specifically, participants may use only labels for classes within COCO class vocabulary. Participants are encouraged to use unsupervised or self-supervised learning methods to augment labeled data and improve their models’ performance.
• Participants may train methods on the COCO dataset (without LVIS labels).
• Participants may use our pre-computed object proposals, available here.
• As with the Long-Tail Challenge, the challenge organizers will evaluate submissions to identify the best-performing method and the most innovative approach. Winners will be invited to present their work at a workshop.
• Participants are required to submit their code and allow the challenge organizers to review their work to ensure fairness and prevent overfitting. The code will be used to verify that no training or tuning was performed on the test set or the held-out classes, and that the results represent the performance of the submitted methods.
• A benchmark with the current results is available here.

Organizers

Idil Esen Zulfikar

Ali Athar

Achal Dave

Aljosa Osep

Mark Weber

Yang Liu

Sabarinath Mahadevan

Jonathon Luiten

Tarasha Khurana

Paul Voigtlaender

Pavel Tokmakov

Advising Committee

Bastian Leibe

Deva Ramanan

Affiliations