Taken from: https://www.vision.rwth-aachen.de/page/mots


Annotation Format
We provide two alternative and equivalent formats, one encoded as png images, and one encoded as txt files. The txt files are smaller, and faster to be read in, but the cocotools are needed to decode the masks. For code to read the annotations also see mots_tools/blob/master/mots_common/io.py

Note that in both formats an id value of 10,000 denotes an ignore region and 0 is background. The class id can be obtained by floor divison of the object id by 1000 (class_id = obj_id // 1000) and the instance id can be obtained by the object id modulo 1000 (instance_id = obj_id % 1000). The object ids are consistent over time.

The class ids are the following

car 1
pedestrian 2
png format
The png format has a single color channel with 16 bits and can for example be read like this:

import PIL.Image as Image
img = np.array(Image.open("000005.png"))
obj_ids = np.unique(img)
# to correctly interpret the id of a single object
obj_id = obj_ids[0]
class_id = obj_id // 1000
obj_instance_id = obj_id % 1000
When using a TensorFlow input pipeline for reading the annotations, you can use

ann_data = tf.read_file(ann_filename)
ann = tf.image.decode_image(ann_data, dtype=tf.uint16, channels=1)


txt format
Each line of an annotation txt file is structured like this (where rle means run-length encoding from COCO):

time_frame id class_id img_height img_width rle
An example line from a txt file:

52 1005 1 375 1242 WSV:2d;1O10000O10000O1O100O100O1O100O1000000000000000O100O102N5K00O1O1N2O110OO2O001O1NTga3
Which means

time frame 52
object id 1005 (meaning class id is 1, i.e. car and instance id is 5)
class id 1
image height 375
image width 1242
rle WSV:2d;1O10000O10000O1O100O100O1O100O1000000000000000O100O...1O1N

image height, image width, and rle can be used together to decode a mask using cocotools.