val_utils/docs/KITTI-format.txt

Taken from download link found at: http://www.cvlibs.net/datasets/kitti/eval_tracking.php

###########################################################################
#           THE KITTI VISION BENCHMARK SUITE: TRACKING BENCHMARK          #
#              Andreas Geiger    Philip Lenz    Raquel Urtasun            #
#                    Karlsruhe Institute of Technology                    #
#                Toyota Technological Institute at Chicago                #
#                             www.cvlibs.net                              #
###########################################################################

For recent updates see http://www.cvlibs.net/datasets/kitti/eval_tracking.php.

This file describes the KITTI tracking benchmarks, consisting of 21 
training sequences and 29 test sequences. 

Despite the fact that we have labeled 8 different classes, only the classes 
'Car' and 'Pedestrian' are evaluated in our benchmark, as only for those 
classes enough instances for a comprehensive evaluation have been labeled. 

The labeling process has been performed in two steps: First we hired a set 
of annotators, to label 3D bounding boxes for tracklets in 3D Velodyne 
point clouds. Since for a pedestrian tracklet, a single 3D bounding box 
tracklet (dimensions have been fixed) often fits badly, we additionally 
labeled the left/right boundaries of each object by making use of Mechanical
Turk. We also collected labels of the object's occlusion state, and computed 
the object's truncation via backprojecting a car/pedestrian model into the
image plane.

NOTE: WHEN SUBMITTING RESULTS, PLEASE STORE THEM IN THE SAME DATA FORMAT IN
WHICH THE GROUND TRUTH DATA IS PROVIDED (SEE BELOW), USING THE FILE NAMES
0000.txt 0001.txt ... CREATE A ZIP ARCHIVE OF THEM AND STORE YOUR
RESULTS (ONLY THE RESULTS OF THE TEST SET) IN ITS ROOT FOLDER. FOR A 
RE-SUBMISSION, _ONLY_ THE RE-SUBMITTED RESULTS WILL BE SHOWN IN THE TABLE.

Data Format Description
=======================

The data for training and testing can be found in the corresponding folders.
The sub-folders are structured as follows:

  - image_02/%04d/ contains the left color camera sequence images (png)
  - image_03/%04d/ contains the right color camera sequence images  (png)
  - label_02/ contains the left color camera label files (plain text files)
  - calib/ contains the calibration for all four cameras (plain text files)

The label files contain the following information. 
All values (numerical or strings) are separated via spaces, each row 
corresponds to one object. The 17 columns represent:

#Values    Name      Description
----------------------------------------------------------------------------
   1    frame        Frame within the sequence where the object appearers
   1    track id     Unique tracking id of this object within this sequence
   1    type         Describes the type of object: 'Car', 'Van', 'Truck',
                     'Pedestrian', 'Person_sitting', 'Cyclist', 'Tram',
                     'Misc' or 'DontCare'
   1    truncated    Integer (0,1,2) indicating the level of truncation.
                     Note that this is in contrast to the object detection
                     benchmark where truncation is a float in [0,1].
   1    occluded     Integer (0,1,2,3) indicating occlusion state:
                     0 = fully visible, 1 = partly occluded
                     2 = largely occluded, 3 = unknown
   1    alpha        Observation angle of object, ranging [-pi..pi]
   4    bbox         2D bounding box of object in the image (0-based index):
                     contains left, top, right, bottom pixel coordinates
   3    dimensions   3D object dimensions: height, width, length (in meters)
   3    location     3D object location x,y,z in camera coordinates (in meters)
   1    rotation_y   Rotation ry around Y-axis in camera coordinates [-pi..pi]
   1    score        Only for results: Float, indicating confidence in
                     detection, needed for p/r curves, higher is better.
                     

Here, 'DontCare' labels denote regions in which objects have not been labeled,
for example because they have been too far away from the laser scanner. To
prevent such objects from being counted as false positives our evaluation
script will ignore objects tracked in don't care regions of the test set.
You can use the don't care labels in the training set to avoid that your object
detector/tracking algorithm is harvesting hard negatives from those areas, 
in case you consider non-object regions from the training images as negative 
examples.

The reference point for the 3D bounding box for each object is centered on the
bottom face of the box. The corners of bounding box are computed as follows with
respect to the reference point and in the object coordinate system:
x_corners = [l/2, l/2, -l/2, -l/2,  l/2,  l/2, -l/2, -l/2]^T
y_corners = [0,   0,    0,    0,   -h,   -h,   -h,   -h  ]^T
z_corners = [w/2, -w/2, -w/2, w/2, w/2, -w/2, -w/2, w/2  ]^T
with l=length, h=height, and w=width.

The coordinates in the camera coordinate system can be projected in the image
by using the 3x4 projection matrix in the calib folder, where for the left
color camera for which the images are provided, P2 must be used. The
difference between rotation_y and alpha is, that rotation_y is directly
given in camera coordinates, while alpha also considers the vector from the
camera center to the object center, to compute the relative orientation of
the object with respect to the camera. For example, a car which is facing
along the X-axis of the camera coordinate system corresponds to rotation_y=0,
no matter where it is located in the X/Z plane (bird's eye view), while
alpha is zero only, when this object is located along the Z-axis of the
camera. When moving the car away from the Z-axis, the observation angle
(\alpha) will change.

An overview of the coordinate systems, reference point and geometrical 
definitions is given in cs_overview.pdf.

To project a point from Velodyne coordinates into the left color image,
you can use this formula: x = P2 * R0_rect * Tr_velo_to_cam * y
For the right color image: x = P3 * R0_rect * Tr_velo_to_cam * y

Note: All matrices are stored row-major, i.e., the first values correspond
to the first row. R0_rect contains a 3x3 matrix which you need to extend to
a 4x4 matrix by adding a 1 as the bottom-right element and 0's elsewhere.
Tr_xxx is a 3x4 matrix (R|t), which you need to extend to a 4x4 matrix 
in the same way!

The sensors were not moved between the different days while taking footage.
However, the full camera calibration was performed for every day separately.
Therefore, only 'Tr_imu_velo' is constant for all sequences.

Note that while all this information is available for the training data,
only the data which is actually needed for the particular benchmark must
be provided to the evaluation server. However, all 17 values must be provided
at all times, with the unused ones set to their default values (=invalid).
Additionally a 18'th value must be provided
with a floating value of the score for a particular tracked detection, where 
higher indicates higher confidence in the detection. The range of your scores 
will be automatically determined by our evaluation server, you don't have to
normalize it, but it should be roughly linear.

Tracking Benchmark
==================

The goal in the object tracking task is to estimate object tracklets for the 
classes 'Car', 'Pedestrian', and (optional) 'Cyclist'. The tracking
algorithm must provide as output the 2D 0-based bounding boxes in each image in
the sequence using the format specified above, as well as a score, indicating
the confidence in the particular frame for this track. All other values must be
set to their default values (=invalid), see above. One text file per sequence
must be provided in a zip archive, where each file can contain many detections,
depending on the number of objects per sequence. In our evaluation we only
evaluate detections/objects larger than 25 pixel (height) in the image and do
not count Vans as false positives for cars or Sitting Persons as wrong positives
for Pedestrians due to their similarity in appearance. (All ignored objects 
are considered as DontCare areas.) As evaluation criterion we follow the 
HOTA, CLEARMOT and Mostly-Tracked/Partly-Tracked/Mostly-Lost metrics.

Raw Data
========

Raw data is mapped to the tracking benchmark sequences and available for 
download.

The velodyne and positioning data for training and testing can be found in the 
corresponding folders. The sub-folders are structured as follows:

  - velodyne/%04d/ contains the raw velodyne point clouds (binary file)
  - oxts/ contains the raw position (oxts) data (plain text files)