Vulcan Datasets: Data & code for evaluating vision based mapping algorithms

In the paper, "Aniket Murarka and Benjamin Kuipers. A stereo vision based mapping algorithm for detecting inclines, drop-offs, and obstacles for safe local navigation. IROS, 2009", we presented a methodology for quantitatively evaluating and comparing the performance of vision based mapping algorithms.

The methodology consists of comparing the maps, constructed by the vision based mapping algorithm for several standard datasets, against ground truth maps of those datasets and computing various (false negative and false positive) error rates. Different mapping algorithms can then be compared on the basis of their overall error rates over all datasets.

As part of the paper, five standard datasets along with ground truth maps are being made available here. These were collected by driving around our robot, Vulcan, in various urban environments and collecting stereo, laser, and odometric data simultaneously. The laser data provided the basis for the ground truth maps.

Each dataset has between 350-500 stereo images (color and greyscale) and several hundred laser scans. The environments have many interesting properties: they all have either drop-offs or inclines (in addition to obstacles); large untextured regions; fair to good lighting; and both indoor and outdoor spaces.

In addition to the datasets we are also providing evaluation and sample code (written in MATLAB) to help in using the datasets. In particular the code contains functions for: (i) reading the data; (ii) transforming points from one coordinate frame to another; (iii) computing depth using a standard correlation stereo algorithm; (iv) building a 3D model using an occupancy grid method; (v) building a simple annotated 2D grid map (called a safety map); (vi) evaluating the safety map by comparing it to the ground truth map; and (vii) displaying the data.

To evaluate your mapping algorithm follow the links on the right to download the standard datasets, ground truth maps, evaluation code, and sample code. Instructions are included in the downloads.

Citation: If you use the datasets and code in your work please cite the IROS-09 paper. If you only use the code please cite the following URL: "http://www.eecs.umich.edu/~kuipers/research/pubs/Murarka-iros-09.html" (this is the permanent URL that links to this page).

Bugs: I appreciate bug reports or reports of documentation errors. Please send me email at: maniket at gmail com

Last updated on December 16th, 2013.