MiddEvalSDK

Software development kit for the Middlebury Stereo Evaluation v.3

Version 1.6
Daniel Scharstein
September 14, 2015


=============================== Contents ==================================

README.txt          -- this file

CHANGES.txt         -- change log

COPYING_LESSER.txt, 
COPYING.txt         -- license (GNU Lesser General Public License)

alg-ELAS/           -- sample stereo algorithm: Libelas by A. Geiger,
                       adpated with permission from  
                       http://www.cvlibs.net/software/libelas/

alg-ELAS/run        -- sample run script (adapt this for your algorithm)

code/               -- C++ code used by the scripts

runalg              -- script to run algorithm on datasets
runeval             -- script to evaluate algorithm results
runevalF            -- script to evaluate algorithm results with full-size GT
runviz              -- script to visualize .pfm disparities as .png
makezip             -- script to create zip archive of results
cleanalg            -- script to remove files created by algorithms


In addition to the SDK files, you will also need to download input
data and ground truth data, and unzip at the same level as the SDK
files.  This should result in a subset of the following directories:

testF/
testH/
testQ/
trainingF/
trainingH/
trainingQ/


============================= Quick Start =================================


Follow these steps to get started.  We'll use the quarter-resolution
files for now.


1. Download and unzip MiddEval3-data-Q.zip and MiddEval3-GT0-Q.zip.
You should have a directory MiddEval3 with the following contents:

COPYING.txt         README.txt  cleanalg  makezip  runeval   runviz  testQ/
COPYING_LESSER.txt  alg-ELAS/   code/     runalg   runevalF  trainingQ/


2. Compile Libelas as follows:

cd alg-ELAS/build
cmake ..
make
cd ../..


3. Compile the tools in code/ as follows:

cd code/imageLib
make
cd ..
make
cd ..


4. Run ELAS:

./runalg                         -- shows usage
./runalg Q Motorcycle ELAS       -- run ELAS on one dataset
./runalg Q Motorcycle ELAS v2    -- run again, save results with suffix 'v2'
./runalg Q training ELAS         -- run ELAS on all 15 training sets


5. Evaluate results by ELAS:

./runeval                        -- shows usage
./runeval Q Motorcycle 1 ELAS    -- evaluate ELAS with error thresh t=1.0
./runeval Q Motorcycle 2 ELAS_s  -- evaluate "sparse" ELAS results, t=2.0
./runeval Q Motorcycle 1         -- evaluate all results for Motorcycle, t=1.0
./runeval Q training 0.5 ELAS    -- evaluate ELAS on all training sets, t=0.5

   *** NOTE: For efficiency, the evaluation script 'runeval' uses the
   GT file disp0GT.pfm with the SAME resolution as your disparity map.
   In contrast, the "official" online evaluation evaluates at FULL
   resolution, and upsamples your results if necessary.  Error
   thresholds need to be converted accordingly, e.g., a threshold of
   2.0 at full resolution would correspond to a threshold of 0.5 at
   quarter resolution.  Even with this conversion, the numbers differ
   slightly when evaluating with the GT at full resolution.  To get
   the official numbers, you can use the script 'runevalF'.  You'll
   need to download the full-size files MiddEval3-data-F.zip and
   MiddEval3-GT0-F.zip in order to use this script.  Once you do, try

./runevalF Q Motorcycle 1.0  ELAS   -- official numbers using full-res GT
./runeval  Q Motorcycle 0.25 ELAS   -- approximate numbers using Q-size GT

   In the online table for the training sets, the name of each method
   links to a zip file of the results.  You can download such a zip
   file, say results_SGM_Q.zip, unzip it *within* MiddEval3, and
   evaluate these results as well:

./runeval Q training 1 ELAS SGM

   Finally, to get evaluation output in column format ready to be imported
   into other programs such as Excel or gnuplot, use option -b ("brief"):

./runevalF -b Q all 2.0
./runeval  -b Q all 0.5


6. Examine disparity maps

   We highly recommend to install Heiko Hirschmueller's "cvkit" (link
   from webpage), which includes a powerful image viewer "sv".  Once
   installed, try the following:

sv trainingQ/Motorcycle/disp*pfm

   Hit 'h' for help, 'm' to change colormaps, 'k' to keep the colormap
   setting, and the arrow keys to switch between files.  Most amazing:
   hit 'p' to start the 3D viewer 'plyv' and visualize the disparities
   in 3D! (plyv reads the camera parameters from the calib.txt files.)

   Alternatively, run

./runviz Q

   to translate all disp*pfm files into disp*-c.png color encodings,
   and view those using any image viewer.  On subsequent runs, runviz
   will only re-convert any pfm files that have changed.


7. Add your own algorithm:

   Create a directory alg-XXX, where XXX is a (short) name for your
   method.  Then create a "run" script within your alg-XXX directory,
   using the one in alg-ELAS as a model.  Once done, test your
   algorithm.  For example, calling the new method "Ours":

./runalg Q Motorcycle Ours
./runeval Q Motorcycle 1 Ours

   A few words on the disparity encoding: We use floating-point
   disparities in PFM format, with INFINITY representing unknown
   values.  If your algorithm produces disparities in other formats
   (e.g., scaled 16-bit PNG or floating-point TIF), that's ok, and you
   can convert the output using the "imgcmd" tool (part of cvkit).
   Alternately, you can use the tool code/disp2pfm to convert integer
   disparities (e.g. in pgm format) to pfm.
   See the alg-ELAS/run script for more detailed information.  Note,
   however, that many of the half-size datasets and most of the
   full-size datasets have disparity ranges that exceed 255.  Thus, if
   your method produces 8-bit integer disparities, you are limited to
   working with the quarter-resolution datasets and won't be able to
   achieve accurate results when evaluated at full resolution.

   Alternately, you can change your implementation to write PFM
   disparities directly.  You can use the following code as examples:
     alg-ELAS/src/main.cpp
     code/imageLib/ImageIO.cpp


8. Submit your results:

   Our online web interface supports uploading of zip files of results
   on either just the 15 training images or all 30 training and test
   images.  You can upload the results on just the training set in
   order to see a temporary table comparing your results with those
   uploaded by others.  To do this, run

./makezip Q training Ours

   and upload the resulting file resultsQ-Ours.zip using the web
   interface.  This scripts ensures that all necessary files are
   present and that the runtimes files contain numbers.

   Once you are ready to submit your results to the permanent table,
   you must upload a complete set of results on ALL 30 datasets (15
   training, 15 test).  All images must have the same resolution (one
   of Q, H, or F).  You must use constant parameter settings (except
   for disaprity ranges), and you cannot "mix and match" resolutions
   (e.g., use trainingQ/Piano and trainingH/Teddy).  All of this is
   ensured if you use "runalg <res> all", e.g.:

./runalg Q all Ours

   Then, create a zip file with all results using

./makezip Q all Ours

   and upload the resulting file using the web interface.  You will
   then be able to provide information about your method and request
   publication of both training and test results.  Anonynous listings
   for double-blind review processes are possible.  Note that in order
   to prevent parameter tuning to the test data, you will have only
   one shot at evaluating your results on the test data.  In other
   words, you may not request publication for two different sets of
   results by the same method. Also, your results on the training data
   will be available for download by others.  So please be sure to
   only submit your final set of results for publication.  You may of
   course upload and evaluate results on the training set as often as
   you want.


9. To clean results created by algs (disp*pfm, time*txt, disp*-c.png), use:

./cleanalg                       -- shows usage
./cleanalg Q ELAS                -- removes all results by ELAS
./cleanalg Q ELASv2              -- removes all results by ELASv2
./cleanalg Q ELAS\*              -- removes all results by ELAS (all versions)