GTDLBench

Abstract

With its effective and wide real applications, deep learning attracts great attention nowadays. Many deep learning frameworks emerged to facilitate deep learning research, development and applications. However, few research has been conducted to effectively measure these frameworks. And it still remains a critical task on choosing the optimal framework for specific deep learning models and applications. GTDLBench aims at introducing a hybrid benchmark to systematically measure the performance of these deep learning frameworks as well as the potential performance of different hardware devices. The performance metrics derived from both deep learning and systems field are introduced to present an easy and holistic approach for deep learning benchmarking.

Metrics

• Training Time (s)
• Testing Time (s)
• Accuracy (%)

URL: http://git-disl.github.io/GTDLBench/

Tutorial

1. Installation: https://youtu.be/QD5Cy6Y0fkk
2. Dataset preparation: https://youtu.be/ZAYTXQP_zSc
3. MNIST: https://youtu.be/HuPWPFB7MnU
4. CIFAR-10: https://youtu.be/PI3CiBrYc5A

Please cite the following papers:

Papers under GTDLBench

Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond

A Comparative Measurement Study of Deep Learning as a Service Framework

Experimental Characterizations and Analysis of Deep Learning Frameworks

@INPROCEEDINGS{GTDLBenchICDCS, 
    author={{Liu}, Ling and {Wu}, Yanzhao and {Wei}, Wenqi and {Cao}, Wenqi and {Sahin}, Semih and {Zhang}, Qi}, 
    booktitle={2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)}, 
    title="{Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond}",
    year={2018},
    pages={1258-1269}, 
    doi={10.1109/ICDCS.2018.00125}, 
    ISSN={2575-8411}, 
    month={July},
}

@ARTICLE{GTDLBenchTSC,
    author={Wu, Yanzhao and Liu, Ling and Pu, Calton and Cao, Wenqi and Sahin, Semih and Wei, Wenqi and Zhang, Qi},
    journal={IEEE Transactions on Services Computing},
    title={A Comparative Measurement Study of Deep Learning as a Service Framework},
    year={2019},
    volume={},
    number={},
    pages={1-1},
    doi={10.1109/TSC.2019.2928551},
}

@INPROCEEDINGS{GTDLBenchBigData, 
    author={{Wu}, Yanzhao and and {Cao}, Wenqi and {Sahin}, Semih and {Liu}, Ling}, 
    booktitle={2018 IEEE 38th International Conference on Big Data}, 
    title="{Experimental Characterizations and Analysis of Deep Learning Frameworks}", 
    year={2018},
    volume={},
    number={},
    pages={372-377},
    doi={10.1109/BigData.2018.8621930},
}