Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TFLite MobileNet quantized (int8) fails with OpenCV preprocessing for MobileNet (rgbf32) #40

Open
psyhtest opened this issue Aug 15, 2019 · 1 comment

Comments

@psyhtest
Copy link
Contributor

Running MobileNet quantized with TFLite on images preprocessed with OpenCV for MobileNet into rgbf32 fails:

$ ck benchmark program:image-classification-tflite --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,normalized-for.mobilenet \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.001913s
All images loaded in 0.002327s
All images classified in 0.460941s
Average classification time: 0.043231s
Accuracy top 1: 0.0 (0 of 10)
Accuracy top 5: 0.0 (0 of 10)
--------------------------------

The same works fine with TF-C++:

$ ck benchmark program:image-classification-tf-cpp --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,normalized-for.mobilenet \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.081466s
All images loaded in 0.002623s
All images classified in 0.528722s
Average classification time: 0.011366s
Accuracy top 1: 0.7 (7 of 10)
Accuracy top 5: 0.9 (9 of 10)
--------------------------------

or with TFLite on images preprocessed using OpenCV into rgb8:

$ ck benchmark program:image-classification-tflite --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,rgb8 \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.001789s
All images loaded in 0.000663s
All images classified in 0.477427s
Average classification time: 0.043434s
Accuracy top 1: 0.7 (7 of 10)
Accuracy top 5: 0.9 (9 of 10)
--------------------------------
@psyhtest
Copy link
Contributor Author

Perhaps it's somewhat similar to mlcommons/inference#334?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant