Skip to content

InferenceCommandLines

Jump to bottom
Valentina edited this page Nov 27, 2023 · 6 revisions

MXNet: examples of inference command lines

Sync mode:

python inference_mxnet_sync_mode.py --model_name mobilenetv2_1.0 --input ./data \
                                    --input_name data --input_shape 3 3 224 224 \
                                    --norm --mean 0.485 0.456 0.406 --std 0.229 0.224 0.225 \
                                    --batch_size 3 --labels labels/image_net_labels.json \
                                    --task classification --output_names output

Async mode:

python inference_mxnet_async_mode.py --model_name mobilenetv2_1.0 --input ./data \
                                     --input_name data --input_shape 3 3 224 224 \
                                     --norm --mean 0.485 0.456 0.406 --std 0.229 0.224 0.225 \
                                     --batch_size 3 --labels labels/image_net_labels.json \
                                     --task classification --output_names output

TensorFlow: examples of inference command lines

Infer models stored in the savedmodel-format:

python inference_tensorflow.py --model ./public/densenet-121-tf/densenet-121.savedmodel/ \
                               --input ./data/ --batch_size 3 --task classification \
                               --output_names predictions --labels labels/image_net_synset.txt \
                               --input_shape 224 224 3 --mean 123.68 116.78 103.94 \
                               --input_scale 58.395 57.12 57.375
python inference_tensorflow.py --model ./public/googlenet-v4-tf/saved_model/ \
                               --input ./data/ --batch_size 3 --task classification \
                               --output_names InceptionV4/Logits/Predictions \
                               --labels labels/image_net_synset_first_class_base.txt \
                               --input_shape 299 299 3 --mean 127.5 127.5 127.5 \
                               --input_scale 127.7 127.5 127.5
python inference_tensorflow.py --model ./public/efficientnet-b0/efficientnet-b0/saved_model/ \
                               --input ./data/ --batch_size 4 --task classification \
                               --output_names logits --labels labels/image_net_synset.txt \
                               --input_shape 224 224 3 --mean 0.485 0.456 0.406 \
                               --input_scale 0.229 0.224 0.225

Infer models stored in the pb-format:

python inference_tensorflow.py --model ./public/googlenet-v4-tf/inception_v4.frozen.pb \
                               --input ./data/ --batch_size 3 --task classification \
                               --input_name "input:0" --output_names InceptionV4/Logits/Predictions \
                               --labels labels/image_net_synset_first_class_base.txt \
                               --input_shape 299 299 3 --mean 127.5 127.5 127.5 \
                               --input_scale 127.7 127.5 127.5

Infer models stored in the meta-format:

python inference_tensorflow.py --model ./public/efficientnet-b0/efficientnet-b0/model.ckpt.meta \
                               --input ./data/ --batch_size 4 --task classification \
                               --output_names logits --labels labels/image_net_synset.txt \
                               --input_shape 224 224 3 --input_name "sub:0" \
                               --mean 0.485 0.456 0.406 --input_scale 0.229 0.224 0.225

TensorFlow Lite: examples of inference command lines

python inference_tensorflowlite.py --model ./public/densenet-121-tf/densenet-121.tflite \
                                   --input ./data/ --batch_size 3 --task classification \
                                   --output_names output --labels labels/image_net_synset.txt \
                                   --mean [123.675,116.28,103.53] --input_scale [58.395,57.12,57.375]

Infer model converted form the TensorFlow to TensorFlow Lite format:

# Please, set correct paths to convert model from TF to TF lite format
python tflite_converter.py --model-path ./public/mobilenet-v1-1.0-224-tf/mobilenet_v1_1.0_224_frozen.pb \
                           --input-names input --output-names MobilenetV1/Predictions/Reshape_1 \
                           --source-framework tf
python inference_tensorflowlite.py --model ./public/mobilenet-v1-1.0-224-tf/mobilenet_v1_1.0_224.tflite \
                                   --input ./data/ --batch_size 3 \
                                   --task classification --output_names output \
                                   --labels labels/image_net_synset_first_class_base.txt \
                                   --mean [123.675,116.28,103.53] --input_scale [58.395,57.12,57.375]

PyTorch: examples of inference command lines

Infer models stored in the pth-format:

python inference_pytorch.py --model_name resnet50 --weights ./public/resnet-50-pytorch/resnet50-19c8e357.pth \
                            --input ./data/ --labels labels/image_net_labels.json --batch_size 4 \
                            --input_names data --input_shapes data[4,3,224,224] -ni 1 \
                            --task classification --mean [123.675,116.28,103.53] \
                            --input_scale [58.395,57.12,57.375]

Apache TVM: examples of inference command lines

Infer model stored in the PyTorch-format:

python inference_tvm_pytorch.py --model_name resnet50 --weights ./public/resnet-50-pytorch/resnet50-19c8e357.pth \
                                --input ./data/ILSVRC2012_val_00000023.JPEG --labels labels/image_net_synset.txt \
                                --batch_size 1 --input_name data --input_shape 1 3 224 224 -ni 1 \
                                --task classification --norm --mean 0.485 0.456 0.406 --std 0.229 0.224 0.225 \
                                --opt_level 0 --layout NCHW

Infer model stored in the Caffe-format:

python inference_tvm_caffe.py --task classification -is 1 3 224 224 -m ./public/googlenet-v1/googlenet-v1.prototxt \
                              -w ./public/googlenet-v1/googlenet-v1.caffemodel -i ./data/ --mean 0.408 0.459 0.482 \
                              -b 4 -l labels/image_net_synset.txt --layout NCHW --not_softmax --channel_swap 2 1 0

Infer model stored in the ONNX-format:

# Please, set correct paths converting from TF to ONNX format
python -m tf2onnx.convert --saved-model densenet-121.savedmodel/ --output densenet-121-tf.onnx
python inference_tvm_onnx.py --model ./public/densenet-121-tf/densenet-121-tf.onnx --task classification \
                             --input_shape 1 224 224 3 --input ./data/ --batch_size 4 \
                             --labels labels/image_net_synset.txt --mean 123.68 116.78 103.94
                             --std 58.395 57.12 57.375 --layout NHWC --not_softmax --channel_swap 0 1 2 \
                             -ni 1 --output_names output

Infer models converted to the TVM-format:

# Please, set correct paths converting from ONNX to TVM format
python onnx_to_tvm_converter.py -mn densenet-121-tf -is 4 224 224 3 \
                                -m ./public/densenet-121-tf/densenet-121-tf.onnx \
                                -o ./public/densenet-121-tf/tvm_bs4
python inference_tvm.py -mn densenet-121-tf -m ./public/densenet-121-tf/tvm_bs4/densenet-121-tf.json \
                        -w  ./public/densenet-121-tf/tvm_bs4/densenet-121-tf.params -i ./data/ \
                        -b 4 -l labels/image_net_synset.txt -is 4 224 224 3 --not_softmax \
                        -t classification --channel_swap 0 1 2 --layout NHWC --mean 123.68 116.78 103.94 \
                        --std 58.395 57.12 57.375 --input_name input_1

Infer models tuned using Apache TVM:

# Please, set correct paths tuning the model
python tvm_meta_schedule.py -m ./public/densenet-121-tf/tvm_bs4/densenet-121-tf.json \
                            -p ./public/densenet-121-tf/tvm_bs4/densenet-121-tf.params \
                            -t "llvm -mcpu=core-avx2 --num-cores=6" -n 64 --max_trials_per_task 4 \
                            -o ./public/densenet-121-tf/tvm_bs4/densenet-121-tf.so
python inference_tvm.py -mn densenet-121-tf -m ./public/densenet-121-tf/tvm_bs4/densenet-121-tf.so \
                        -w  ./public/densenet-121-tf/tvm_bs4/densenet-121-tf.params \
                        -i ./data/ -b 4 -l labels/image_net_synset.txt -is 4 224 224 3 --not_softmax \
                        -t classification --channel_swap 0 1 2 --layout NHWC --mean 123.68 116.78 103.94 \
                        --std 58.395 57.12 57.375 --input_name input_1
Clone this wiki locally