How to Run With Polygraphy Graph Surgeon

[sam-h-bean]

Hey I love your repo and want to use it but because my model does not work with TRT out of the box I need to run the Polygraphy Graph Surgeon to Sanitize before converting.

I don't see a way to run your entrypoint for this use case. Basically I want to export to ONNX, then run the sanitize CLI, then use that ONNX model with your code. I have a fork where I'm just removing the convert to ONNX step as a hack but am wondering how we might support this use case natively in the repo.

[pommedeterresautee]

Thank you @sam-h-bean. For the sanitization we have just implemented onnxoptimizer which is called during the onnx export:

transformer-deploy/src/transformer_deploy/backends/onnx_utils.py

Line 52 in d99e08e

def clean_graph(proto: onnx.ModelProto) -> onnx.ModelProto:

Would it work for you?

[sam-h-bean]

@pommedeterresautee I'm not sure, I need to run the following command to make it work with TRT.

export POLYGRAPHY_AUTOINSTALL_DEPS=1
polygraphy surgeon sanitize ./model.onnx --fold-constants --output model-folded.onnx --save-external-data

I'm not sure if this functionality is covered by your clean graph method

[sam-h-bean]

@pommedeterresautee I tried doing it this way and still ran into the following

[08/09/2022-01:08:34] [TRT] [E] 2: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )

[pommedeterresautee]

can you let me know which model it is?

[sam-h-bean]

Codegen-16B-multi

We've had issues with it with other tools as well such as pytorch-tensorrt