ONNX Export

[minimaxir]

ONNX has good transformers export and is a point of focus for the ONNX team. ONNX exporting may be more useful/flexible than TorchScript (I have a specific secret use case in mind...)

[minimaxir]

A demo script that allows an ONNX export to support past is available here: huggingface/transformers#4805 (comment)

That may be dependent on #7 though.

[minimaxir]

Rather, that's without past: this fragment supports past: https://github.com/microsoft/onnxruntime/blob/7c8e1580a13ce333e47a41146bccfc90b3a70db5/onnxruntime/python/tools/transformers/benchmark_gpt2.py#L246

[minimaxir]

ONNX export currently deprioritized by lack of Python 3.9 support, which has become increasingly comment (it will likely be added in the next version of onnx, but no ETA on that.

[minimaxir]

The latest version of Transformers now explicitly has Python exporting, which removes one of the annoying parts: https://huggingface.co/transformers/serialization.html#onnx-onnxruntime

However, a bunch of generation is tied explicitly to PyTorch, so may have to reevaluate that.

[C00reNUT]

The latest version of Transformers now explicitly has Python exporting, which removes one of the annoying parts: https://huggingface.co/transformers/serialization.html#onnx-onnxruntime

However, a bunch of generation is tied explicitly to PyTorch, so may have to reevaluate that.

Here is a repo where they combine ONNX model and generation strategies implemented in Transformers - https://github.com/rayhern/convert-gpt2-xl-to-onnx