CoreMLGPT2 crashing on iPhone X

[krishkumar]

Message from debugger: Terminated due to memory issue

Is the GPT2-512 model working better on newer devices?

[julien-c]

iPhone X "only" has 3 GB RAM and the OS pretty aggressively kills your app if it goes above 1.7 GB used memory.

Your best bet is to run the model on either an iPhone XS (4 GB RAM) or an iPad Pro (up to 6 GB RAM).

[krishkumar]

Makes sense. Thanks!

Wondering if we can train a lightweight or dumber version of the GPT2 model that can run on older devices like the X.

[julien-c]

Yes – quantization could work here, see #1. It's also built in to CoreML so it shouldn't be too hard to try.

https://developer.apple.com/documentation/coreml/reducing_the_size_of_your_core_ml_app

[krishkumar]

Thanks @julien-c.

I attempted to quantize this to half precision as suggested by the Apple docs.

Getting this error which might or might not be related to the model itself -

/usr/local/lib/python3.6/dist-packages/coremltools/models/utils.py in _convert_nn_spec_to_half_precision(spec)
    320                             ' not yet implemented\n')
    321         else:
--> 322             raise Exception('Unknown layer ' + layer_type)
    323 
    324     return spec

TypeError: must be str, not NoneType

Refer to this Colab notebook to try it out - https://colab.research.google.com/drive/1QC90lE-LUDEUXMUGer-5HHLLFRG6dF7F

[julien-c]

Hmm, I think your version of coremltools is way too old :)

Also related to this issue I believe that @LysandreJik converted a smaller version of gpt2-small that successfully runs on iPhone X.

[krishkumar]

You are right. But I only managed to get a different error with the latest version of coremltools - 3.0b4

/usr/local/lib/python3.6/dist-packages/coremltools/models/utils.py in _convert_nn_spec_to_half_precision(spec)
    332                             ' not yet implemented\n')
    333         else:
--> 334             raise Exception('Unknown layer ' + layer_type)
    335 
    336     return spec

Exception: Unknown layer expandDims

Is the smaller model available in the Resources folder of this project? I will check them out.