Any plan for pretraining?

[jbmaxwell]

I'm curious whether there's any plan to support pretrianing models from scratch?

[julien-c]

So, one way you could go about it, would be to use pytorch-transformers to pre-train or fine-tune your model, then use the script in model_generation as a starting point to convert to CoreML.

We do not have any plan to experiment with training on device, because realistically those models are way too large to be trained on anything other than a cutting-edge GPU 🙃

[jbmaxwell]

Excellent, thanks for the reply! I wondered about using gpt2.py as a starting point... Of course, I wouldn't expect to train one of these beasts on the device! That would be madness! Maybe one day... haha...
Just curious; would a model generated in this way be iOS 13+ only, or are the basic layer/objects compatible with iOS 12?

[julien-c]

Check out this tweet: https://twitter.com/julien_c/status/1154894146328563715

Short answer: iOS 13+ only

[jbmaxwell]

Ah, too bad, but not surprising. The new hotness is the new hotness for a reason! ;-)