Questions about package

[AriMKatz]

Hello,

Interesting package. Can you talk a bit about the possible use cases? Is this simply for a more robust but less flexible gpu backend for flux models, or will it allow using layers and operators defined in pytorch?

Also, there's been a bunch of work on GPUArrays recently. Would it make sense to subtype from that?

[DhairyaLGandhi]

The starting motivation was to have the kernels from torch exposed in Julia, to accelerate some of the tasks that have been optimised there. The example in the README already demonstrates how it can work with Flux, without much issue.

The GPUArrays backend (specifically for CuArrays), is something that we have been very interested in to make it completely seamless, and is being actively worked on. This way this package can augment existing infrastructure, while being fairly lightweight itself.

You can already use most operators that torch exposes (the C++ api), but hasn't added much poilish for defining Modules and Layers yet. It will largely depend on what the community feels is most beneficial, since one could simply define the layers in Flux, like usual, and have the same layers be available through torch.

[AriMKatz]

How would this work with flux models that use particulars of cuarrays, such as custom kernels?

Edit: At some point would it be possible to just import and compose layers from something like this https://github.com/huggingface/transformers, with flux models/ optimizers?

[DhairyaLGandhi]

Custom kernels are interesting. Many of them have their own versions in torch, and if they are simply composed of common primitives (mapreduces, bmms etc.), they should just work.

The memory layout needs to be moved between tensors and CuArrays to get this to run seamlessly. Its something that might need better thought.

It should be doable already if the layers were written in Flux, for the most part. Like I am going to add in different conv kernels, for convtranspose, and depthwiseconv.

[msaroufim]

Followup question, are there any interesting examples of things can be done with torch.jl that can't just be done with pytorch and python. Tagging @ChrisRackauckas in case he's also interested

[ChrisRackauckas]

In theory you could solve and fit stiff ODEs with GPU acceleration via DifferentialEquations.jl + Torch.jl, but no one has wrapped the lu-factorization so that precludes that for now. You could get a nice GMRES from IterativeSolvers.jl if someone wraps the QR. These aren't extra special though. I'm not sure something extremely extra special can be done here.