torch.jit.ScriptModule with Cuda has error

[Sprinzl]

Hello,

I trying to get LLMs and Visionmodels running with Torchsharp. I got quite success with this.
I use huggingface and timm.
But it only works with 'cpu' not with 'gpu'. I have no idea why.

greeting Michael

c# side of the coin (bad code):::
private static torch.jit.ScriptModule model = torch.jit.load("D:/Save/fbdeit_scripted.pt");
public static void Test()
{
using var grad = torch.no_grad();
var path = "D:\Save\MrBeam.bmp";//24 bit Bitmap
var image = new DataExtraction().ExtractData(path);
image.ToString( ).Show();
var resutl = (Tensor) model.forward(image.unsqueeze(0).to(float32));
// throws me an error if cuda is set ?????????????????????
//var resutl = (Tensor) model.To("Cuda").forward(image.unsqueeze(0).to(float32).To("Cuda"));
var clsidx = torch.argmax(resutl);
((int)clsidx).ToString().Show();
}

python side of the coin (yes really bad code) :::

import urllib
from PIL import Image
import torch
import timm
import requests
import torch
import torchvision.transforms as transforms
from timm.data.constants import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD

model = torch.hub.load('facebookresearch/deit:main', 'deit_base_patch16_224', pretrained=True)
model.eval()

url = "file:///D:/Save/Löwe.bmp"
image = Image.open(urllib.request.urlopen(url))

transform = transforms.Compose([
transforms.Resize(256, interpolation=3),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD),
])

img = transform(image)

out = model.to('cuda')(img.unsqueeze(0).to('cuda'))
clsidx = torch.argmax(out)
out.size()

scripted_model = torch.jit.script(model.to('cpu'))
scripted_model.save("D:/Save/fbdeit_scripted.pt")
frozen = torch.jit.load("D:/Save/fbdeit_scripted.pt")
frozen.forward(img.unsqueeze(0))
clsidx = torch.argmax(out)
clsidx

Getting the picture right is pain too:::::

public Tensor ExtractData(string Path)
{
using var grad = torch.no_grad();
var InputImg = System.Drawing.Image.FromFile(Path);//24 byte bmp
var myArray = (byte[])new ImageConverter().ConvertTo(InputImg, typeof(byte[]));
var width =BitConverter.ToInt32(myArray[18..(18 + 4)]);
var height = BitConverter.ToInt32(myArray[22..(22 + 4)]);
var channels = BitConverter.ToInt32(myArray[50..(50 + 4)]);
var offset = BitConverter.ToInt32(myArray[10..(10 + 4)]);
var data = myArray[offset..];
//(H x W x C)
var tensor = torch.tensor(data, dtype: ScalarType.Int32).view(height,width,3).to(float32);
var _resize = 256;
var _inter = InterpolationMode.Bilinear;
var IMAGENET_DEFAULT_MEAN = torch.tensor( new double[] { 0.229, 0.224, 0.225 });
var IMAGENET_DEFAULT_STD = torch.tensor(new double[] { 0.485, 0.456, 0.406 });

 Func<Tensor, Tensor> resize = (x) =>
 {
     x = x.permute(2,1,0);//(C x H x W)
     var h = x.size(1);
     var w = x.size(2);
     var ratio = (double)_resize / Math.Min(h, w);
     x = torch.nn.functional.interpolate(x.unsqueeze(0), scale_factor: new double[] { ratio, ratio }, mode: _inter, align_corners: false, recompute_scale_factor: false);
     return x.squeeze(0).permute(2, 1, 0);//(H x W x C)
 };
 

 Func<Tensor, Tensor> range = (x) =>
 {
     x = x / 255.0f;
     x[x.isnan()] = 0.0f;
     return x;
 };


 Func<Tensor, Tensor> flip0 = (x) =>
 { return x.permute(2,0,1); };

 Func<Tensor, Tensor> flip1 = (x) =>
 { return x.permute(1,2,0); };

 Func<Tensor, Tensor> normalize = (x) => 
 {return  (x - IMAGENET_DEFAULT_MEAN) / IMAGENET_DEFAULT_STD;};


 var trans = torchvision.transforms.Compose
     (
         transforms.Lambda(resize),
         transforms.Lambda(flip0),
         transforms.CenterCrop(224),
         transforms.Lambda(range),
         transforms.Lambda(flip1),
         transforms.Lambda(normalize)
     );

 tensor = trans.call(tensor);
 return tensor.permute(2,0,1);

}

[alinpahontu2912]

Hey @Sprinzl, can you tell me the torchsharp version and the operating system you are using? Can you check if you have the correct cuda-enabled nuggets isntalled?

[Sprinzl]

Hi, @alinpahontu2912
i am using TorchSharp-cuda-windows with LibTorch 2.5.1. Cuda usally works fine for me if I use it for my own modules. Maybe it has something to do with the Libtorch from Python because I run it from local folders.... I am using Windows.
I found a difference.
for python: '2.6.0+cu118'
for c#: Torch 2.4.0.0

[alinpahontu2912]

Hey @Sprinzl, thank you for the fast answer. I don't understand what you mean by running in local folders. Torchsharp is currently being built with libtorch 2.4.0, so that is ok. Can you show me the installed nuggets in your project ? I suspect it might be some misconfiguration there.

You only need to install TorchSharp-cuda-windows package and that will automatically get you all required dependencies ( meaning torchsharp and libtorch-cuda). Did you also install the separate libtorch nugget yourself ?

[Sprinzl]

It might be something with the python pip version of torch and cuda. I do not know. I think I will kill Python and reinstall.

[alinpahontu2912]

I think I see some issues with your code. Would you mind trying:
var result = (Tensor) model.cuda().forward(image.unsqueeze(0).to(torch.float32).cuda());
I believe you need to specify data types as torch._datatype for them to be properly defined and also the way to change the device processing the model/doing the training is by using .cpu() or .cuda().
Would you mind giving this a try?

[Sprinzl]

I tried. It did not work. What I found out, or maybe it is a guess, is what you make a forward pass before you save the model with jit. It has to be also set to eval. But GPU - no idea. I aready know what I have to set it to the right tensor dtype. Next guess is to reinstall torch to the right alining version in python, because it uses different cuda-backends. Also I do not know if backprop has some influence on this issue.