How do I change a few things in the implementation?

[chiragpr]

I would like to run on my own dataset, and also, print the accuracy on the source and target domains. In the paper implementations, I don't see any accuracy metric. Please guide the amateur learner looking at this to change the dataset to their own, and to implement accuracy terms. Thanks

[KevinMusgrave]

I've created this Google Colab notebook that you can run:

https://colab.research.google.com/drive/109TaIdhLGu7LLFvm5gZ401tYYsKJrnNP?usp=sharing

It's kind of like the existing example notebooks, but with fewer helper functions. That is, there's more boilerplate, but you get to see more clearly what's going on.

The "Datasets" and "Validators" sections might be most relevant to you.

Let me know if you need help with any specific parts. And if you want less boilerplate, I can show you which pytorch-adapt functions to use.

[chiragpr]

Thanks a ton for your reply. I was able to understand this one much better. But however, using the code in https://colab.research.google.com/drive/109TaIdhLGu7LLFvm5gZ401tYYsKJrnNP?usp=sharing, I found that there was an error which says " object of type 'type' has no len()" when I copy/pasted this code :

from torchvision.datasets import MNIST
from torchvision.datasets import SVHN
from torchvision import transforms as T
from pytorch_adapt.datasets import SourceDataset, TargetDataset, CombinedSourceAndTargetDataset, DataloaderCreator

num_classes = 10
transform = T.ToTensor()
src_dataset = MNIST
target_dataset = SVHN

src_dataset = SourceDataset(src_dataset)
target_dataset = TargetDataset(target_dataset)
dataset = CombinedSourceAndTargetDataset(src_dataset, target_dataset)

#THE ISSUE IS IN THE SECOND LINE OF THIS BLOCK OF CODE
dc = DataloaderCreator(batch_size=32, num_workers=2)
dataloaders = dc(train=dataset, src_train=src_dataset, target_train=target_dataset)

into the "Datasets" cell of the same colab. Can you shed some more light as to how I customize the datasets (source, target)? I can not thank you enough for your time, but it is very helpful for me since I am working on my masters thesis in UDA, and plan to use this repository.

[KevinMusgrave]

I think it's because you haven't initialized the datasets.

Change this:

src_dataset = MNIST
target_dataset = SVHN

to this:

src_dataset = MNIST(root="mnist", download=True)
target_dataset = SVHN(root="svhn", download=True)

[chiragpr]

Great! I am finally able to load datasets. But however in this case, this error comes up in the "Traning+Evaluation" code block : "TypeError: cannot convert dictionary update sequence element #0 to a sequence".
Any thoughts on how I debug this? Thanks a ton again.

[chiragpr]

I request you to give an implementation of an algorithm with any two datasets as template (say MNIST and SVHN in my case) so that we can better appreciate and understand the code. Thank you again and eagerly waiting!

[KevinMusgrave]

Good idea! I've updated the notebook to run on MNIST and SVHN: https://colab.research.google.com/drive/109TaIdhLGu7LLFvm5gZ401tYYsKJrnNP?usp=sharing

You might want to start with a model that is already trained on the source dataset, instead of a randomly initialized model. If you're using MNIST as the source dataset, you can use these pretrained models:

from pytorch_adapt.models import mnistC, mnistG

generator = mnistG(pretrained=True).to(device)
classifier = mnistC(pretrained=True).to(device)

[chiragpr]

Thank you! That was indeed very helpful.
Could you also please demonstrate how the "Algorithm" section of the code should be changed, in say ADDA (Using ADDAHook) or the GAN (GANHook)? What should this "list of optimizers" for the discriminator and generator look like?

[KevinMusgrave]

You have to pass in a separate list of optimizers for the discriminator and generator steps. Every optimizer in each list gets called at each iteration. Here's how to use GANHook (with MCCHook):

from pytorch_adapt.hooks import GANHook

discriminator = torch.nn.Linear(1200, 1).to(device)
model_dict = {"G": generator, "C": classifier, "D": discriminator}

d_optimizer = torch.optim.SGD(discriminator.parameters(), lr=0.001)

d_opts = [d_optimizer]
g_opts = [optimizer]

hook = GANHook(d_opts=d_opts, g_opts=g_opts, post_g=[MCCHook()])

[KevinMusgrave]

This notebook might also help: https://github.com/KevinMusgrave/pytorch-adapt/blob/main/examples/getting_started/PaperImplementationsAsHooks.ipynb

It shows how a bunch of different hooks are initialized.

[chiragpr]

I hope you can give me a few days to figure the hooks out before closing this issue. Right now I'm studying your datasets and validators library.
The code :

discriminator = torch.nn.Linear(1200, 1).to(device)
model_dict = {"G": generator, "C": classifier, "D": discriminator}

d_optimizer = torch.optim.SGD(discriminator.parameters(), lr=0.001)

d_opts = [d_optimizer]
g_opts = [optimizer]

hook = GANHook(d_opts=d_opts, g_opts=g_opts, post_g=[MCCHook()])

worked out almost fine but had an error in the training loop which said:
Target size (torch.Size([32])) must be the same as input size (torch.Size([32, 1]))
This might seem like a trivial issue but where am I going wrong? Thanks again for your time as always.

[KevinMusgrave]

Ah I forgot to flatten the output of the discriminator:

discriminator = torch.nn.Sequential(torch.nn.Linear(1200, 1), torch.nn.Flatten(start_dim=0)).to(device)

[chiragpr]

Hi, I am trying to run the code in your colab notebook on my local GPU. But for some reason, .to_device() does not seem to work and it always gives me "Torch not compiled with CUDA enabled error". I removed the .to_device() from MNISTFeatures().to_device and the generator.to_device() and it seemed to fix the problem for that block of code.
Is this an internal problem with my GPU , or a pytorch salvaging measure? Please help me out.

[KevinMusgrave]

Do you have an Nvidia GPU? How did you install PyTorch?

[chiragpr]

Yes. GTX1050… Installed the latest pytorch using pip

[KevinMusgrave]

I haven't run into this issue before. Maybe this will help: pytorch/pytorch#30664 (comment)

[chiragpr]

So when you copy that notebook (PA Issue 81) into a py file, and run it on your GPU, it runs without any errors?

[KevinMusgrave]

I haven't run that specific notebook, but I'm using pytorch everyday. Using to(device) has never been an issue.

You can test pytorch alone by looking at some of the tutorials. For example, just test the following (without the if statement) and see if it works: https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#tensor-operations

[coder-chica97]

Hi @KevinMusgrave, excellent work maintaining this repository! I want to run some of these algorithms on a custom dataset as well, but the PaperImplementationsAsHooks notebook is quite confusing to me. I don’t know how get_data is to be used on my dataset.
The notebook that you provided above has the MNIST>SVHN MCC implementation. But then again, I plan to use resnet50 or resnet34 instead of MNISTFeatures().
So, how do you modify the “PaperImplementationsAsHooks” example notebook to incorporate your own dataset, and produce results of algorithms from all the papers?

[KevinMusgrave]

@coder-chica97 Here's a copy of the "PA Issue 81" notebook, but using a ResNet model and a suitable dataset: https://colab.research.google.com/drive/1MU9uSxZBU483B1_KQlO8_xP165oyj_nB?usp=sharing

The only changes are in the "Datasets" and "Model + Optimizer" section.

[KevinMusgrave]

@coder-chica97 The "PaperImplementationsAsHooks" is meant to show the data and model requirements for each algorithm.

For example, at each iteration, the DANN algorithm requires:

• 3 models: ["G", "C", "D"]
• 5 tensors labeled: ["src_imgs", "target_imgs", "src_labels", "src_domain", "target_domain"]

These need to be passed to the hook in a single dictionary. In the notebook, this is done by unpacking the model dictionary and data dictionary: hook({**models, **data}).

A similar thing is done in the "PA Issue 81 ResNet" notebook, in the "Training + Evaluation" section:

    for data in tqdm(dataloaders["train"]):
        data = batch_to_device(data, device)
        _, loss = hook({**model_dict, **data})

The reason why data comes in the correct format is because of the wrapper classes used in the "Datasets" section:

# pytorch-adapt wrappers
src_dataset = SourceDataset(real_dataset)
target_dataset = TargetDataset(clipart_dataset)
target_dataset_with_labels = TargetDataset(clipart_dataset, supervised=True)
dataset = CombinedSourceAndTargetDataset(src_dataset, target_dataset)

# create dataloaders
val_names = ["src_train", "target_train", "target_train_with_labels"]
dc = DataloaderCreator(batch_size=16, num_workers=2, val_names=val_names)
dataloaders = dc(train=dataset, src_train=src_dataset, target_train=target_dataset, target_train_with_labels=target_dataset_with_labels)

There are ways to simplify some of the code by using more of this library's helper functions. But if you want to be closer to "raw" pytorch, then the "PA Issue 81 ResNet" notebook is probably best.