Reproducing CIFAR-10 Results

[ardywibowo]

Hello,
I was trying to modify the code to work on the CIFAR-10 dataset and I'm having some trouble. I have changed the following:

1. LR = 0.03
2. Weight Decay = 0.0005
3. Momentum = 0.9
4. Batch Size = 512
5. Removed GaussianBlur and Cropping
6. Removed the 2nd set of Linear, BatchNorm, and ReLU layers in the projector.
7. Modified the normalization parameters to CIFAR-10 statistics.
8. Changed architecture to resnet18. The paper mentions ResNet18 for CIFAR, but I wasn't sure if it's different from resnet18.

But I can't seem to get it to work. I'm getting only about 37% accuracy with both the Linear finetuning and kNN. Do you have any additional implementation details that I may have missed?

[ardywibowo]

Update: Changed the augmentations to include resized cropping to a 32x32 image size and it improved the accuracy. Will update with new results once it finishes training.

Update 2:
Training has just finished and I was able to get a decent 84% accuracy with kNN. I'm still wondering if I'm missing any additional details.

[elias-ramzi]

Hi, could you share the resized cropping you used ?
Thanks !

[ardywibowo]

Sure,
It's the same as the one done in the original code: transforms.RandomResizedCrop(32, scale=(0.2, 1.)).

[endernewton]

For details, it may be helpful to check the demo released in the moco codebase: https://colab.research.google.com/github/facebookresearch/moco/blob/colab-notebook/colab/moco_cifar10_demo.ipynb

[elias-ramzi]

Hi again,

With the CIFAR ResNet18 I am getting 88.4% on knn accuracy. What were your results (on the supplementary there is not the exact number for the knn evaluation) ?

I am not sure on several things (might be details): What is the size you used for the projection head and the predictor ? In the MoCo demo they use the default arguments for RandomResizeCrop what were your values ? Also could there be a difference coming from the fact that I am working on a single GPU (and not using the trick used in the MoCo demo to split the batches) ?

Thanks :)

[ardywibowo]

@elias-ramzi
What did you do with your implementation? I'm only getting 84%. Did you copy the MoCo demo augmentation?

[elias-ramzi]

No I was use using the same augmentations as SimSiam without gaussian blur.
Did you use the ResNet version of the MoCo demo (changed conv1, and remove maxpool) ?

[ardywibowo]

I see. No I didn't. Thanks! Let me try that.

Did you use 128 as the feature_dim or the default 2048? If you changed it to 128, did you change the pred_dim as well to 128/4 = 32?

[elias-ramzi]

In the supplementary material it is said that they used 2048, so I used the default settings for the projection head (with only two layers).

[bhneo]

Hi again,

With the CIFAR ResNet18 I am getting 88.4% on knn accuracy. What were your results (on the supplementary there is not the exact number for the knn evaluation) ?

I am not sure on several things (might be details): What is the size you used for the projection head and the predictor ? In the MoCo demo they use the default arguments for RandomResizeCrop what were your values ? Also could there be a difference coming from the fact that I am working on a single GPU (and not using the trick used in the MoCo demo to split the batches) ?

Thanks :)

Hi bro, how many parameters in your CIFAR ResNet18, I can get only 78% with CIFAR resnet 18, but 89% with resnet 18

[elias-ramzi]

Hi, sorry for the late response:

I have 14580800 parameters for the CIFAR resnet18

[meobach]

hi @ardywibowo , have you tried it with tiny-imagenet dataset, I have trained and used the pre-trained model to do an linear classification but result look really bad

[liuzuo-byte]

我在高光谱数据上跑这个代码的时候，线性分类精度只有55%，请问各位大佬，我需要在哪些地方改进，我的数据增强就是单一的高斯噪声，我的代码有问题吗，还是说我要修改优化器还是啥

您好，我正在尝试修改代码以在CIFAR-10数据集上工作，我遇到了一些麻烦。我更改了以下内容：

1. LR = 0.03
2. 重量衰减 = 0.0005
3. 动量 = 0.9
4. 批大小 = 512
5. 删除了高斯模糊和裁剪
6. 已删除投影仪中的第二组线性、批处理规范和 ReLU 图层。
7. 已将规范化参数修改为 CIFAR-10 统计信息。
8. 已将体系结构更改为 resnet18。这篇论文提到了CIFAR的ResNet18，但我不确定它是否与resnet18不同。

但我似乎无法让它工作。线性微调和 kNN 的准确率只有 37% 左右。你们是否有任何我可能错过的其他实现细节？

你好，你后面是如何提高的线性分类精度呢

[liuzuo-byte]

我在高光谱数据上跑这个代码的时候，线性分类精度只有55%，请问各位大佬，我需要在哪些地方改进，我的数据增强就是单一的高斯噪声，我的代码有问题吗，还是说我要修改优化器还是啥