Which layer of DINOv2 do you align with?

[Luciennnnnnn]

Hi, I'm trying to reproduce the training of VA-VAE, which layer of DINOv2 do you align with?

[JingfengYao]

@Luciennnnnnn We simply use the last layer.

[Luciennnnnnn]

@JingfengYao Is this right?

features = self.foundation_model.forward_features(rescale_inputs)["x_norm_patchtokens"]

[Luciennnnnnn]

By the way, how do you align the resolution of latent vector and the feature of DINOv2?

[JingfengYao]

Here are my implementations:

def get_dinov2_encoder():
    """
    Load the DINOv2 pretrained ViT-L encoder from the timm library.
    """
    model = timm.create_model("hf-hub:timm/vit_large_patch14_dinov2.lvd142m", pretrained=True, dynamic_img_size=True)
    model.requires_grad_(False)
    return model

def forward_dinov2(self, x):
    b, c, h, w = x.shape
    if h == 256 and w == 256:
        x = nn.functional.interpolate(x, size=(224, 224), mode='bilinear', align_corners=False)
    return self.model.forward_features(x)[:, 1:].reshape(b, h//16, w//16, -1).permute(0, 3, 1, 2)

[Luciennnnnnn]

Thanks! I want to use a vae with 8x downsampling, what's your opinion on aligning resolution?

[JingfengYao]

@Luciennnnnnn DINOv2 should support resolution between 224 to 518. In my case, I would likely begin by feeding a 448-sized image directly into DINOv2. That said, since this configuration remains untested, its efficacy cannot be ascertained at this stage.

[Luciennnnnnn]

That's sounds reasonable, I see REPA use same strategy.