Facing bug when adapt Vader to dynamicrafter

[Caixy1113]

Hi,im trying to adapt the Vader to dynamicrafter, but I facing some bug when accelerator.backward(loss),here is the error,.
torch.utils.checkpoint: Recomputed values for the following tensors have different metadata than during the forward pass. tensor at position 8: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 9: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 10: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 11: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 12: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 13: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 14: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 15: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 16: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 17: saved metadata: {'shape': torch.Size([320, 64, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 18: saved metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 19: saved metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 20: saved metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 21: saved metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 22: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 23: saved metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 64, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 24: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 25: saved metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 26: saved metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 27: saved metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 29: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 30: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 31: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 32: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 33: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 34: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 35: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 36: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 37: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 38: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 39: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 40: saved metadata: {'shape': torch.Size([320, 64, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 41: saved metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 42: saved metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 43: saved metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 44: saved metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 45: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 46: saved metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 47: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 48: saved metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 49: saved metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 50: saved metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 51: saved metadata: {'shape': torch.Size([320, 2560]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 52: saved metadata: {'shape': torch.Size([64, 2, 1280]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 64, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 53: saved metadata: {'shape': torch.Size([64, 2, 1280]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 54: saved metadata: {'shape': torch.Size([64, 2, 1280]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 55: saved metadata: {'shape': torch.Size([1280, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} File "/home/cxy/DynamiCrafter/scripts/main/train_t2v_lora.py", line 982, in run_training accelerator.backward(loss) File "/home/cxy/DynamiCrafter/scripts/main/train_t2v_lora.py", line 1143, in <module> run_training(args) torch.utils.checkpoint.CheckpointError: torch.utils.checkpoint: Recomputed values for the following tensors have different metadata than during the forward pass. tensor at position 8: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 9: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 10: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 11: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 12: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 13: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 14: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 15: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 16: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 17: saved metadata: {'shape': torch.Size([320, 64, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 18: saved metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 19: saved metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 20: saved metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 21: saved metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 22: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 23: saved metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 64, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 24: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 25: saved metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 26: saved metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 27: saved metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 29: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 30: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 31: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 32: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 33: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 34: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 35: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 36: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 37: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 38: saved metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 39: saved metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 40: saved metadata: {'shape': torch.Size([320, 64, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 41: saved metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 42: saved metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 43: saved metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 44: saved metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 45: saved metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 46: saved metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 47: saved metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([64, 2, 320]), 'dtype': torch.bool, 'device': device(type='cuda', index=0)} tensor at position 48: saved metadata: {'shape': torch.Size([320]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 49: saved metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 50: saved metadata: {'shape': torch.Size([64, 2, 1]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([4, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 51: saved metadata: {'shape': torch.Size([320, 2560]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([128, 4]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 52: saved metadata: {'shape': torch.Size([64, 2, 1280]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 64, 2]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 53: saved metadata: {'shape': torch.Size([64, 2, 1280]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} tensor at position 54: saved metadata: {'shape': torch.Size([64, 2, 1280]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 2]), 'dtype': torch.float32, 'device': device(type='cuda', index=0)} tensor at position 55: saved metadata: {'shape': torch.Size([1280, 320]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)} recomputed metadata: {'shape': torch.Size([320, 2, 64]), 'dtype': torch.float16, 'device': device(type='cuda', index=0)}

[mihirp1998]

@Caixy1113 is this fixed?