RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED when PARSynthesizer fit() in GPU env.

[jalr4ever]

Environment Details

• SDV version: 1.16.1
• Python version: 3.8.19
• Operating System: Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-119-generic x86_64)
• GPU: Tesla V100-SXM2-32GB-LS

Error Description

When CUDA is available, an error occurs when using PARSynthesizer.

sdv/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 1142, in forward
result = _VF.gru(input, batch_sizes, hx, self._flat_weights, self.bias,
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

Steps to reproduce

Just run the code below:

import pandas as pd
from sdv.metadata import SingleTableMetadata
from sdv.evaluation.single_table import evaluate_quality, run_diagnostic
from sdv.sequential import PARSynthesizer
from datetime import datetime


real_data_dict = {
    'equ_id': ['12345678901', '12345678901', '12345678901', '12345678901', '12345678901'],
    'datetime': [
        '2024-09-01 00:00:00',
        '2024-09-01 01:00:00',
        '2024-09-01 02:00:00',
        '2024-09-01 03:00:00',
        '2024-09-01 04:00:00'
    ],
    'load_rate': [10.5, 20.0, 15.3, 22.1, 19.8],
}

real_data = pd.DataFrame(real_data_dict)

print(real_data.head(10))

# Create metadata
metadata = SingleTableMetadata()
metadata.detect_from_dataframe(real_data)
metadata.update_column(column_name='equ_id', sdtype='id', regex_format="^\d{11}$")
metadata.set_sequence_key('equ_id')
metadata.set_sequence_index('datetime')

# fit model
synthesizer = PARSynthesizer(metadata, verbose=True, epochs=64)
synthesizer.fit(real_data)

[jalr4ever]

I downgraded the version of the torch dependency to 2.2.2, and now it runs without errors.

[srinify]

Hi @jalr4ever 👋 I ran your code in Google Colab on a T4 GPU and I didn't run into this error. And we unfortunately don't have access to a Tesla GPU right now.

I"ll keep this issue open for now in the hopes that someone else can reproduce it! That would help us understand what changes we might need to make.