[MetricCollection/DDP] Get TypeError: _sync_dist() got an unexpected keyword argument 'process_group' when calling compute(). #438
Answered
by
SkafteNicki
Eleven1Liu
asked this question in
Q&A
-
|
Hi, I was trying to run MetricCollection Belows are the python code and the error, is there any suggestions? EnvironmentCUDA 10.1
python 3.6
pytorch-lightning 1.3.5
torch 1.8.0
torchmetrics 0.4.1
Python
trainer = pl.Trainer(logger=False,
num_sanity_val_steps=0,
gpus=torch.cuda.device_count(),
accelerator='ddp',
progress_bar_refresh_rate=0,
max_epochs=args.epochs,
callbacks=[checkpoint_callback, earlystopping_callback])
class Model(pl.LightningModule):
...
def validation_epoch_end(self, step_outputs):
return self._shared_eval_epoch_end(step_outputs, 'val')
def _shared_eval_epoch_end(self, step_outputs, split):
# self.eval_metric is a MetricCollection
metric_dict = self.eval_metric.compute() # error starts from here
self.log_dict(metric_dict)
....
return metric_dict
...ErrorError starts from executing |
Beta Was this translation helpful? Give feedback.
Answered by
SkafteNicki
Aug 10, 2021
Replies: 1 comment 4 replies
-
|
Hi @Eleven1Liu, |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
@SkafteNicki Sure. Thanks for help! Our code depends on an open source package (LibMultiLabel). Please do the following steps:
# run.py
import os
import pytorch_lightning as pl
import torch
from pytorch_lightning.callbacks.early_stopping import EarlyStopping
from pytorch_lightning.callbacks.model_checkpoint import ModelCheckpoint
from pytorch_lightning.utilities.parsing import AttributeDict
from libmultilabel import data_utils
from libmultilabel import networks
from libmultilabel.model import Model
from libmultilabel.utils import init_device
config = AttributeDict({
'data_dir': 'data/rcv1',
'model_name': 'KimCNN',
'init_weight': 'kaiming_uniform',
'network_config': {
'dropout': 0.2,
'filter_sizes': [2, 4, 8],
'num_filter_per_size': 128
},
'embed_file': 'glove.6B.300d',
'val_metric': 'Micro-F1',
'epochs': 2, # for testing
'patience': 5,
'cpu': False,
'batch_size': 64
})
# Load dataset
datasets = data_utils.load_datasets(data_dir=config.data_dir)
# Set up trainer
checkpoint_dir = os.path.join('./checkpoint_dir')
os.makedirs(checkpoint_dir, exist_ok=True)
checkpoint_callback = ModelCheckpoint(dirpath=checkpoint_dir, filename='best_model',
save_last=True, save_top_k=1,
monitor=config.val_metric, mode='max')
earlystopping_callback = EarlyStopping(patience=config.patience,
monitor=config.val_metric, mode='max')
trainer = pl.Trainer(logger=False,
num_sanity_val_steps=0,
gpus=0 if config.cpu else 1,
accelerator='ddp', # if use ddp here
max_epochs=config.epochs,
callbacks=[checkpoint_callback, earlystopping_callback])
# Set up models
word_dict = data_utils.load_or_build_text_dict(dataset=datasets['train'], embed_file=config.embed_file)
classes = data_utils.load_or_build_label(datasets)
network = getattr(networks, config.model_name)(
embed_vecs=word_dict.vectors,
num_classes=len(classes),
**dict(config.network_config)
)
init_weight = networks.get_init_weight_func(init_weight=config.init_weight)
network.apply(init_weight)
model = Model(
classes=classes,
word_dict=word_dict,
network=network,
**dict(config)
)
# Set up dataset loader
device = init_device()
train_loader = data_utils.get_dataset_loader(
data=datasets['train'],
word_dict=model.word_dict,
classes=model.classes,
device=device,
batch_size=config.batch_size
)
val_loader = data_utils.get_dataset_loader(
data=datasets['val'],
word_dict=model.word_dict,
classes=model.classes,
device=device,
batch_size=config.batch_size
)
# Start training
trainer.fit(model, train_loader, val_loader)
model = Model.load_from_checkpoint(checkpoint_callback.best_model_path) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Eleven1Liu, |
Beta Was this translation helpful? Give feedback.
-
|
Hi again, pip install git+https://github.com/PytorchLightning/metrics.git@masteryou should be able to run your code without errors :] |
Beta Was this translation helpful? Give feedback.
-
|
@SkafteNicki It works! Thanks for help :) |
Beta Was this translation helpful? Give feedback.
Answer selected by
Eleven1Liu
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @Eleven1Liu,
Would it be possible for you to send a fully reproduceable script?