How to specify a GPU from all GPUs

[biglinn]

Hi,

I'm going to trian deepmd-kit models by DPGEN workflow. And I have a doubt.
When there are multiple GPUs on the server, Could I specify a specific GPU in the machine.json?

[biglinn]

If the job directly runs at the workstation with multiple GPUs, will there be a situation where different GPUs repeatedly run the job?

Because I find there is repetitive information in the dpgen.log, like this:

[njzjz]

The environment variables CUDA_VISIBLE_DEVICES can be used. There is a lot of information on this environment variable on the Internet.

For setting environment variables in the jobs, refer to https://docs.deepmodeling.com/projects/dpdispatcher/en/latest/resources.html#argument:resources/envs

[njzjz]

Do you use them together? They might provide conflict CUDA_VISIBLE_DEVICES.

[biglinn]

I used them together, while I also used them respectively:
like this:

"strategy": {
    "if_cuda_multi_devices": true
},

and this:

"env": {
    "CUDA_VISIBLE_DEVICES": 2
},

Only the if_cuda_multi_devices dose work.

[biglinn]

Oh, I find this variable is shown as unspecified in the train.log:

Is this setting not valid?

"env": {
    "CUDA_VISIBLE_DEVICES": 2
},

[njzjz]

It's envs instead of env.

[biglinn]

Oh, it is!
Haha, how stupid of me to make this mistake.