The new Deepmdkit v2.2.11 and v3 is much slower and occupies much more GPU memory than before

[Pdeedee]

Did any body encounters the problem?With the same param.json and machine.json, the newer version of Deepmd is slow and occupies more memory.

[wanghan-iapcm]

Could you please provide an example and the version of "faster deepmd-kit" so that we can reproduce the efficiency decrease?

[Pdeedee]

of course! The fast version I use was 2.2.7.
This is the speed I was using 2.2.7 with se_atten_v2,

This is the speed I was using 3.0 with se_atten_v2
.
Moreover, the version of both 2.2.11 and 3.0 occupies 98% memory of 4090, but 2.2.7 only occupies about 2% while training.

[Pdeedee]

An interesting thing I found was that when the GPU is alread with another task, the occupation would decrease to 50%. I observe the occupation decrease when the train.log decrease the neighborlist batch size. Would it probably be the neighborlist build bug.

[Pdeedee]

This is the param.json automatically generated by deepmd 3.0
{
"model": {
"type_map": [
"H",
"C",
"N",
"O"
],
"descriptor": {
"type": "se_atten_v2",
"sel": 120,
"rcut_smth": 0.5,
"rcut": 6.0,
"neuron": [
25,
50,
100
],
"resnet_dt": false,
"axis_neuron": 16,
"seed": 1024089868,
"attn": 128,
"attn_layer": 0,
"attn_dotr": true,
"attn_mask": false,
"precision": "float64",
"_comment2": " that's all"
},
"fitting_net": {
"neuron": [
200,
200,
200
],
"resnet_dt": true,
"seed": 2744276785,
"_comment": " that's all"
},
"_comment": " that's all"
},
"learning_rate": {
"type": "exp",
"decay_steps": 5000,
"start_lr": 0.001,
"stop_lr": 3.51e-08,
"_comment": "that's all"
},
"loss": {
"type": "ener",
"start_pref_e": 0.1,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0.1,
"limit_pref_v": 1,
"_comment": " that's all"
},
"training": {
"numb_steps": 800000,
"seed": 2674951016,
"disp_file": "lcurve.out",
"disp_freq": 2000,
"save_freq": 10000,
"_comment": "that's all",
"training_data": {
"systems": [
"../data.init/cl20nptvasp/deepmdl/H24C24N48O48",
"../data.init/big1/cl20iter7test_set/H24C24N48O48",
"../data.iters/iter.000000/02.fp/data.001",
"../data.iters/iter.000000/02.fp/data.000",
"../data.iters/iter.000001/02.fp/data.001",
"../data.iters/iter.000001/02.fp/data.000",
"../data.iters/iter.000002/02.fp/data.001",
"../data.iters/iter.000002/02.fp/data.000",
"../data.iters/iter.000004/02.fp/data.001",
"../data.iters/iter.000004/02.fp/data.000",
"../data.iters/iter.000005/02.fp/data.001",
"../data.iters/iter.000005/02.fp/data.000",
"../data.iters/iter.000006/02.fp/data.001",
"../data.iters/iter.000006/02.fp/data.000",
"../data.iters/iter.000007/02.fp/data.001",
"../data.iters/iter.000007/02.fp/data.000",
"../data.iters/iter.000008/02.fp/data.001",
"../data.iters/iter.000008/02.fp/data.000",
"../data.iters/iter.000009/02.fp/data.001",
"../data.iters/iter.000009/02.fp/data.000",
"../data.iters/iter.000010/02.fp/data.001",
"../data.iters/iter.000010/02.fp/data.000",
"../data.iters/iter.000011/02.fp/data.001",
"../data.iters/iter.000011/02.fp/data.000"
],
"batch_size": [
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1,
1
]
}
}
}

[njzjz]

when the GPU is alread with another task

Did you test the performance in this situation? Normally a GPU can only do one task at the same time.

[Pdeedee]

With 2.2.7, I can train 2 models in only 1 4090,but with 2.2.11, I can only train 1.

[njzjz]

The memory behavior is desired. See #3699.

[wanghan-iapcm]

did you use the same input for 2.2.7 and 2.2.11? please provide the input files.
it would be grateful if you provide the minimal dataset that reproduce the issue.