Deterministic version of CUDA forces and stresses kernels #3693

mtaillefumier · 2024-04-19T14:35:03Z

Calculations of the forces and stress are deterministic on GPU. It does not imply that the DeepMD code is deterministic by default as TensorFlow also requires to be set up properly either at runtime or during the initialization phase.

To obtain the same model parameters, add the following variables to the job scripts

export TF_DETERMINISTIC_OPS=1
export TF_INTER_OP_PARALLELISM_THREADS=0
export TF_INTRA_OP_PARALLELISM_THREADS=0

Details of the changes:

Remove the use of atomic operations in the forces and stress kernels.
Use template programming to minimize code duplication and minor refactoring

Authors :

(@asedova) A. Sedova (ORNL)
(@mtaillefumier) M. Taillefumier (ETH Zurich / CSCS)

See PR #3684 for comments

Summary by CodeRabbit

New Features
- Enhanced force and virial calculations in molecular dynamics simulations for improved accuracy and performance.
- Added a new documentation guide on achieving determinism in DeepMD simulations.
Refactor
- Optimized memory access and thread synchronization in GPU calculations.
Documentation
- Provided detailed guidance on configuring deterministic runs in DeepMD using TensorFlow settings.
Bug Fixes
- Removed deprecated functions to streamline codebase and enhance maintainability.

codecov · 2024-04-19T15:25:19Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.15%. Comparing base (ee47e75) to head (0041399).

Additional details and impacted files

@@           Coverage Diff           @@
##            devel    #3693   +/-   ##
=======================================
  Coverage   82.15%   82.15%           
=======================================
  Files         511      511           
  Lines       47364    47364           
  Branches     2952     2952           
=======================================
  Hits        38910    38910           
  Misses       7561     7561           
  Partials      893      893

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

njzjz · 2024-04-19T20:29:39Z

It seems that the current change cannot make the GPU CI pass.

mtaillefumier · 2024-04-20T07:06:41Z

It seems that the current change cannot make the GPU CI pass.

The errors are unclear to me. It is not a compilation problem (which I would have fixed asap) but something deep in python. One file is just a text file, the two other files are source code. The there is no change in the API, no memory leak (I checked for this), etc.... I can also compile deepmd in containers etc....

NB : I use this docker file

FROM tensorflow/tensorflow:2.16.1-gpu

CMD ["/bin/bash"]
RUN apt-get update -qq && apt-get install -qq --no-install-recommends \
    cmake \
    gcc-12 \
    g++-12 \
    gfortran-12 \
    git
ENV DP_VARIANT=cuda

RUN git clone https://github.com/mtaillefumier/deepmd-kit.git
git checkout devel
WORKDIR deepmd-kit
ENV DP_VARIANT=cuda

RUN pip install .
ENTRYPOINT []

mtaillefumier · 2024-04-21T10:47:18Z

Something is wrong with this patch. It works with master but fails with the devel branch. I am missing something.

coderabbitai · 2024-04-30T13:12:24Z

Warning

Rate Limit Exceeded

@mtaillefumier has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 22 minutes and 59 seconds before requesting another review.

How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.
Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.
Please see our FAQ for further information.

Commits

Files that changed from the base of the PR and between 549ec52 and 0041399.

Walkthrough

The recent updates focus on enhancing CUDA kernels for molecular dynamics simulations by optimizing memory access, thread configurations, and introducing flags for radial calculations. Additionally, a new guide on achieving deterministic results with DeepMD has been introduced, improving reproducibility across different hardware setups.

Changes

Files	Change Summary
`.../gpu/prod_force.cu`	Modified CUDA kernels for force calculations, added boolean flags, and updated function declarations.
`.../gpu/prod_virial.cu`	Refactored virial computation kernels for radial-only calculations and improved synchronization.
`doc/determinism.md`	Introduced guidance on setting TensorFlow environment for deterministic results in DeepMD computations.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

mtaillefumier · 2024-04-30T13:15:37Z

Sorry for the delay. I think I found the root of the problem. A missing synchronization barrier.

coderabbitai

Actionable comments posted: 5

source/lib/src/gpu/prod_virial.cu

source/lib/src/gpu/prod_force.cu

coderabbitai

Actionable comments posted: 3

doc/determinism.md

coderabbitai

Actionable comments posted: 2

doc/determinism.md

coderabbitai

Actionable comments posted: 2

doc/determinism.md

- Remove atomic operations in the forces and stress kernels. - Use template programming to minimize code duplication. Authors : - A. Sedova (ORNL) - M. Taillefumier (ETH Zurich / CSCS)

for more information, see https://pre-commit.ci

coderabbitai

Actionable comments posted: 2

coderabbitai

Actionable comments posted: 2

doc/determinism.md

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Taillefumier Mathieu <[email protected]>

coderabbitai

Actionable comments posted: 2

doc/determinism.md

mtaillefumier · 2024-05-03T07:09:26Z

I found why this PR fails. I missed something important when I modified these kernels and I know how to correct them now. I will close the PR for the moment so that I can push changes without triggering the CI/CD all the time.

github-actions bot added Core CUDA ROCM Docs labels Apr 19, 2024

mtaillefumier mentioned this pull request Apr 19, 2024

Deterministic version of CUDA forces and stresses kernels #3684

Closed

njzjz added the Test CUDA Trigger test CUDA workflow label Apr 19, 2024

github-actions bot removed the Test CUDA Trigger test CUDA workflow label Apr 19, 2024

njzjz requested review from denghuilu, njzjz and wanghan-iapcm and removed request for denghuilu April 19, 2024 20:29

njzjz linked an issue Apr 19, 2024 that may be closed by this pull request

Reproducibility of LAMMPS run with DP potential #3270

Open

njzjz added the Test CUDA Trigger test CUDA workflow label Apr 20, 2024

github-actions bot removed the Test CUDA Trigger test CUDA workflow label Apr 20, 2024

njzjz added the Test CUDA Trigger test CUDA workflow label Apr 21, 2024

github-actions bot removed the Test CUDA Trigger test CUDA workflow label Apr 21, 2024

mtaillefumier force-pushed the determinism branch from 5d042fd to dd7729b Compare April 21, 2024 08:32

njzjz added the Test CUDA Trigger test CUDA workflow label Apr 21, 2024

github-actions bot removed the Test CUDA Trigger test CUDA workflow label Apr 21, 2024

mtaillefumier force-pushed the determinism branch from dd7729b to 39ceb1a Compare April 30, 2024 13:12

coderabbitai bot reviewed Apr 30, 2024

View reviewed changes

source/lib/src/gpu/prod_virial.cu Outdated Show resolved Hide resolved

source/lib/src/gpu/prod_force.cu Outdated Show resolved Hide resolved

coderabbitai bot reviewed Apr 30, 2024

View reviewed changes

doc/determinism.md Outdated Show resolved Hide resolved

doc/determinism.md Outdated Show resolved Hide resolved

coderabbitai bot reviewed Apr 30, 2024

View reviewed changes

doc/determinism.md Outdated Show resolved Hide resolved

coderabbitai bot reviewed Apr 30, 2024

View reviewed changes

doc/determinism.md Outdated Show resolved Hide resolved

doc/determinism.md Outdated Show resolved Hide resolved

mtaillefumier force-pushed the determinism branch 2 times, most recently from e2626d9 to f8a0b31 Compare April 30, 2024 14:03

github-actions bot added Python OP C++ LAMMPS Gromacs Examples i-PI C labels Apr 30, 2024

Deterministic version of CUDA forces and stresses kernels

13c83f8

- Remove atomic operations in the forces and stress kernels. - Use template programming to minimize code duplication. Authors : - A. Sedova (ORNL) - M. Taillefumier (ETH Zurich / CSCS)

mtaillefumier force-pushed the determinism branch from f8a0b31 to 13c83f8 Compare April 30, 2024 14:33

[pre-commit.ci] auto fixes from pre-commit.com hooks

4c206a1

for more information, see https://pre-commit.ci

coderabbitai bot reviewed Apr 30, 2024

View reviewed changes

doc/determinism.md Outdated Show resolved Hide resolved

mtaillefumier and others added 2 commits April 30, 2024 16:45

Update doc/determinism.md

549ec52

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Taillefumier Mathieu <[email protected]>

Update doc/determinism.md

0041399

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Taillefumier Mathieu <[email protected]>

coderabbitai bot reviewed Apr 30, 2024

View reviewed changes

doc/determinism.md Outdated Show resolved Hide resolved

doc/determinism.md Show resolved Hide resolved

njzjz added the Test CUDA Trigger test CUDA workflow label Apr 30, 2024

github-actions bot removed the Test CUDA Trigger test CUDA workflow label Apr 30, 2024

mtaillefumier closed this May 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deterministic version of CUDA forces and stresses kernels #3693

Deterministic version of CUDA forces and stresses kernels #3693

mtaillefumier commented Apr 19, 2024 •

edited by coderabbitai bot

Loading

codecov bot commented Apr 19, 2024 •

edited

Loading

njzjz commented Apr 19, 2024

mtaillefumier commented Apr 20, 2024

mtaillefumier commented Apr 21, 2024

coderabbitai bot commented Apr 30, 2024 •

edited

Loading

Rate Limit Exceeded

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

mtaillefumier commented Apr 30, 2024

coderabbitai bot left a comment

coderabbitai bot left a comment

coderabbitai bot left a comment

coderabbitai bot left a comment

coderabbitai bot left a comment

coderabbitai bot left a comment

coderabbitai bot left a comment

mtaillefumier commented May 3, 2024

Deterministic version of CUDA forces and stresses kernels #3693

Deterministic version of CUDA forces and stresses kernels #3693

Conversation

mtaillefumier commented Apr 19, 2024 • edited by coderabbitai bot Loading

Summary by CodeRabbit

codecov bot commented Apr 19, 2024 • edited Loading

Codecov Report

njzjz commented Apr 19, 2024

mtaillefumier commented Apr 20, 2024

mtaillefumier commented Apr 21, 2024

coderabbitai bot commented Apr 30, 2024 • edited Loading

Rate Limit Exceeded

Walkthrough

Changes

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

mtaillefumier commented Apr 30, 2024

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

mtaillefumier commented May 3, 2024

mtaillefumier commented Apr 19, 2024 •

edited by coderabbitai bot

Loading

codecov bot commented Apr 19, 2024 •

edited

Loading

coderabbitai bot commented Apr 30, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)