Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable CvodeF checkpointing #2645

Merged
merged 1 commit into from
Feb 17, 2025
Merged

Conversation

dweindl
Copy link
Member

@dweindl dweindl commented Feb 16, 2025

amici wants to avoid sundials' checkpointing, that's why we set steps (the number of integration steps at which a checkpoint is created) for CVodeAdjInit to the maximum number of integration steps (mxsteps) for the forward problem.

However, the checkpoint is created at step steps, not at step steps + 1.

Therefore, if the forward integration takes exactly mxsteps, a checkpoint is still created.

This is not a problem per se, but it may trigger segfaults under circumstances not fully understood (-> LLNL/sundials#49). Although unlikely, it still occasionally crashes long running optimizations.

This change should make sure that checkpointing never occurs.

amici wants to avoid sundials' checkpointing, that's we set `steps` (the number of integration steps at which a checkpoint is created) for `CVodeAdjInit` to the maximum number of integration steps (`mxsteps`) for the forward problem.

However, the checkpoint is created at step `steps`, not at step `steps + 1`.

Therefore, if the forward integration takes exactly `mxsteps`, a checkpoint is still created.

This is not a problem per se, but it may trigger segfaults under circumstances not fully understood (-> LLNL/sundials#49).
Although unlikely, it still occasionally crashes long running optimizations.

This change should make sure that checkpointing never occurs.
@dweindl dweindl requested a review from a team as a code owner February 16, 2025 15:24
Copy link

codecov bot commented Feb 16, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.17%. Comparing base (66552d7) to head (70552c5).
Report is 1 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff            @@
##           develop    #2645   +/-   ##
========================================
  Coverage    77.17%   77.17%           
========================================
  Files          332      332           
  Lines        23139    23139           
  Branches      1478     1478           
========================================
  Hits         17857    17857           
  Misses        5271     5271           
  Partials        11       11           
Flag Coverage Δ
cpp 74.10% <100.00%> (ø)
cpp_python 33.38% <100.00%> (ø)
petab 39.41% <0.00%> (ø)
python 72.22% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/solver_cvodes.cpp 70.80% <100.00%> (ø)

@dweindl
Copy link
Member Author

dweindl commented Feb 16, 2025

Although I don't quite understand the details, LLNL/sundials#49 is possibly related to the state reinitialization in amici, which may not be compatible with checkpointing happens independently of any state reinitialization.

Basically, the segfault occurs if the forward integration during the backward pass (CVodeB->CVode) takes more steps than the forward integration during the forward pass (CVodeF->CVode).

@dweindl dweindl added this pull request to the merge queue Feb 17, 2025
Merged via the queue into AMICI-dev:develop with commit 624e9ad Feb 17, 2025
20 checks passed
dweindl added a commit to dweindl/AMICI that referenced this pull request Feb 18, 2025
amici wants to avoid sundials' checkpointing, that's we set `steps` (the number of integration steps at which a checkpoint is created) for `CVodeAdjInit` to the maximum number of integration steps (`mxsteps`) for the forward problem.

However, the checkpoint is created at step `steps`, not at step `steps + 1`.

Therefore, if the forward integration takes exactly `mxsteps`, a checkpoint is still created.

This is not a problem per se, but it may trigger segfaults under circumstances not fully understood (-> LLNL/sundials#49).
Although unlikely, it still occasionally crashes long running optimizations.

This change should make sure that checkpointing never occurs.
@dweindl dweindl deleted the nocheckpoint branch February 27, 2025 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants