Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed an issue where applying DirectSolver to StateIndependentsComp was breaking when used with other linear solvers under MPI. #1020

Merged
merged 13 commits into from
Nov 29, 2023

Conversation

robfalck
Copy link
Contributor

@robfalck robfalck commented Nov 16, 2023

Summary

Previously dymos automatically assigned a DirectSolver to StateIndependentsComp when necessary, but this was problematic depending on the layout of the systems on MPI (mostly due to the presence of distributed systems within the ODE.

Dymos needs solvers on phases in the following conditions:

  • solve_segments is True for one or more states in pseudospectral and Birkhoff transcriptions.
  • pseudospectral phases with input_initial set to True for one or more states
  • duration_balance is used and therefore a residual is associated with the time duration of a phase.
    By default, Dymos will automatically assign NewtonSolver and DirectSolver to the phase if one of these conditions is met.

Furthermore, if a Trajectory has its phases under a ParallelGroup and there are explicit connections between these phases, some form of solver is required to ensure that the inputs/ouputs between the phases are consistent.
By default, Dymos will add NonlinearBlockJac and PETScKrylov as the default solvers in trajectory.phases when this is the case.

The user can bypass this behavior by assigning some other solver to the phase or Trajectory.phases. If the user really truly wants to use a RunOnce solver in these situations, then the new Trajectory and Phase option auto_solvers should be set to False.

Adds new options to Trajectory:

  • parallel_phases dictates whether the top level phase group is a ParallelGroup or a standard Group. Setting this to False allows the user to force phases to run in series when operating in parallel.
  • auto_solvers is True by default which causes dymos to attempt to place solvers where it thinks they are necessary. Setting this to False will disable this behavior.

New options for Phase

  • auto_solvers is True by default which causes dymos to attempt to place solvers where it thinks they are necessary. Setting this to False will disable this behavior.

Related Issues

Backwards incompatibilities

None

New Dependencies

None

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

…e of NonlinearBlockJac/PetscKrylov. The solvers are needed in the presence of MPI regardless of number of procs.
Added auto_solvers options to Trajectory and Phases, allowing users to disable the behavior if desired.
Added parallel_phases option to Trajectory, which toggles whether the top level phase container is a parallel group.
@coveralls
Copy link

coveralls commented Nov 28, 2023

Coverage Status

coverage: 92.025%. first build
when pulling 0e041db on robfalck:i1009_2
into 10c2df0 on OpenMDAO:master.

@@ -684,7 +684,7 @@ def configure_solvers(self, phase):
phase : dymos.Phase
The phase object to which this transcription instance applies.
Copy link
Contributor

@swryan swryan Nov 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just noting the docstring here doesn't include the 'requires_solvers' arg


warn = False
if self._has_connected_phases and isinstance(self.phases, om.ParallelGroup) and MPI and self.comm.size > 1:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since self.comm.size will never be > 1 if MPI is not active, you don't really need the and MPI in the if check.

if requires_solvers is not None:
req_solvers.update(requires_solvers)

reasons_txt = '\n'.join(f' {key}: {val}' for key, val in req_solvers.items())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could move inside the if any(req_solvers.values()) block

if requires_solvers is not None:
req_solvers.update(requires_solvers)

reasons_txt = '\n'.join(f' {key}: {val}' for key, val in req_solvers.items())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intersting that the user can put whatever they want in this sentence. Looks like it also prints out for every phase in the model?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention was to try to include a "reason" in the warning that gets printed out, so allowing the string in here seemed like the quickest way to do it.

And yes while a "user" could do this, in this context "user" really means "transcription developer".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would print out for every phase where this behavior is applicable. I agree that that's somewhat noisy, but users are capable of squelching it by explicitly setting the solver themselves.

warn = True
phase.nonlinear_solver = om.NewtonSolver(iprint=0)
phase.nonlinear_solver.options['solve_subsystems'] = True
phase.nonlinear_solver.options['maxiter'] = 1000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is maxiter being raised from 100 to 1000?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had seen one case where it was taking somewhat more than 100 iterations so I just bumped it up.

@robfalck robfalck merged commit 37ac4e5 into OpenMDAO:master Nov 29, 2023
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Solver-based shooting with a Krylov solver fails due to the DirectSolver attached to indep_states
5 participants