Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

arm64 Darwin behaviour #58

Open
dthierry opened this issue May 17, 2021 · 12 comments
Open

arm64 Darwin behaviour #58

dthierry opened this issue May 17, 2021 · 12 comments
Assignees
@dthierry

Comments

@dthierry
Copy link
Owner

As Kevin has reported sometimes, k_aug does not behave as expected in other systems.
This requires a bit more testing, particularly with Pyomo.
@dthierry dthierry self-assigned this May 17, 2021
@kwmcbride
Copy link

For clarification, this issue does not appear on a machine running Linux Mint (the latest version at this time). I will be testing this on a Windows machine in the next couple of days to see if that works too.

At the moment, when I try to use k_aug, all problems fail with a similar message as the following (MacOS 11.2.3, M1):

ReactionModel: Generating base model (self._model)
# TemplateBuilder: Preparing model for p_estimator
# TemplateBuilder: Preparing model for None
WARNING: More finite elements were found in ContinuousSet 'alltime' than the
    number of finite elements specified in apply. The larger number of finite
    elements will be used.
meas_scaling 0.05
W[K_AUG]...	[K_AUG_ASL]No n_rhs declared.
W[K_AUG]...	[K_AUG_ASL]Target log10mu:= -11.
W[K_AUG]...	[K_AUG_ASL]This problem has no degrees of freedom
Pass the option square_override for the desired behaviour
W[K_AUG_ASL]...	[K_AUG_ASL]No ipopt_zL_in suffix declared, setting zL = 0.
W[K_AUG_ASL]...	[K_AUG_ASL]No ipopt_zU_in suffix declared, setting zU = 0.
W[K_AUG]...	[K_AUG_ASL]No f_timestamp suffix declared, Fallback to default writing mode.
I[K_AUG]...	[K_AUG_ASL] Filename for dot_sens dsdp_in_.in
W[K_AUG]...	[K_AUG_ASL]dsdp for linear C(x) + I*p = 0 override.
[K_AUG] 0.1.0, Part of the IDAES PSE framework
Please visit https://idaes.org/
: dsdp_mode=
dsdp_mode=
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]Attempting to make i_pert->d_c > 0.
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
W[K_AUG]...	[INERTIA_STRATEGY]Wrong inertia(neig < m).
W[K_AUG]...	[INERTIA_STRATEGY]d_c is already > 0
The Jacobian might be singular. Asking for better accuracy..
E[K_AUG]...	[MA57_FACTOR]Failure, pivot tol is at it maximum.
W[K_AUG]...	[MA57_FACTOR]Inexact solution is now activated[Warning: results might not be good].
I[K_AUG]...	[K_AUG_ASL]File read successfully.
I[K_AUG]...	[K_AUG_ASL]Number of Right hand sides: 0
I[K_AUG]...	[K_AUG_ASL]Number of variables       : 10810
I[K_AUG]...	[K_AUG_ASL]Number of constraints     : 10810
I[K_AUG]...	[K_AUG_ASL]Number of valid n_dof     : 0
I[K_AUG]...	[ADJUST_MU]	Warning no relevant info from the problem can predict logmu
I[K_AUG]...	[FIND_INEQUALITIES]summary: eq: 10810, leq: 0, geq: 0 
I[K_AUG]...	[K_AUG_ASL]Nonzeroes in the sparse Jacobian 48610
I[K_AUG]...	[GET_HESS_ASL]Objective found
I[K_AUG]...	[GET_HESS_ASL]Nonzeroes in the sparse hessian 6905
I[K_AUG]...	[GET_HESS_ASL]Minimization problem detected
I[K_AUG]...	[GET_HESS_ASL]Current objective 121840159476.264603
I[K_AUG]...	[GET_HESS_ASL]Missing nz in the Hessian of the Lag: 9305
I[K_AUG]...	[K_AUG_ASL]Barrier term added.
I[K_AUG]...	[K_AUG_ASL]MC19 scaling...
I[K_AUG]...	[ASSM_RHS_DCDP]According to the suffixes declared len p is 4 
I[MA57]...	[]***
 Runtime parameters:
   Objective type: METIS_OBJTYPE_VOL
   Coarsening type: Unknown!
   Initial partitioning type: METIS_IPTYPE_GROW
   Refinement type: Unknown!
   Perform a 2-hop matching: Yes
   Number of balancing constraints: 1
   Number of refinement iterations: 227025648
   Random number seed: 304157472
   Number of separators: 1
   Compress graph prior to ordering: Yes
   Detect & order connected components separately: Yes
   Prunning factor for high degree vertices: 21488882.000000
   Allowed maximum load imbalance: 1.004 

Input Error: Incorrect objective type.
 nbrpool statistics
        nbrpoolsize:            0   nbrpoolcpos:            0
    nbrpoolreallocs:            0

I[MA57]...	[ma57bd_]ma57ad_: Status Ok.
I[MA57]...	[MA57_FACTOR]
I[K_AUG]...	[MA57_FACTOR]n_neig = 0
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.000001
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.000032
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.000422
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.002943
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.012635
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.037685
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.085532
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.158160
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.250797
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.354399
I[K_AUG]...	[MA57_FACTOR]n_neig = 10662
I[K_AUG]...	[MA57_FACTOR]Increasing pivtol to fix inertia. pivot_tol 0.459324
I[MA57]...	[Factorize Success.]***
I[MA57]...	[MA57_RATIO]
I[K_AUG]...	[MA57_SOLVE]: Ratio of norm of scaled residuals (reported); NaN 
I[K_AUG]...	[MA57_SOLVE]Accuracy at an acceptable level.

E[MA57]...	[Inertia check Failure!!!.]ERROR: Solver (asl) returned non-zero return code (255)
ERROR: See the solver log above for diagnostic information.
Traceback (most recent call last):

  File "/Users/kevinmcbride/Development/kipet_units/kipet_examples/examples/example_8/Ex_8_estimability.py", line 74, in <module>
    listparams = r1.e_analyzer.rank_params_yao(meas_scaling=meas_uncertainty,

  File "/Users/kevinmcbride/Development/kipet_units/kipet/kipet/estimability_tools/estimability_analysis.py", line 279, in rank_params_yao
    dsdp, idx_to_param = self.get_sensitivities_for_params(tee=True, sigmasq=sigmas)

  File "/Users/kevinmcbride/Development/kipet_units/kipet/kipet/estimability_tools/estimability_analysis.py", line 199, in get_sensitivities_for_params
    k_aug.solve(m, tee=True)

  File "/Users/kevinmcbride/Development/kipet_units/kipet/.venv/lib/python3.8/site-packages/pyomo/opt/base/solvers.py", line 601, in solve
    raise ApplicationError(

ApplicationError: Solver (asl) did not exit normally

@kwmcbride
Copy link

Another issue with this system is that the kaug_hess folder does not appear at all. This works fine on my current Linux machine. I will be testing this on windows too.

@dthierry
Copy link
Owner Author

It seems your problem has a (close to) singular Jacobian, so the back solve fails.
It can happen because of several reasons, e.g. if the initial point is not the same.
I've found that using the ThirdParty-Metis from coin-or has some weird behaviour in the arm64 Macs. So it has to be deactivated somehow. In other words, this could be a) numerical issue or b) Mac M1 thing.

If you generate the .nl file of this particular run, I can analyse the numerical part of it, because we can run it, in say linux, and see if it fails as well. The Metis thing might be more difficult to understand for now.

As for the folder, I've tried the print_kkt option on Mac 11.4 M1 and it does generate the GJH folder with the matrices does it not generate any folder at all in your system? E.g. try running k_aug somenlfile.nl print_kkt in the terminal and see if the folder is generated.

@kwmcbride
Copy link

It's probably Metis on the M1 then. I can run these problems without issue on the linux machine.

I'll try to generate the nl file and get it to you.

I can generate the GJH folder by using k_aug in the terminal.

@kwmcbride
Copy link

I found out that if the option "deb_kkt" is used at the same time as "print_kkt", the GJH folder is not created. This solves part of the problem.

@kwmcbride
Copy link

Okay, at least the reduced hessian parameter selection part works on MacOS now after making adjustments due to the different structure of the output info.

The others still produce the error shown above. I emailed you a copy of the nl file.

@kwmcbride
Copy link

Another point I should make is that k_aug is returning confidence intervals that are basically zero width (mac os). This may be related to Metis.

Sipopt is working though and can be used in the meantime.

@kwmcbride
Copy link

The problem in the last post is fixed. Kipet depended heavily on the old debug files and their absence lead to a chain of problems. I converted my reduced hessian methods to account for this.

@dthierry
Copy link
Owner Author

I've tried the nl file in Mac and I basically get the same situation as you've reported. I've also tried it on Linux and so forth and I get the same failure.
I guess the starting point is different in linux, in other words the local solution from the optimization phase is different. It would make sense if k_aug fails in every platform, otherwise the the local solutions from Ipopt are platform dependent, which is weird but totally possible.
I also noticed that the objective value is very large in comparison with the rest of the problem, so perhaps having different scaling within kipet might help.

@kwmcbride
Copy link

kwmcbride commented Jun 25, 2021 •
edited
Loading

That is interesting. I'll have to run those examples again then.

I have been rewriting a bunch of reduced hessian code to make Kipet work with k_aug. I have finally tackled the covariance issue when using multiple datasets. I now run into this issue:

Solver solution file: '/var/folders/by/rv603c9d51g2trgp9l5bdbj00000gn/T/tmpswqc0zw2.pyomo.sol'
Solver problem files: ('/var/folders/by/rv603c9d51g2trgp9l5bdbj00000gn/T/tmpswqc0zw2.pyomo.nl',)
        0.41 seconds required for solver
        0.00 seconds required to read logfile 
        0.03 seconds required to read solution file
        0.09 seconds required for postsolve

W[K_AUG]...	[K_AUG_ASL]No n_rhs declared.
W[K_AUG]...	[K_AUG_ASL]Target log10mu:= -11.
[K_AUG] 0.1.0, Part of the IDAES PSE framework
Please visit https://idaes.org/
: print_kkt=
print_kkt=
bad line 71855 of /var/folders/by/rv603c9d51g2trgp9l5bdbj00000gn/T/tmpgf4xwy7o.pyomo.nl: d28724
I[K_AUG]...	[K_AUG_ASL]File read successfully.
I[K_AUG]...	[K_AUG_ASL]Number of Right hand sides: 0
I[K_AUG]...	[K_AUG_ASL]Number of variables       : 14366
I[K_AUG]...	[K_AUG_ASL]Number of constraints     : 14364
I[K_AUG]...	[K_AUG_ASL]Number of valid n_dof     : 2
ERROR: Solver (asl) returned non-zero return code (1)
ERROR: See the solver log above for diagnostic information.

I assume this is a modeling error on my part and not related to the error above. Can you confirm this? I don't know what is meant by the d28724 being considered a bad line.

@dthierry
Copy link
Owner Author

I think something is wrong when the nl file is being written. Like passing something that is not a float/integer in your constraint expressions and so forth.
I bet that if you check your printed model you'd find a problematic expression. Or you could just open the nl file and go to the line 71855 and see if there's something there that points you to the model itself.

@kwmcbride
Copy link

You can close this now. I found out what the problem was and have no issues now. The error above is caused when using more than one collocation point. Other than that, the estimability code is working again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dthierry dthierry

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dthierry @kwmcbride