SIGSEGV: Segmentation fault with IED_INIT_MODE=6

[Aariq]

I'm using PEcAn to run ED2 and hoping to use IED_INIT_MODE=6 to start runs with already grown communities. After successfully initializing and simulating for several months, I get the following uninformative error:

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x2b6a6e9f8d5a
#1  0x2b6a6e9f7ef5
#2  0x2b6a6ee7b08f
#3  0x55f1df4ffb41
#4  0x55f1df44fc41
#5  0x55f1df1176c3
#6  0x55f1df0b3b03
#7  0x55f1df0b2abe
#8  0x2b6a6ee5c082
#9  0x55f1df0b2aed
#10  0xffffffffffffffff

The same runs with IED_INIT_MODE=0 run fine.

The .css and .pss files I'm using were generated with an older version of ED2 and don't have the dummy columns needed for IED_INIT_MODE=6, but since the runs get started just fine, I'm thinking that's not likely the issue. I'm also running ED2 in a singularity container on an HPC using the PEcAn modellauncher, if that's helpful. I'm not sure what to try next for debugging this. Any help would be appreciated!

[mpaiao]

This is likely to be some variable initialisation problem (for example, trying to access element 0 or NaN in an array), although ED2 would need to be compiled with debugging options to really get some information about the issue. The default containerised model is compiled for speed, not for detecting errors, so often the code can continue to run for a while even when there are invalid numbers.

However, you mentioned that the pss/css files lack the some columns in the IED_INIT_MODE=6 format. If the number of columns in the pss/css files does not match what ED2 expects, the code may be reading the files incorrectly (like reading more than a line every time it tries to read a line), and there is a chance this is leading to errors. Even if it doesn't cause the crash, it may lead to a vegetation structure that is different from what you intend to simulate...

I recommend editing your pss/css files to include the columns listed in the Wiki in the exact same order as they are described.

[Aariq]

If this was an initialization problem, would simulations be able to run? The simulation runs for 6 months before throwing this error. I'll definitely edit the pss/css files though and see if that helps.

[mpaiao]

Yes, it could happen because Fortran skips many of the safety checks when the code is compiled with -O3.

Actually, the simulation time when the code crashes can be very telling: if the crash happened on the first day of the month, it is likely an initialisation problem that is not triggered until the code tries to update the number of cohorts or patches.

[Aariq]

That is exactly what happens. The last day simulated is the last day of june.

[Aariq]

That was indeed the problem. Now I'm getting new error messages!