HARD CRASH when loading latest version of the cmip6 tables

[sterlingbaldwin]

I updated my cmip6 tables directory, and now get a buffer overflow crash anytime I call cmor.load_table(TABLE_PATH).

*** buffer overflow detected ***: python terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7fa5a84ef877]
/lib64/libc.so.6(+0x36e2500760)[0x7fa5a84ed760]
/export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cmor/_cmor.cpython-37m-x86_64-linux-gnu.so(cmor_CV_set_att+0x2a9)[0x7fa551b21659]
/export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cmor/_cmor.cpython-37m-x86_64-linux-gnu.so(cmor_CV_set_att+0x257)[0x7fa551b21607]
/export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cmor/_cmor.cpython-37m-x86_64-linux-gnu.so(cmor_CV_set_att+0x257)[0x7fa551b21607]
/export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cmor/_cmor.cpython-37m-x86_64-linux-gnu.so(cmor_CV_set_entry+0x10d)[0x7fa551b25bcd]
/export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cmor/_cmor.cpython-37m-x86_64-linux-gnu.so(cmor_load_table_internal+0x8f9)[0x7fa551b34019]
/export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cmor/_cmor.cpython-37m-x86_64-linux-gnu.so(cmor_load_table+0x251)[0x7fa551b34371]
/export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cmor/_cmor.cpython-37m-x86_64-linux-gnu.so(+0xe9b6)[0x7fa551b0b9b6]
python(_PyMethodDef_RawFastCallKeywords+0x1e0)[0x56295ac5e6f0]
python(_PyCFunction_FastCallKeywords+0x21)[0x56295ac5e891]
python(_PyEval_EvalFrameDefault+0x4ede)[0x56295accbfce]
python(_PyFunction_FastCallKeywords+0xfb)[0x56295ac5dcfb]
python(_PyEval_EvalFrameDefault+0x4b69)[0x56295accbc59]
python(_PyEval_EvalCodeWithName+0x2f9)[0x56295ac0c929]
python(_PyFunction_FastCallKeywords+0x387)[0x56295ac5df87]
python(_PyEval_EvalFrameDefault+0x14dc)[0x56295acc85cc]
python(_PyEval_EvalCodeWithName+0xba9)[0x56295ac0d1d9]
python(_PyFunction_FastCallKeywords+0x387)[0x56295ac5df87]
python(_PyEval_EvalFrameDefault+0x14dc)[0x56295acc85cc]
python(_PyFunction_FastCallKeywords+0xfb)[0x56295ac5dcfb]
python(_PyEval_EvalFrameDefault+0x416)[0x56295acc7506]
python(_PyEval_EvalCodeWithName+0x2f9)[0x56295ac0c929]
python(PyEval_EvalCodeEx+0x44)[0x56295ac0d7e4]
python(PyEval_EvalCode+0x1c)[0x56295ac0d80c]
python(+0x1e0c70)[0x56295acd6c70]
python(_PyMethodDef_RawFastCallKeywords+0xe9)[0x56295ac5e5f9]
python(_PyCFunction_FastCallKeywords+0x21)[0x56295ac5e891]
python(_PyEval_EvalFrameDefault+0x47d4)[0x56295accb8c4]
python(_PyEval_EvalCodeWithName+0x2f9)[0x56295ac0c929]
python(_PyFunction_FastCallKeywords+0x325)[0x56295ac5df25]
python(_PyEval_EvalFrameDefault+0x416)[0x56295acc7506]
python(_PyEval_EvalCodeWithName+0x2f9)[0x56295ac0c929]
python(_PyFunction_FastCallDict+0x1d5)[0x56295ac0d9f5]
python(+0x228727)[0x56295ad1e727]
python(+0x23b0f5)[0x56295ad310f5]
python(_Py_UnixMain+0x3c)[0x56295ad311dc]
/lib64/libc.so.6(__libc_start_main+0x100)[0x7fa5a840bd20]
python(+0x1e0d3d)[0x56295acd6d3d]
======= Memory map: ========
56295aaf6000-56295ab51000 r--p 00000000 00:14 56019098                   /export/baldwin32/anaconda3/envs/cwl/bin/python3.7
56295ab51000-56295ad32000 r-xp 0005b000 00:14 56019098                   /export/baldwin32/anaconda3/envs/cwl/bin/python3.7
56295ad32000-56295add9000 r--p 0023c000 00:14 56019098                   /export/baldwin32/anaconda3/envs/cwl/bin/python3.7
56295adda000-56295addd000 r--p 002e3000 00:14 56019098                   /export/baldwin32/anaconda3/envs/cwl/bin/python3.7
56295addd000-56295ae46000 rw-p 002e6000 00:14 56019098                   /export/baldwin32/anaconda3/envs/cwl/bin/python3.7
56295ae46000-56295ae67000 rw-p 00000000 00:00 0 
56295bda4000-56295de25000 rw-p 00000000 00:00 0                          [heap]
7fa3c8000000-7fa3c8021000 rw-p 00000000 00:00 0 
7fa3c8021000-7fa3cc000000 ---p 00000000 00:00 0 
7fa3d0000000-7fa3d0021000 rw-p 00000000 00:00 0 
7fa3d0021000-7fa3d4000000 ---p 00000000 00:00 0 
7fa3d4000000-7fa3d4021000 rw-p 00000000 00:00 0 
7fa3d4021000-7fa3d8000000 ---p 00000000 00:00 0 
7fa3d8000000-7fa3d8021000 rw-p 00000000 00:00 0 
7fa3d8021000-7fa3dc000000 ---p 00000000 00:00 0 
7fa3dc000000-7fa3dc021000 rw-p 00000000 00:00 0 
7fa3dc021000-7fa3e0000000 ---p 00000000 00:00 0 
7fa3e0000000-7fa3e0021000 rw-p 00000000 00:00 0 
7fa3e0021000-7fa3e4000000 ---p 00000000 00:00 0 
7fa3e4000000-7fa3e4021000 rw-p 00000000 00:00 0 
7fa3e4021000-7fa3e8000000 ---p 00000000 00:00 0 
7fa3e8000000-7fa3e8021000 rw-p 00000000 00:00 0 
7fa3e8021000-7fa3ec000000 ---p 00000000 00:00 0 
7fa3ec000000-7fa3ec021000 rw-p 00000000 00:00 0 
7fa3ec021000-7fa3f0000000 ---p 00000000 00:00 0 
7fa3f0000000-7fa3f0021000 rw-p 00000000 00:00 0 
7fa3f0021000-7fa3f4000000 ---p 00000000 00:00 0 
7fa3f4000000-7fa3f4021000 rw-p 00000000 00:00 0 
7fa3f4021000-7fa3f8000000 ---p 00000000 00:00 0 
7fa3f8000000-7fa3f8024000 rw-p 00000000 00:00 0 
7fa3f8024000-7fa3fc000000 ---p 00000000 00:00 0 
7fa3fdab2000-7fa3fdc32000 rw-p 00000000 00:00 0 
7fa3fdd73000-7fa3fddb3000 rw-p 00000000 00:00 0 
7fa3fddb3000-7fa3fddb6000 r--p 00000000 00:14 55971743                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/unicodedata.cpython-37m-x86_64-linux-gnu.so
7fa3fddb6000-7fa3fddbb000 r-xp 00003000 00:14 55971743                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/unicodedata.cpython-37m-x86_64-linux-gnu.so
7fa3fddbb000-7fa3fde9c000 r--p 00008000 00:14 55971743                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/unicodedata.cpython-37m-x86_64-linux-gnu.so
7fa3fde9c000-7fa3fde9d000 r--p 000e8000 00:14 55971743                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/unicodedata.cpython-37m-x86_64-linux-gnu.so
7fa3fde9d000-7fa3fdeba000 rw-p 000e9000 00:14 55971743                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/unicodedata.cpython-37m-x86_64-linux-gnu.so
7fa3fdeba000-7fa3fdffa000 rw-p 00000000 00:00 0 
7fa3fdffa000-7fa3fdffc000 r--p 00000000 00:14 55971725                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/_json.cpython-37m-x86_64-linux-gnu.so
7fa3fdffc000-7fa3fe007000 r-xp 00002000 00:14 55971725                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/_json.cpython-37m-x86_64-linux-gnu.so
7fa3fe007000-7fa3fe009000 r--p 0000d000 00:14 55971725                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/_json.cpython-37m-x86_64-linux-gnu.so
7fa3fe009000-7fa3fe00a000 r--p 0000e000 00:14 55971725                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/_json.cpython-37m-x86_64-linux-gnu.so
7fa3fe00a000-7fa3fe00b000 rw-p 0000f000 00:14 55971725                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/lib-dynload/_json.cpython-37m-x86_64-linux-gnu.so
7fa3fe00b000-7fa3fe3ed000 rw-p 00000000 00:00 0 
7fa3fe3ed000-7fa3fe3f2000 r-xp 00000000 08:01 121634919                  /lib64/libnss_dns-2.12.so
7fa3fe3f2000-7fa3fe5f1000 ---p 00005000 08:01 121634919                  /lib64/libnss_dns-2.12.so
7fa3fe5f1000-7fa3fe5f2000 r--p 00004000 08:01 121634919                  /lib64/libnss_dns-2.12.so
7fa3fe5f2000-7fa3fe5f3000 rw-p 00005000 08:01 121634919                  /lib64/libnss_dns-2.12.so
7fa3fe5f3000-7fa3fe600000 r-xp 00000000 08:01 121634921                  /lib64/libnss_files-2.12.so
7fa3fe600000-7fa3fe7ff000 ---p 0000d000 08:01 121634921                  /lib64/libnss_files-2.12.so
7fa3fe7ff000-7fa3fe800000 r--p 0000c000 08:01 121634921                  /lib64/libnss_files-2.12.so
7fa3fe800000-7fa3fe801000 rw-p 0000d000 08:01 121634921                  /lib64/libnss_files-2.12.so
7fa3fe80e000-7fa3fe810000 r--p 00000000 00:14 55993029                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cdms2/_bindex.cpython-37m-x86_64-linux-gnu.so
7fa3fe810000-7fa3fe813000 r-xp 00002000 00:14 55993029                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cdms2/_bindex.cpython-37m-x86_64-linux-gnu.so
7fa3fe813000-7fa3fe814000 r--p 00005000 00:14 55993029                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cdms2/_bindex.cpython-37m-x86_64-linux-gnu.so
7fa3fe814000-7fa3fe815000 r--p 00005000 00:14 55993029                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cdms2/_bindex.cpython-37m-x86_64-linux-gnu.so
7fa3fe815000-7fa3fe816000 rw-p 00006000 00:14 55993029                   /export/baldwin32/anaconda3/envs/cwl/lib/python3.7/site-packages/cdms2/_bindex.cpython-37m-x86_64-linux-gnu.so
7fa3fe816000-7fa3fec97000 rw-p 00000000 00:00 0 
7fa3fec97000-7fa3feca1000 r--p 00000000 00:14 55729645                   /export/baldwin32/anaconda3/envs/cwl/lib/libgomp.so.1.0.0
7fa3feca1000-7fa3fecb8000 r-xp 0000a000 00:14 55729645                   /export/baldwin32/anaconda3/envs/cwl/lib/libgomp.so.1.0.0
7fa3fecb8000-7fa3fecc1000 r--p 00021000 00:14 55729645                   /export/baldwin32/anaconda3/envs/cwl/lib/libgomp.so.1.0.0
7fa3fecc1000-7fa3fecc2000 ---p 0002a000 00:14 55729645                   /export/baldwin32/anaconda3/envs/cwl/lib/libgomp.so.1.0.0

[mauzey1]

@sterlingbaldwin This is a problem I mentioned in PCMDI/cmip6-cmor-tables#259. New entries in CMIP6_CV.json have strings with a length greater than 1024, the maximum length of strings in CMOR. Shortening the strings to 1024 characters seems to get rid of this bug. I will see why long strings in the table are causing this problem.

[durack1]

Thanks @sterlingbaldwin this is a dupe of #530 as defined in PCMDI/cmip6-cmor-tables#259

[sterlingbaldwin]

Looks like you can just bump the CMOR_MAX_STRING value defined here:

cmor/include/cmor.h

Line 11 in 10716a2

#define CMOR_MAX_STRING 1024

[mauzey1]

@sterlingbaldwin I tried increasing that number to 2048 but it caused errors. There must be other variables that rely on it being 1024.

[durack1]

@mauzey1 @sterlingbaldwin it would be nice to fix this 1024 char issue, alongside the proposed solution catching issues with PCMDI/cmip6-cmor-tables#261

[taylor13]

As noted above, this is also discussed in #530

[taylor13]

I think that simply increasing 1024 to 2048 is unacceptable (even if you could make it work). The size of the internal table is doubled by doing this, and it's already huge.

[dnadeau4]

I think CMOR2 loaded the table on the Stack memory and not the Heap. In CMOR3 the CV files is loaded on the Heap memory, so you can make it has big as you want. You might be able to find some Compiler flag to increase the Stack memory.

You can also load the table dynamically on the Heap memory using "malloc", but it involves a change in CMOR architecture.

[dnadeau4]

https://github.com/PCMDI/cmor/blob/master/Src/cmor.c#L87-L90

[sterlingbaldwin]

This issue has been resolved and can be closed. Although the underlying issue of the hard coded max character limit is still there, it shouldnt cause any problems as long as a) requests to add super long strings to the table data are rejected and, b) things arent automatically merged into master when the tests are failing.