Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem with NCDF_VARGET call under GDL ... Problem reading Unsigned Byte Arrays #1983

Open
phubanks opened this issue Feb 7, 2025 · 19 comments
Open

Problem with NCDF_VARGET call under GDL ... Problem reading Unsigned Byte Arrays #1983

phubanks opened this issue Feb 7, 2025 · 19 comments

Comments

@phubanks
Copy link

phubanks commented Feb 7, 2025

Hello GDL Support Team,

Thanks for all your hard work over there. You guys and gals have been very responsive so far about issues with GDL. And I really appreciate it.

I am having trouble running a short IDL program under GDL (version 1.0.5) which is readng a NetCDF4 file

I am getting the following error under (GDL) ... meanwhile my code works fine with no errors under IDL
It appears from what i've been able to debug ... that the issue occurs when i try to read an
UNSIGNED BYTE array with values that range from 0 to 255

(note that signed byte arrays have values that range from -128 to 127, and those arrays work fine)

The GDL error occurs when i use the following command ... to read in an unsigned byte array with values ranging from 0 to 255.

NCDF_VARGET, groupid, varid, data

Its interesting but this call to NCDF_VARGET works FINE under IDL but fails under GDL.

I can send you my PRO program and testing NetCDF4 file that you can download if you give me your email address, or provide me another way to upload a file to you. Note the NetCDF4 file is big at nearly 500MB.

So one question i have is ... perhaps there is some OPTION to NCDF_VARGET under GDL which is missing? I cant find anything in the manual about that.

My guess is that NCDF_VARGET call under GDL is demanding that the byte array be signed and that the values stay in the -128 to 127 range ... but when it sees an array with values ranging from 0 to 255 (as occurs in an unsigned byte array) that call chokes. This is not an issue under IDL but fails under GDL, so i think there might be a bug in a call under GDL? or if not perhaps I am not calling this function correctly?

Any help or insight you can offer would be fantastic.

Some output from my program is show below ... just before the program errors and quits:

varname = Quality_Assurance

  Get Info on Var to be read with:   NCDF_VARINQ(groupid, varid) 
  More info about this variable about to be read 
   Possible DataTypes: BYTE, CHAR, INT, LONG, FLOAT, DOUBLE 
    { VarName, DataType, NDims, NumAtts, Dimensions_ID_Vector }

NCDF_VARINQ Result = { Quality_Assurance BYTE 3 11 2 0 1 }

Reading Data with: NCDF_VARGET, groupid, varid, data

% Warning in NCDF_VARGET: NC_ERANGE during BYTE reading
% NCDF_VARGET: (NC_ERROR=-60)

% Execution halted at: READ_ATMOS_L2_NC 184 /home/phubanks/_m7_profile_7.0.3/read_atmos_l2_nc.pro
% WRITE_YORI_INPUT_PROFILE_GDL 489 /home/phubanks/_m7_profile_7.0.3/write_yori_input_profile_gdl.pro
% $MAIN$

GDL> exit
@phubanks
Copy link
Author

phubanks commented Feb 7, 2025

if you send me your email i can send you the test file and the PRO code.
Of if you can provide another way to get the file and the PRO code to you.

Thank you so much!

@phubanks
Copy link
Author

phubanks commented Feb 7, 2025

the PRO code runs very fast. Just a few seconds. So you can easily try running it on your end. Works great under IDL ... Note that I can read unsigned byte arrays (0 to 255) with that call, with no issues under IDL

The same read of an unsigned byte array (0 to 255) fails with these erros under GDL
% Warning in NCDF_VARGET: NC_ERANGE during BYTE reading
% NCDF_VARGET: (NC_ERROR=-60)

I think under GDL that call is expecting byte values to always range from -128 to 127 (as it would for a signed byte arrays) so when it encounters an unsigned byte array with values from 0 to 255, it chokes. If you could change that call to allow for values up to 255 ... instead of up to 127 ... i think it would solve the problem.
Maybe. But you guys will know best. Its possible i need some other unknown keyword set under GDL, which i didnt need under IDL. But I cant find any references alluding to that.

THanks for looking at this!

@phubanks
Copy link
Author

phubanks commented Feb 7, 2025

pro read_mod06_testbyte

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;; Test Read Program
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

DEBUG = 1

ncid = NCDF_OPEN('MOD06_L2.A2003001.2140.007.2023121190627.nc')

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Use NCDF_NCIDINQ to drill down through groups
group_id1 = NCDF_NCIDINQ(ncid, 'HDFEOS')
if(DEBUG EQ 1) then print,' group_id1 = ', group_id1

group_id2 = NCDF_NCIDINQ(group_id1, 'SWATHS')
if(DEBUG EQ 1) then print,' group_id2 = ', group_id2

group_id3 = NCDF_NCIDINQ(group_id2, 'mod06')
if(DEBUG EQ 1) then print,' group_id3 = ', group_id3

group_id4 = NCDF_NCIDINQ(group_id3, 'Data Fields')
if(DEBUG EQ 1) then print,' group_id4 = ', group_id4

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Set the master group id, where the variables are located
groupid = group_id4

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; TRY READING IN 3 VARIABLES !!
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; FIRST VARIABLE
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Get the Variable ID
if(DEBUG EQ 1) then print,' '
if(DEBUG EQ 1) then print,' '
if(DEBUG EQ 1) then print,' Reading Short Variable: Cloud_Effective_Radius'
if(DEBUG EQ 1) then print,' Valid Data Range: 400 to 6000'
if(DEBUG EQ 1) then print,' Fill Value: -9999'
varid = NCDF_VARID(group_id4, 'Cloud_Effective_Radius')
if(DEBUG EQ 1) then print,' varid = ', varid

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Lets get more information about this variable about to be read in
Result = NCDF_VARINQ(groupid, varid)

  if(DEBUG EQ 1) then print,' Get Info on Var to be read with:   NCDF_VARINQ(groupid, varid) '
  if(DEBUG EQ 1) then print,' More info about this variable about to be read '
  if(DEBUG EQ 1) then print,' Possible DataTypes: BYTE, CHAR, INT, LONG, FLOAT, DOUBLE '
  if(DEBUG EQ 1) then print,' { VarName, DataType, NDims, NumAtts, Dimensions_ID_Vector } '
  if(DEBUG EQ 1) then print,' NCDF_VARINQ Result = ', Result

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Read Data
NCDF_VARGET, group_id4, varid, data

if(DEBUG EQ 1) then print,' Read Data '
if(DEBUG EQ 1) then help, data
if(DEBUG EQ 1) then print,' min(data) = ', min(data)
if(DEBUG EQ 1) then print,' max(data) = ', max(data)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; SECOND VARIABLE
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Get the Variable ID
if(DEBUG EQ 1) then print,' '
if(DEBUG EQ 1) then print,' '
if(DEBUG EQ 1) then print,' Reading Signed Byte Variable: Cloud_Effective_Emissivity_Day'
if(DEBUG EQ 1) then print,' Valid Data Range: 0 to 100'
if(DEBUG EQ 1) then print,' Fill Value: 127'
varid = NCDF_VARID(group_id4, 'Cloud_Effective_Emissivity_Day')
if(DEBUG EQ 1) then print,' varid = ', varid

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Lets get more information about this variable about to be read in
Result = NCDF_VARINQ(groupid, varid)

  if(DEBUG EQ 1) then print,' Get Info on Var to be read with:   NCDF_VARINQ(groupid, varid) '
  if(DEBUG EQ 1) then print,' More info about this variable about to be read '
  if(DEBUG EQ 1) then print,' Possible DataTypes: BYTE, CHAR, INT, LONG, FLOAT, DOUBLE '
  if(DEBUG EQ 1) then print,' { VarName, DataType, NDims, NumAtts, Dimensions_ID_Vector } '
  if(DEBUG EQ 1) then print,' NCDF_VARINQ Result = ', Result

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Read Data
NCDF_VARGET, group_id4, varid, data

if(DEBUG EQ 1) then print,' Read Data '
if(DEBUG EQ 1) then help, data
if(DEBUG EQ 1) then print,' min(data) = ', min(data)
if(DEBUG EQ 1) then print,' max(data) = ', max(data)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; THIRD VARIABLE
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Get the Variable ID
if(DEBUG EQ 1) then print,' '
if(DEBUG EQ 1) then print,' '
if(DEBUG EQ 1) then print,' Reading Unsigned Byte Variable: Quality_Assurance_1km'
if(DEBUG EQ 1) then print,' Valid Data Range: 1 to 255'
if(DEBUG EQ 1) then print,' Fill Value: 0'
varid = NCDF_VARID(group_id4, 'Quality_Assurance_1km')
if(DEBUG EQ 1) then print,' varid = ', varid

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Lets get more information about this variable about to be read in
Result = NCDF_VARINQ(groupid, varid)

  if(DEBUG EQ 1) then print,' Get Info on Var to be read with:   NCDF_VARINQ(groupid, varid) '
  if(DEBUG EQ 1) then print,' More info about this variable about to be read '
  if(DEBUG EQ 1) then print,' Possible DataTypes: BYTE, CHAR, INT, LONG, FLOAT, DOUBLE '
  if(DEBUG EQ 1) then print,' { VarName, DataType, NDims, NumAtts, Dimensions_ID_Vector } '
  if(DEBUG EQ 1) then print,' NCDF_VARINQ Result = ', Result

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Read Data
NCDF_VARGET, group_id4, varid, data

if(DEBUG EQ 1) then print,' Read Data '
if(DEBUG EQ 1) then help, data
if(DEBUG EQ 1) then print,' min(data) = ', min(data)
if(DEBUG EQ 1) then print,' max(data) = ', max(data)

NCDF_CLOSE, ncid

end

@phubanks
Copy link
Author

phubanks commented Feb 7, 2025

above is what my PRO code looks like.

The whole thing (all three variables read in) works fine under IDL

Under GDL, the 3rd variable read CHOKES under GDL ... when it tried to read in an Unsigned Byte array (value range 0 to 255)

@GillesDuvert
Copy link
Contributor

@phubanks see #1922. Changes were made by @rsiddans to support the NC_UINT case.
The patch was applied on Nov 28, 2024. But the v1.1 is dated Nov 2, 2024, so do not have them.
I suggest you use the Weekly Release to assert if the most recent GDL works. Or rebuild gdl from the latest github sources following the instructions in the wiki.

@phubanks
Copy link
Author

phubanks commented Feb 7, 2025

Hello GillesDuvert,

I REALLY appreciate your response to me. Thank you!

You mentioned a fix to support the NC_UINT case done in late november 2024. Did that change also support a fix to the NC_UBYTE case, since my issue is with reading unsigned byte arrays.

I was trying to interpret fully what you wrote, do i have this correct?

So are you saying that someone there added a patch to an interim (unreleased version) in late Nov 2024 --

however it sounds like there was never a new "official release" of GDL with that patch included, correct?
Your last official release was v1.1 ... and that was released about 3 weeks prior to the patch being coded. So your last official release of GDL (v1.1) does NOT contain that unsigned variable patch

So, will there be a new official release soon (v1.2 perhaps) that will include that UNSIGNED BYTE read enhancement?
Any rough idea on when that might happen.

In the meantime, sounds like you are suggesting i have my sys admins install a later version of GDL, which is currently marked as "unstable" from your weekly archive. Correct?

Does unstable mean it might have some problems or bugs?

Thanks so much for any information you can provide

@GillesDuvert
Copy link
Contributor

In fact, there will be a 1.1.1 right now. And @rsiddans pull request concerned AFAIK the ubyte too.
But the 1.1.1 will be nothing more than a copy of the 'unstable' of this week, and 'unstable' is just another term for 'nighly build' I guess. So you do not have to wait for the 1.1.1, really.

@GillesDuvert
Copy link
Contributor

If you use the current wekly release and your problem disappears, please tell us, so that I could close this issue and indicate in the coming 1.1.1 release notes that NCDF_commands support now the unsigned types.

@phubanks
Copy link
Author

phubanks commented Feb 8, 2025

Hello GillesDuvert,

Thank you SO MUCH for your quick response. I really appreciate it!

You wrote that "there will be a 1.1.1 right now",
but I just checked the page
https://github.com/gnudatalanguage/gdl/releases
and its still showing v1.1 as the latest official release.

Any rough idea on when that releases page will show the new v1.1.1 version as being available..

The reason i ask is i have to ask my sys admins to install it, and I think they'll be a lot more comfortable installing an official release vs. your unofficial weekly release (I am worried about them balking at the term "unstable". Even tho as you said the weekly release will be identical to the upcoming v1.1.1. release. I still have to "sell" the idea to my sys admins -- convincing other people who are in power to install that.

If you could update the official releases page to show a v1.1.1 is now available, and show that v1.1.1 as a list of "assets" I will have a much easier time convincing the sys admins to install that update.

Do you think you might update that releases page within the next few days? or do you have a rough estimate in terms of days or weeks when you think that might happen.

Thanks again for all your hard work on this.
It is very much appreciated!

@GillesDuvert
Copy link
Contributor

ah I see... I do not know your personal situation but I believe that
gdl can be installed in userspace, that is, without any sysadmin intervention provided that the needed libraries are already present on the system, which is your case as you already run the 1.1 .
Anyway the 1.1.1 is on its way this WE.

@phubanks
Copy link
Author

phubanks commented Feb 8, 2025

there is one thing different between the weekly version and the last official version ...
you offer a LINUX version thta includes all the flavors including Ubuntu in the v1.1 version
but dont offer that in the weekly version

@phubanks
Copy link
Author

phubanks commented Feb 8, 2025

thank you!!

@GillesDuvert GillesDuvert mentioned this issue Feb 9, 2025
Closed
@GillesDuvert
Copy link
Contributor

1.1.1 is out.
People with a little practice of unix/linux just need to download and unzip gdl-Linux-x86_64-standard.zip , normally it should go in /usr/local (hence the need to be root/sysadmin) but it may work everywhere, provided the needed shared libraries are present on the system, or in the LD_LIBRARY_PATH . This is what I meant when I said that GDL can be installed in user space.

@rsiddans
Copy link
Contributor

Just to say that my patch for NC_UINT also attempted to fix other unsigned types including NC_UBYTE, though was not possible to test all datatypes. I did try a few different ncdf files, though so would be fairly optimistic. If you can point me at an e.g. of the .nc file you are trying to read I could try it out (I see it's MODIS L2 but only have access to hdf versions of those).

BTW I can confirm it not necessary to be sys admin to get working copy of gdl working - i do that using git following the instructions for "Cloning GDL" on this page https://gnudatalanguage.github.io/
I clone the code to a dir under my home dir, then following the instructions in README and INSTALL.CMake.

@phubanks
Copy link
Author

phubanks commented Feb 10, 2025 via email

@rsiddans
Copy link
Contributor

OK I tested it and I think the file is read fine with the version of gdl I have (working version from last month). I attach output from running your script in gdl and idl. So you should be fine with the gdl v1.1.1 release.

output_gdl.txt
output_idl.txt

@phubanks
Copy link
Author

phubanks commented Feb 10, 2025 via email

@slayoo
Copy link
Member

slayoo commented Feb 10, 2025

By the way I got this email over the weekend …
@slayoo has invited you to join the gnudatalanguage organization
I just wanted to understand what that meant.

That's a GitHub users group where we invite anyone who uses GDL, so that one may @-tag the group in case of new releases or announcement: https://github.com/orgs/gnudatalanguage/teams/gdlteam

@slayoo
Copy link
Member

slayoo commented Feb 10, 2025

So will the v1.1.1 release become an official release sometime this week?

v1.1.1 is official since yesterday, see: https://github.com/gnudatalanguage/gdl/releases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@slayoo @GillesDuvert @phubanks @rsiddans