set up and test of FGAT for mpas-jedi Ens3DVar

[TingLei-NOAA]

It was found in the fv3-jedi by @XuLu-NOAA , the fgat used in the 3dEnvar doesn't behave as expected (https://github.com/JCSDA-internal/fv3-jedi/issues/1233, NOAA-EMC/HDASApp#7)
While the reason is being investigated, a set-up and test of FGAS for mpas-jedi Ens3Dvar would ,1) hopefully, contribute the investigation efforts ;2) help interested JEDI user/developers have a better understanding of the related optimization details using the observations.

[ShunLiu-NOAA]

@TingLei-NOAA do you mean you will investigate halo file Inconsistency Issue in RDASApp?

[TingLei-NOAA]

@ShunLiu-NOAA The halo mismatch problem with hdasapp's tests had been resolved ( it is never a problem for rdasapp). In @XuLu-NOAA previous runs, a halo 10 grid file was used and caused the misplacement of the positions in the jedi results, which had been identified and resolved. The current issue is the fgat 's behavior is not as expected. So, I am planning to run a mpas-jedi test using fgat to help investigate what the problem is. If time allowed and we have to dig into details by ourselves, this would also be a good time for us to understand the details how fgat in jedi was implemented.

[XuLu-NOAA]

@TingLei-NOAA Also, for your info, the 4DEnVar works as expected in JEDI (NOAA-EMC/HDASApp#8). The 4DEnVar implicitly applies FGAT in addition to the varying ensemble covariance. But it suggests JEDI could process the FGAT info correctly. Do you think it will be helpful comparing the log file between 3D-FGAT and 4DEnVar?

[TingLei-NOAA]

@XuLu-NOAA Thanks for the update. Did you compare 4densvar using and not using fgat?

[XuLu-NOAA]

@XuLu-NOAA Thanks for the update. Did you compare 4densvar using and not using fgat?

In my understanding, 4DEnVar automatically uses FGAT implicitly as the first guesses at different times are always provided. Otherwise, the y-h(x) would provide multiple values. So I cannot run 4DEnVar without FGAT.

[TingLei-NOAA]

@XuLu-NOAA Conceptually, 4densvar assigns ensembles to different observation time levels and , i think, it can run without fgat , though, I never tried it like with GSI ( i believe its coding allow this use) and not sure if jedi's implementation give the options for 4denvar using or not using fgat.

[TingLei-NOAA]

A set of regional mpas-jedi experiments using fgat are finished using one obs on 700hpa. The comparison of increment on level 18 ( the model level closest to 700 hpa around the observation) showed some "instability " , which is to be investigated. More details could refer to https://docs.google.com/presentation/d/17XQV0CEb6WHNZmdFWesaoxyPsVb9uMe5/edit?usp=sharing&ouid=104470932239919354212&rtpof=true&sd=true.
The attached is the OMB/OMA

[XuLu-NOAA]

Thanks, Ting! This is very encouraging! Can you point to me your yaml file for the FGAT configuration? So I can compare it with mine and figure the issue of my test.

[TingLei-NOAA]

Xu, that is the file https://github.com/TingLei-NOAA/RDASApp/blob/feature/tl_dev/rrfs-test/testinput/test-ens3dvar-fgat-m03hrob.yaml which I am using . Your fv3-jedi fgat yaml looks good me. Maybe we need to be careful on the obs time. In my case , exactly putting the obs at the beginning of the da window would cause this ob missing in the cost function calculation.