Test the UFO of ATMS brightness temperature with HAFS

[XuLu-NOAA]

This is to record the process of assimilating ATMS brightness temperature using JEDI 3DEnVar with HAFS background.

Data processing:

1. Link gfs.t12z.atms.tm00.bufr_d to ./testinput
2. Then use $HDASApp/build/bin/bufr2ioda.x /scratch2/NCEPDEV/hwrf/scrub/Xu.Lu/JEDI_Fix/WorkingYaml/bufr_ncep_atms.yaml
   To generate the gfs.t12z.atms_n20.tm00.nc and gfs.t12z.atms_npp.tm00.nc in the testout.
3. Ready for JEDI DA.

Preliminary Yaml setup
/scratch2/NCEPDEV/hwrf/scrub/Xu.Lu/JEDI_Fix/WorkingYaml/atms_n20_jedi.yaml

[XuLu-NOAA]

For bias correction:

1. Link the gdas.t18z.abias & gdas.t18z.abias_pc from GFS.
2. Use $HDASApp/build/bin/satbias2ioda.x /scratch2/NCEPDEV/hwrf/scrub/Xu.Lu/JEDI_Fix/WorkingYaml/satbias_converter_atms.yaml to generate the satbias_atms_npp.nc.
3. Copy satbias_atms_npp.nc as satbias_atms_npp_cov.nc, both are needed by JEDI.
4. Setup bias correction section in the yaml, /scratch2/NCEPDEV/hwrf/scrub/Xu.Lu/JEDI_Fix/WorkingYaml/atms_npp_jedi_bc.yaml:
   obs bias:
   input file: Data/obs/satbias_atms_npp.nc
   output file: Data/bc/out_satbias_atms_npp.nc
   variational bc:
   predictors:
   - name: constant
   - name: lapseRate
   order: 2
   tlapse: &atms_npp_tlapse Data/obs/atms_npp.tlapse.txt
   - name: lapseRate
   tlapse: *atms_npp_tlapse
   - name: emissivityJacobian
   - name: sensorScanAngle
   order: 4
   - name: sensorScanAngle
   order: 3
   - name: sensorScanAngle
   order: 2
   - name: sensorScanAngle
   covariance:
   minimal required obs number: 20
   variance range: [1.0e-6, 10.0]
   step size: 1.0e-4
   largest analysis variance: 10000.0
   prior:
   input file: Data/obs/satbias_atms_npp_cov.nc
   inflation:
   ratio: 1.1
   ratio for small dataset: 2.0
   output file: Data/bc/out_satbias_atms_npp_cov.nc

[XuLu-NOAA]

Testing with a single ob test in comparison with GSI. It looks the increment pattern matches at certain model levels, e.g. model level 9:

But question is raised for the nearby model levels, where strange increment pattern emerges at level 10:

Or the increments switches between positive and negative in JEDI:

These increment patterns look suspicious as compared to the GSI patterns. Needs further investigation if from localization? Or try to increase the ensemble size?

[XuLu-NOAA]

It looks the vertical localization is the issue after the EMC internal discussion with RRFS DA group. Decreasing the vertical localization value from 0.3 to 0.001, the Increment patterns are matching between GSI and JEDI across vertical levels:

The vertical localization is in sigma level instead of logp according to the discussion NOAA-EMC/RDASApp#53. Needs to figure out a proper value of sigma or using logp in consistent with GSI.
This may also help resolve the inconsistent increment magnitude puzzle in #6

[XuLu-NOAA]

Somehow in the previous test, although the same ob is reading in, GSI print out different values as compared to JEDI.

Further optimize the single ob test by forcing the GSI and JEDI reading in same obs value. The table and vertical increment profile indicates high similarity between the current GSI and JEDI configurations:

Horizontal increments are also more consistent:

[XuLu-NOAA]

Further test with full channels/multiple data DA between GSI and JEDI. I converted the diag files from GSI to nc for JEDI to ensure the same data assimilated. There is a reasonable similarity to conclude that the DA in JEDI is reasonably ready for use.