Surface ===================== Data directory structure ------------------------------------ The Surface data is organized as xml files every 15 minutes : .. code-block:: bash > ls -la /g3/tianwh/share/conobs/rec_RSURF_20190* -rw-r--r-- 1 tianwh obs 11482070 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060100_g_06.dat -rw-r--r-- 1 tianwh obs 11654320 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060106_g_06.dat -rw-r--r-- 1 tianwh obs 11637320 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060112_g_06.dat -rw-r--r-- 1 tianwh obs 11630570 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060118_g_06.dat -rw-r--r-- 1 tianwh obs 11549820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060200_g_06.dat -rw-r--r-- 1 tianwh obs 11672570 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060206_g_06.dat -rw-r--r-- 1 tianwh obs 11687820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060212_g_06.dat -rw-r--r-- 1 tianwh obs 11619320 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060218_g_06.dat -rw-r--r-- 1 tianwh obs 11566570 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060300_g_06.dat -rw-r--r-- 1 tianwh obs 11562820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060306_g_06.dat -rw-r--r-- 1 tianwh obs 11619320 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060312_g_06.dat -rw-r--r-- 1 tianwh obs 11468820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060318_g_06.dat -rw-r--r-- 1 tianwh obs 11398070 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060400_g_06.dat -rw-r--r-- 1 tianwh obs 11605820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060406_g_06.dat ...... Data format ------------------- CIMISS format data is looks like: .. code-block:: bash RSURF 2019 6 1 0 45928 6 9 9 00081 81.00 -5.53 -36.87 19.00 1.00 100.00 1013.20 23.05 21.35 347.00 0.00 999999.00 999999.00 999999.00 1011.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 00081 81.00 -5.53 -36.87 19.00 1.00 0.00 1012.80 24.15 21.75 345.00 0.10 999999.00 999999.00 999999.00 1010.60 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 00081 81.00 -5.53 -36.87 19.00 1.00 200.00 1012.90 22.25 21.25 333.00 0.00 999999.00 999999.00 999999.00 1010.70 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 00086 86.00 -19.98 -43.96 1214.00 1.00 0.00 1016.00 20.75 14.25 289.00 1.80 999999.00 999999.00 999999.00 882.20 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 00086 86.00 -19.98 -43.96 1214.00 1.00 100.00 1016.30 20.25 14.35 306.00 1.50 999999.00 999999.00 999999.00 882.30 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 00086 86.00 -19.98 -43.96 1214.00 31.00 2100.00 1015.10 21.35 13.95 288.00 3.90 999999.00 999999.00 999999.00 881.70 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 00086 86.00 -19.98 -43.96 1214.00 1.00 200.00 1016.30 20.45 14.35 283.00 2.50 999999.00 999999.00 999999.00 882.40 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 01001 1001.00 70.94 -8.67 10.00 31.00 2200.00 1015.80 -0.40 -1.80 341.00 10.70 999999.00 999999.00 999999.00 1014.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01001 1001.00 70.94 -8.67 10.00 1.00 200.00 1014.60 -0.40 -1.97 337.00 10.30 999999.00 999999.00 999999.00 1013.40 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01001 1001.00 70.94 -8.67 10.00 1.00 100.00 1015.00 -0.40 -2.06 340.00 11.60 999999.00 999999.00 999999.00 1013.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01001 1001.00 70.94 -8.67 10.00 31.00 2300.00 1015.20 -0.40 -2.23 341.00 10.40 999999.00 999999.00 999999.00 1014.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01001 1001.00 70.94 -8.67 10.00 31.00 2100.00 1015.80 -0.20 -1.89 340.00 13.60 999999.00 999999.00 999999.00 1014.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01001 1001.00 70.94 -8.67 10.00 1.00 0.00 1015.00 -0.20 -1.58 344.00 12.10 999999.00 0.00 999999.00 1013.80 0.00 0.00 0.00 0.00 0.00 8.00 0.00 8.00 0.00 01002 1002.00 80.06 16.25 8.00 1.00 100.00 1020.50 -1.90 -2.89 109.00 6.80 999999.00 999999.00 999999.00 1019.70 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01002 1002.00 80.06 16.25 8.00 31.00 2200.00 1020.80 -0.20 -2.30 89.00 6.90 999999.00 999999.00 999999.00 1020.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01002 1002.00 80.06 16.25 8.00 31.00 2300.00 1020.70 -1.20 -2.59 111.00 6.30 999999.00 999999.00 999999.00 1019.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01002 1002.00 80.06 16.25 8.00 31.00 2100.00 1020.70 0.00 -1.96 85.00 7.20 999999.00 999999.00 999999.00 1019.90 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01002 1002.00 80.06 16.25 8.00 1.00 0.00 1020.70 -1.90 -3.02 119.00 7.30 999999.00 999999.00 999999.00 1019.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01002 1002.00 80.06 16.25 8.00 1.00 200.00 1020.30 -1.70 -2.96 111.00 8.60 999999.00 999999.00 999999.00 1019.50 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01003 1003.00 77.00 15.54 10.00 1.00 100.00 1014.70 -1.30 -2.83 110.00 13.70 999999.00 999999.00 999999.00 1013.30 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01003 1003.00 77.00 15.54 10.00 31.00 2300.00 1014.40 1.50 -3.16 62.00 7.60 999999.00 999999.00 999999.00 1013.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01003 1003.00 77.00 15.54 10.00 31.00 2200.00 1014.00 1.40 -3.26 59.00 12.50 999999.00 999999.00 999999.00 1012.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00 01003 1003.00 77.00 15.54 10.00 31.00 2100.00 1013.50 1.40 -3.45 57.00 13.10 999999.00 999999.00 999999.00 1012.10 0.00 1.00 1.00 0.00 0.00 8.00 8.00 8.00 0.00 01003 1003.00 77.00 15.54 10.00 1.00 0.00 1014.40 1.10 -2.99 53.00 9.80 999999.00 0.00 999999.00 1013.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 8.00 0.00 Data extractor --------------------- A python code is used to extract the desired information from this CIMISS file: .. code-block:: bash > cd /g5/nwpc_ep8/obsproc/nwprod/decoders > rm cimiss_data_surface > ./read_nmc_cimiss_rsurf_rgwst.py -f /g3/tianwh/share/conobs/rec_RSURF_2019060406_g_06.dat If you want to batch process number of CIMISS files, you can use following command: .. code-block:: bash > # This command will find all CIMISS files and prcessing the file one by one > rm cimiss_data_surface > decod_nmc_cimiss_rsing.py -d /g3/tianwh/share/conobs The information we want to extract from CIMISS is: .. code-block:: bash import fortranformat as ff output_line = ff.FortranRecordWriter('(I8, 3f10.2, 5A8, 2f10.2, 2I8, f10.2)') output.write(output_line.write([id, lat, lon, elv, obstime.strftime("%Y"), obstime.strftime("%m"), \ obstime.strftime("%d"), obstime.strftime("%H"), obstime.strftime("%M"), \ p, T, round(rh), round(dd), ff]) + "\n") the content of :code:`cimiss_data_surface` is: .. code-block:: bash > less cimiss_data_surface 56076 33.85 102.6333 3429 2018 9 1 16 0 674.7 10.8 97 149 3.4 674002 40.6644 109.6661 1052 2018 9 1 16 0 890.2 15 95 2 1.3 735904 31.0231 118.4314 9 2018 9 1 16 0 999999 26.4 999999 158 1.7 868552 34.55 109.2975 357 2018 9 1 16 0 965.8 24.9 74 356 0.9 718262 22.8133 112.4264 49 2018 9 1 16 0 999999 25.4 999999 225 0 708128 26.8458 116.7783 343 2018 9 1 16 0 970.6 22.9 96 176 1.1 737152 30.7728 117.3278 28.3 2018 9 1 16 0 1005.9 26.3 97 135 0.6 756060 29.1036 119.8056 58 2018 9 1 16 0 999999 26.5 999999 210 1.1 804130 27.4056 110.4781 200 2018 9 1 16 0 999999 23.1 999999 999999 999999 672002 43.2972 116.1158 1284 2018 9 1 16 0 860 13.9 94 217 7.6 663689 40.2278 119.22 300 2018 9 1 16 0 999999 19.2 999999 999999 999999 752628 30.1492 121.1961 5 2018 9 1 16 0 999999 28.1 999999 237 1.6 860410 34.9769 108.5269 1193 2018 9 1 16 0 999999 19.8 999999 999999 999999 868283 34.2217 108.5681 394.5 2018 9 1 16 0 959.8 26.9 64 302 1 754019 29.9078 120.4844 59 2018 9 1 16 0 999999 24.6 999999 324 0 878120 36.0133 108.1183 1129 2018 9 1 16 0 876.3 17.5 99 329 1.4 751675 29.6344 119.5692 95 2018 9 1 16 0 999999 24.7 999999 333 0 Decoder source code ---------------------------- 1. Source code directory: .. code-block:: bash > cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dccimisssurf/sorc 2. Subroutines to decode surface data * :code:`lsdcod.f` .. note:: * The *cimiss.tbl* and :code:`decod_WMO.Res40.headers` are not used, although they are required as arguments and read in. * The path and file name of :code:`cimiss_data_surface` file are hard coded in the subroutines. 4. Compile the code .. code-block:: bash > make Decode and convert to BUFR format --------------------------------------- 1. enter into the exec directory .. code-block:: bash > cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dccimisssurf/exec > ls -la total 2028 drwxr-xr-x. 3 zwtd zwtd 211 Dec 20 22:45 . drwxr-xr-x. 6 zwtd zwtd 82 Dec 20 22:42 .. lrwxrwxrwx. 1 zwtd zwtd 34 Dec 9 23:48 bufrtab.000 -> ../../decod_shared/fix/bufrtab.000 -rw-r--r--. 1 zwtd zwtd 1121683 Dec 9 23:48 cimiss.tbl lrwxrwxrwx. 1 zwtd zwtd 31 Dec 9 23:48 decod_WMO.Res40.headers -> ../parm/decod_WMO.Res40.headers -rwxr-xr-x. 1 zwtd zwtd 940696 Dec 20 00:24 decod_dccimiss -rw-r--r--. 1 zwtd zwtd 462 Dec 20 22:45 decod_dccimiss.log -rwxr-xr-x. 1 zwtd zwtd 484 Dec 20 00:30 run.ksh -rwxr-xr-x. 1 zwtd zwtd 776 Dec 20 22:44 run_dccimisssurf.py drwxr-xr-x. 2 zwtd zwtd 58 Dec 20 22:45 tmp 2. we provide a script to run the decoder in batch mode: .. code-block:: bash > ./run_dccimisssurf.py -s 2018121600 -e 2018121700 -i 1 .. note:: * given the starting datetime and ending datetime, it iterates all cycles (every 1 hours) * the units of interval is hour (-i) * this script call run.ksh 3. run the decoder script .. code-block:: bash > run.ksh > cat run.ksh !/bin/bash export DBNBUFRT=120 export TRANJB=/g5/nwpc_ep8/obsproc/nwprod/ush/tranjb export tank_dir=/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003 export DBNROOT=`pwd` rm tmp/* rm decod_dccimiss.log ./decod_dccimiss -d decod_dccimiss.log -b 240 -c $1 bufrtab.000 cimiss.tbl decod_WMO.Res40.headers ls -la tmp/* BUFR_FILES=$(echo tmp/BUFR*) echo ${BUFR_FILES} for file in ${BUFR_FILES} do ${TRANJB} ${tank_dir} ${file} done .. note:: * -c $1 : Set the **current time** (201809011600) used to calculate the time departures of the obs. data. * -b 240 : Number of hours to decode prior to "current" time (default) * The observations with date/time between **current time** - 240 hours and **current time** + 3 are **kept**. 4. The generated BUFR format file will be saved at .. code-block:: bash > ls -la tmp -rw-r--r-- 1 xinzhang staff 4199744 Sep 21 18:45 tmp/BUFR.0.cimiss.1.6436.1536097072.8 Transfer bufr data to BUFR Tanks ------------------------------------ * put data in BUFR **tanks**: .. code-block:: bash > /g5/nwpc_ep8/obsproc/nwprod/ush/tranjb /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003 tmp/BUFR.0.cimiss.1.6436.1536097072.8 > ls -al /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx01 -rw-r--r-- 1 vagrant vagrant 4235328 Sep 27 04:42 /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx001 .. note:: * if environmental variable **SCREEN=ON** : * Define **Run Time** is the system time when the tranjb is running. * Only observations with date/time between **Run Time** - 10 days and **Run Time** + 12 hours are kept. * for retrospective run, set **SCREEN=OFF** * :code:`/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/yyyymmdd/bmmm/xxsss` (where mmm is WMO BUFR message type and xxx is local BUFR message subtype) * 000.001 (in dump group mnemonic adpsfc): Surface synoptic fixed land reports * BUFR format * Arranged by UTC day and continuously grow throughout the day, if you run decoders many time, the content of the file will grow * No QC (other than rudimentary checks inside decoders) * No duplicate checking * Interested users can use utility :code:`debufr` to check the content of the bufr file:: > /g5/nwpc_ep8/obsproc/nwprod/util/exec/debufr /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx001 the output is in :code:`debufr.out`.