======== Aircraft ======== .. contents :: Data directory structure ============================= The Aircraft data is organized as CIMISS files every 6 hours :: > ls -la /g3/tianwh/share/conobs/rec_RSING_*: -rw-r--r-- 1 tianwh obs 34351730 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060100_g_06.dat -rw-r--r-- 1 tianwh obs 16639350 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060106_g_06.dat -rw-r--r-- 1 tianwh obs 25511010 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060112_g_06.dat -rw-r--r-- 1 tianwh obs 31514910 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060118_g_06.dat -rw-r--r-- 1 tianwh obs 29063650 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060200_g_06.dat -rw-r--r-- 1 tianwh obs 14376670 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060206_g_06.dat -rw-r--r-- 1 tianwh obs 24181990 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060212_g_06.dat -rw-r--r-- 1 tianwh obs 32740190 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060218_g_06.dat -rw-r--r-- 1 tianwh obs 32471950 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060300_g_06.dat -rw-r--r-- 1 tianwh obs 20091890 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060306_g_06.dat -rw-r--r-- 1 tianwh obs 25002950 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060312_g_06.dat -rw-r--r-- 1 tianwh obs 32896430 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060318_g_06.dat -rw-r--r-- 1 tianwh obs 34783490 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060400_g_06.dat -rw-r--r-- 1 tianwh obs 21610470 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060406_g_06.dat -rw-r--r-- 1 tianwh obs 29883770 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060412_g_06.dat -rw-r--r-- 1 tianwh obs 34280610 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060418_g_06.dat -rw-r--r-- 1 tianwh obs 35147210 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSING_2019060500_g_06.dat Data format =========== CIMISS format data is looks like :: RSING 2019 6 1 0 245369 5 4 4 00HQVXZA 44.49 150.37 1.00 241.00 9.00 11247.00 -43.95 222.00 38.60 9.00 9.00 9.00 9.00 00HQVXZA 46.01 152.68 1.00 221.00 9.00 11217.00 -47.95 216.00 38.60 9.00 9.00 9.00 9.00 00HQVXZA 47.45 155.04 1.00 201.00 9.00 10638.00 -48.95 229.00 45.30 9.00 9.00 9.00 9.00 00HQVXZA 53.18 167.60 1.00 37.00 3.00 10368.00 -48.45 267.00 13.40 9.00 9.00 9.00 9.00 00HQVXZA 56.68 178.56 31.00 2340.00 3.00 10368.00 -50.45 345.00 44.20 9.00 9.00 9.00 9.00 00HQVXZA 49.34 158.56 1.00 135.00 3.00 10368.00 -47.45 232.00 25.20 9.00 9.00 9.00 9.00 00HQVXZA 58.23 -172.53 31.00 2301.00 3.00 10367.00 -49.95 7.00 11.80 9.00 9.00 9.00 9.00 00HQVXZA 51.98 164.39 1.00 56.00 3.00 10366.00 -50.15 207.00 23.10 9.00 9.00 9.00 9.00 00HQVXZA 52.59 165.94 1.00 47.00 3.00 10366.00 -49.45 227.00 20.60 9.00 9.00 9.00 9.00 00HQVXZA 50.69 161.37 1.00 116.00 3.00 10366.00 -47.95 274.00 25.70 9.00 9.00 9.00 9.00 00HQVXZA 50.02 159.94 1.00 125.00 3.00 10364.00 -47.65 249.00 24.20 9.00 9.00 9.00 9.00 00HQVXZA 57.15 -179.26 31.00 2330.00 3.00 10364.00 -46.95 358.00 29.80 9.00 9.00 9.00 9.00 00HQVXZA 54.93 171.98 1.00 12.00 9.00 10363.00 -48.95 320.00 32.90 9.00 9.00 9.00 9.00 00HQVXZA 59.00 -160.01 31.00 2212.00 9.00 10363.00 -44.95 83.00 6.20 9.00 9.00 9.00 9.00 00HQVXZA 53.75 169.23 1.00 28.00 9.00 10363.00 -48.95 287.00 18.50 9.00 9.00 9.00 9.00 00HQVXZA 57.55 -177.04 31.00 2320.00 3.00 10362.00 -47.95 1.00 20.60 9.00 9.00 9.00 9.00 00HQVXZA 51.34 162.86 1.00 106.00 3.00 10362.00 -47.65 259.00 14.40 9.00 9.00 9.00 9.00 00HQVXZA 54.28 170.43 1.00 21.00 9.00 10333.00 -47.95 314.00 23.10 9.00 9.00 9.00 9.00 00HQVXZA 55.56 174.12 1.00 1.00 9.00 10333.00 -50.95 328.00 34.00 9.00 9.00 9.00 9.00 00HQVXZA 58.77 -167.53 31.00 2241.00 9.00 10333.00 -50.95 39.00 12.90 9.00 9.00 9.00 9.00 00HQVXZA 57.53 -177.16 31.00 2321.00 9.00 10333.00 -46.95 5.00 21.10 9.00 9.00 9.00 9.00 00HQVXZA 50.32 160.57 1.00 121.00 9.00 10333.00 -46.95 261.00 21.60 9.00 9.00 9.00 9.00 Data extractor ================ A python code is used to extract the desired information from this CIMISS file:: > cd /g5/nwpc_ep8/obsproc/nwprod/decoders > rm cimiss_amdar_data > ./read_nmc_cimiss_rsing.py -f /g3/tianwh/share/conobs/rec_RSING_2019060100_g_06.dat If you want to batch process number of CIMISS files, you can use following command:: > # This command will find all CIMISS files and prcessing the file one by one > decod_nmc_cimiss_rsing.py -d /g3/tianwh/share/conobs The information we want to extract from CIMISS is. :: import fortranformat as ff output_line = ff.FortranRecordWriter('(A8, 5A8, 4f10.2, I8, f10.2)') output.write(output_line.write([call_name, obstime.strftime("%Y"), obstime.strftime("%m"), \ obstime.strftime("%d"), obstime.strftime("%H"), obstime.strftime("%M"), lat, lon, phi, \ T, dd, ff]) + "\n") the content of :code:`micaps_amdar_data` is:: > less micaps_amdar_data 01ZXL5JA 2019 06 30 14 38 34.61 -109.08 11884.00 -56.45 277 7.20 01ZXL5JA 2019 06 30 14 32 33.93 -109.51 11883.00 -55.25 277 7.70 01ZXL5JA 2019 06 30 14 50 35.94 -108.19 11882.00 -56.95 295 9.30 01ZXL5JA 2019 06 30 14 59 37.01 -107.83 11882.00 -57.15 282 11.80 01ZXL5JA 2019 06 30 14 56 36.64 -107.90 11882.00 -56.95 280 10.30 01ZXL5JA 2019 06 30 14 41 34.94 -108.86 11881.00 -56.25 273 9.80 01ZXL5JA 2019 06 30 14 47 35.61 -108.41 11881.00 -56.45 282 9.30 01ZXL5JA 2019 06 30 14 44 35.28 -108.64 11881.00 -56.45 278 9.30 01ZXL5JA 2019 06 30 14 53 36.28 -107.99 11881.00 -56.65 286 9.80 01ZXL5JA 2019 06 30 14 35 34.27 -109.29 11880.00 -56.45 273 6.70 01ZXL5JA 2019 06 30 14 29 33.59 -109.72 11468.00 -52.75 287 6.70 01ZXL5JA 2019 06 30 14 26 33.26 -109.94 9933.00 -39.75 158 1.00 01ZXL5JA 2019 06 30 14 23 32.90 -110.16 8393.00 -27.25 232 3.10 01ZXL5JA 2019 06 30 14 19 32.55 -110.38 6529.00 -12.65 265 0.50 01ZXL5JA 2019 06 30 14 19 32.51 -110.40 6358.00 -11.75 305 1.50 01ZXL5JA 2019 06 30 14 19 32.48 -110.42 6135.00 -10.45 310 1.50 01ZXL5JA 2019 06 30 14 18 32.45 -110.44 5902.00 -8.75 334 1.50 01ZXL5JA 2019 06 30 14 18 32.42 -110.46 5637.00 -7.45 336 3.10 01ZXL5JA 2019 06 30 14 18 32.39 -110.47 5379.00 -5.75 340 2.10 01ZXL5JA 2019 06 30 14 17 32.35 -110.49 5122.00 -3.75 15 2.10 01ZXL5JA 2019 06 30 14 17 32.32 -110.51 4757.00 -0.75 47 3.10 01ZXL5JA 2019 06 30 14 17 32.29 -110.53 4485.00 1.55 34 1.50 01ZXL5JA 2019 06 30 14 16 32.26 -110.55 4214.00 3.55 24 3.10 01ZXL5JA 2019 06 30 14 16 32.24 -110.58 3946.00 5.85 31 3.10 01ZXL5JA 2019 06 30 14 16 32.22 -110.62 3665.00 8.85 34 3.60 01ZXL5JA 2019 06 30 14 15 32.21 -110.65 3390.00 11.05 17 3.60 01ZXL5JA 2019 06 30 14 15 32.19 -110.68 3270.00 11.55 28 1.50 01ZXL5JA 2019 06 30 14 15 32.17 -110.71 3111.00 13.85 65 2.10 01ZXL5JA 2019 06 30 14 14 32.15 -110.73 2809.00 16.35 157 1.50 01ZXL5JA 2019 06 30 14 14 32.14 -110.76 2511.00 18.85 126 4.60 01ZXL5JA 2019 06 30 14 14 32.12 -110.79 2160.00 22.05 131 4.10 01ZXL5JA 2019 06 30 14 13 32.11 -110.81 1942.00 23.55 131 3.10 01ZXL5JA 2019 06 30 14 13 32.10 -110.83 1797.00 24.25 103 2.10 01ZXL5JA 2019 06 30 14 13 32.08 -110.85 1606.00 25.85 108 2.60 01ZXL5JA 2019 06 30 14 13 32.08 -110.86 1576.00 26.05 110 3.10 01ZXL5JA 2019 06 30 14 13 32.08 -110.86 1527.00 26.35 111 3.60 01ZXL5JA 2019 06 30 14 12 32.08 -110.87 1487.00 27.05 117 3.60 01ZXL5JA 2019 06 30 14 12 32.08 -110.88 1445.00 27.35 127 4.10 01ZXL5JA 2019 06 30 14 12 32.08 -110.88 1397.00 27.55 133 4.60 01ZXL5JA 2019 06 30 14 12 32.08 -110.89 1351.00 28.05 132 4.60 01ZXL5JA 2019 06 30 14 12 32.08 -110.89 1305.00 28.35 124 4.10 01ZXL5JA 2019 06 30 14 12 32.08 -110.90 1260.00 28.35 109 4.10 01ZXL5JA 2019 06 30 14 12 32.09 -110.90 1220.00 28.85 108 4.10 01ZXL5JA 2019 06 30 14 12 32.09 -110.91 1171.00 29.25 103 3.60 01ZXL5JA 2019 06 30 14 12 32.09 -110.91 1104.00 29.55 103 3.10 01ZXL5JA 2019 06 30 14 12 32.10 -110.92 1023.00 28.85 119 2.10 01ZXL5JA 2019 06 30 14 11 32.10 -110.92 944.00 28.85 153 1.00 01ZXL5JA 2019 06 30 14 11 32.10 -110.92 866.00 29.25 104 0.00 01ZXL5JA 2019 06 30 14 11 32.11 -110.93 802.00 31.55 0 0.00 020UFPRA 2019 06 30 14 53 35.03 -96.72 11582.00 -52.15 24 6.70 020UFPRA 2019 06 30 14 38 35.03 -94.32 11552.00 -52.55 30 14.40 020UFPRA 2019 06 30 14 23 35.01 -91.94 11552.00 -53.05 66 2.10 020UFPRA 2019 06 30 10 45 30.08 -83.79 11552.00 -53.75 233 12.90 020UFPRA 2019 06 30 14 08 34.09 -89.61 10973.00 -50.05 107 2.60 020UFPRA 2019 06 30 13 53 34.04 -87.33 10942.00 -49.55 161 6.70 020UFPRA 2019 06 30 10 30 29.02 -82.58 10668.00 -46.25 249 4.60 Decoder source code ===================== 1. Source code directory:: > cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dcmicapsamdar/sorc 2. Subroutines to decode Aircraft data * :code:`afdcod.f` .. note:: * The :code:`pirep.tbl`, :code:`airep.tbl` are not used, although they are required as arguments and read in. * The path and file name of :code:`cmicaps_amdar_data` file are hard coded in the subroutines. 4. Compile the code :: > make Decode and convert to BUFR format ================================= 1. enter into the exec directory :: > cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dcmicapsamdar/exec > ls -la total 1944 drwxr-xr-x 9 xinzhang staff 288 Sep 25 21:24 . drwxr-xr-x 5 xinzhang staff 160 Sep 21 18:45 .. lrwxr-xr-x 1 xinzhang staff 25 Sep 21 18:45 airep.tbl -> ../dictionaries/airep.tbl lrwxr-xr-x 1 xinzhang staff 34 Sep 21 18:45 bufrtab.004 -> ../../decod_shared/fix/bufrtab.004 -rwxr-xr-x 1 xinzhang staff 985984 Sep 21 18:45 decod_dcmicapsamdar lrwxr-xr-x 1 xinzhang staff 25 Sep 21 18:45 pirep.tbl -> ../dictionaries/pirep.tbl -rwxr-xr-x 1 xinzhang staff 410 Sep 21 18:45 run.ksh drwxr-xr-x 3 xinzhang staff 96 Sep 21 18:45 tmp 2. we provide a script to run the decoder in batch mode:: > ./run_dcmicapsamdar.py -s 2018121612 -e 2018121700 -i 1 .. note :: * given the starting datetime and ending datetime, it iterates all cycles (every 1 hours) * the units of interval is hour (-i) * this script call run.ksh 3. run the decoder script :: > run.ksh > cat run.ksh #!/bin/bash export DBNBUFRT=120 export TRANJB=/g5/nwpc_ep8/obsproc/nwprod/ush/tranjb export tank_dir=/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003 export SCREEN="OFF" export DBNROOT=`pwd` rm tmp/* rm decod_dcmicapsadmar.log ./decod_dcmicapsadmar -v 4 -d decod_dcmicapsadmar.log -b 240 -c $1 pirep.tbl airep.tbl bufrtab.004 ls -la tmp/* BUFR_FILES=$(echo tmp/BUFR*) echo ${BUFR_FILES} for file in ${BUFR_FILES} do ${TRANJB} ${tank_dir} ${file} done .. note:: * -c $1 : Set the **current time** (201809120200) used to calculate the time departures of the obs. data. * -b 240 : Number of hours to decode prior to "current" time (default) * The observations with date/time between **current time** - 240 hours and **current time** + 3 are **kept**. 4. The generated BUFR format file will be saved at :: > ls -la tmp/BUFR.0.aircraft.1.1933.1537419287.73 -rw-r--r-- 1 xinzhang staff 1806552 Sep 21 18:45 tmp/BUFR.0.aircraft.1.1933.1537419287.73 Transfer bufr data to BUFR Tanks ================================ * put data in BUFR **tanks**:: > /g5/nwpc_ep8/obsproc/nwprod/ush/tranjb /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003 tmp/BUFR.0.aircraft.1.1933.1537419287.73 > ls -al /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180912/b004/xx003 -rw-r--r-- 1 vagrant vagrant 1828720 Sep 19 22:54 /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180912/b004/xx003 .. note:: * if environmental variable **SCREEN=ON** : * Define **Run Time** is the system time when the tranjb is running. * Only observations with date/time between **Run Time** - 10 days and **Run Time** + 12 hours are kept. * for retrospective run, set **SCREEN=OFF** * /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/yyyymmdd/bmmm/xxsss (where mmm is WMO BUFR message type and xxx is local BUFR message subtype) * 004.003 (in dump group mnemonic aircft): AMDAR format aircraft data from ASDAR/ACARS reporting systems * BUFR format * Arranged by UTC day and continuously grow throughout the day, if you run decoders many time, the content of the file will grow * No QC (other than rudimentary checks inside decoders) * No duplicate checking * Interested users can use utility *debufr* to check the content of the bufr file:: > /g5/nwpc_ep8/obsproc/nwprod/util/exec/debufr /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180912/b004/xx003 the output is in :code:`debufr.out`.