2009/02/02 esmp tag: 8 model yrs/day (2.5 hrs/model yr) in omsk 10 model yrs/day with perth strat_cloud speedup ------------------------------------------------------------------------------------------------------------------------------- Job Name Status Notes ------------------------------------------------------------------------------------------------------------------------------- ESM2.1 ****** ESM2.1U_Control-1860_perth_stocks_sep182008 testing Perth+co2 unit mods+topaz diags+stocks - lm3v non_mosaic code ESM2.1U_Control-1860_perth_alpha_o2_mmm_ddyyyy testing Add o2 mods to perth + mods for co2,topaz diags .... ESM2.1 Historical Runs ********************** ESM2.1U_Control-1860_baseline_co2rest_286_co2rad_286_qco2_286 stopped 0008 test run: use fmsdataset defaults instead of MS versions ESM2.1U_Control-1860_data_ov_for_co2_restore_rad_nov182008 done 0105 identical to ESM2.1U_Control-1860_baseline_nov182008 except with topaz fluxes/rates diagnostics ESM2.1U_Control-1860_baseline_nov182008 done 0100 extend Feb19 spinup: use lnd/ocn co2 data overrides at 286 Perth + co2 unit mods + alpha_o2 mods + topaz diags with lm3v non_mosaic code Keep wsink=10 (as in feb19 spinup) 100 year run to check climate. co2 restore=false, co2 rad=false, qco2=286, rad_aerosols=control ESM2.1U_Control-1860_co2rest_286_co2rad_286_qco2_286 0219 esmd Control for historical runs (same code as extended run), but use co2 restoring instead of lnd/ocn co2 data overrides co2 restore=286, co2 rad=286 , qco2=286, rad_aerosols=control restarted 12/30/2008 from year 0157 ESM2.1U_1860_H1_co2rest_286_co2rad_286_qco2_interactive 2256 esmd Historical run 1. Start from year 101 of historical control. co2 restore=286, co2 rad=286, qco2=interactive, rad_aerosols=control 2009/01/29 restart file for year 2248 missing. reload_commands file was pointing to year 2248. No history file for year 2247. restarted model from year 2247 ESM2.1U_1860_H2_co2rest_hist_co2rad_286_qco2_interactive done 2000 Historical run 2. Start from year 101 of historical control. co2 restore=historical, co2 rad =286, qco2=interactive, rad_aerosols=control ESM2.1U_1860_H3_co2rest_286_co2rad_hist_qco2_286 1927 esmd Historical run 3. Start from year 101 of historical control. co2 restore=286, co2 rad=historical, qco2=286, rad_aerosols=historical 2009/01/29 history file for year 1918 missing. (model was at year 1921) restarted model from year 1918 ESM2.1U_1860_H4_co2rest_hist_co2rad_hist_qco2_interactive done 2000 Historical run 4. Start from year 101 of historical control. co2 restore=historical, co2 rad=historical, qco2=interactive, rad_aerosols=historical ESM2.1U_1860_H5_co2rest_hist_co2rad_286_qco2_286 1948 esmd Historical run 5. Start from year 101 of historical control. co2 restore=historical, co2 rad=286, qco2=286, rad_aerosols=control ESM2.1 SRES A1B Runs ********************** ESM2.1U_H2_SRES_A1B_co2rest_a1b_co2rad_286_qco2_interactive 2183 esmp Continue ESM2.1U_1860_H2_co2rest_hist_co2rad_286_qco2_interactive from year 2001, but use A1B forcing. 2009/01/29 restart file for year 2147 missing. reload_commands file was pointing to year 2147. restarted model from year 2146 ESM2.1U_H3_SRES_A1B_co2rest_286_co2rad_a1b_qco2_286 Continue ESM2.1U_1860_H3_co2rest_286_co2rad_hist_qco2_286 from year 2001, but use A1B forcing. ESM2.1U_H4_SRES_A1B_co2rest_a1b_co2rad_a1b_qco2_interactive 2056 esmp Continue ESM2.1U_1860_H4_co2rest_hist_co2rad_hist_qco2_interactive from year 2001, but use A1B forcing. 2009/01/29 history file for year 2019 missing. (model was at year 2025) restarted model from year 2019 ESM2.1U_H5_SRES_A1B_co2rest_a1b_co2rad_286_qco2_286 Continue ESM2.1U_1860_H4_co2rest_hist_co2rad_hist_qco2_interactive from year 2001, but use A1B forcing. ............................................................................................................................................................... ESM2M ***** ESM2M_1860_darkiceA_submeso_river_regrid_prescribed_co2 (dec302008) 0064 esmd Perth + conservation fixes + co2 units + river_regrid + updated topaz (dec31) ESM2M_1860_darkiceA_submeso_prescribed_co2 (dec092008) stopped 0052 Perth + conservation fixes + co2 units + updated topaz (dec17) 140 pes. not loadbalanced ESM2M_1860_darkiceA_submeso_river_regrid_prescribed_co2 (dec092008) stopped 0018 Perth + conservation fixes + co2 units + river_regrid + updated topaz (dec17) (years 1 and 2 were on 140pes) resubmitted 2008/12/24 on 110pes: load balanced test on 110pes: change atmos_npes=30, fv_layout to 1,30 ESM2M_1860_darkiceA_submeso_sweby_all_prescribed_co2 (oct272008) suspended 0002 Perth + conservation fixes + co2 units + advect_sweby_all = true ESM2M_1860_darkiceA_submeso_prescribed_co2 (oct272008) suspended 0052 Perth + conservation fixes + co2 units ESM2M_1860_darkiceA_submeso_river_regrid_prescribed_co2 (oct272008) suspended 0035 Perth + conservation fixes + co2 units + river_regrid ............................................................................................................................................................... ESM2G ***** ESM2G_1860_mica_river_regrid_bergs_prescribed_co2 (dec312008) 0044 esmd Perth + conservation fixes + co2 units + river_regrid + updated topaz (dec31) crashed in year 45 with Nans ESM2G_1860_mica_bergs_prescribed_co2 (dec042008) stopped 0096 Perth + conservation fixes + co2 units + updated topaz (dec04) 120 pes. not loadbalanced ESM2G_1860_mica_river_regrid_bergs_prescribed_co2 (dec162008) stopped 0076 Perth + conservation fixes + co2 units + river regrid + updated topaz (dec17) load balanced test on 129pes. 8 yrs/day on priority tag ESM2G_1860_mica_bergs_prescribed_co2 (oct282008) suspended 0068 Perth + conservation fixes + co2 units ESM2G_1860_mica_river_regrid_bergs_prescribed_co2 (oct282008) crashed 0033 Perth + conservation fixes + co2 units + river regrid 2008/11/13 restarted from year 33 crashed again GOLD Day 12065.000: Nan ............................................................................................................................................................... CM2.1 ***** CM2.1U_Control-1860_D4 done 2109 CM2.1 with forcing fields for land/ocean spinups (start year = 2060) 50 year run 2008/11/14 suspended with frepriority 2008/11/24 crashed model year 2089 - bad temperatures. applied bandaid ............................................................................................................................................................... OCEAN_ICE ********* ............................................................................................................................................................... ESM2.4 ****** ESM2.4SQ_A_Control_1990_B5_topaz_river_regrid_nov052008 stopped 0003 CM2.4 + Topaz + river regrid timed out and killed. Suspect model is blowing up and therefore running out of time. ESM2.4SQ_A_Control_1990_B5_topaz_nov052008 stopped 0003 CM2.4 + Topaz timed out and killed. Suspect model is blowing up and therefore running out of time. ESM2.4SQ_A_Control_1990_B5_topaz_river_regrid_nov052008_1x0m8d_660pe ran day 9 CM2.4 + Topaz + river regrid ESM2.4SQ_A_Control_1990_B5_topaz_river_regrid_nov052008_2x0m4d_660pe ran day 5 CM2.4 + Topaz + river regrid unable to compare - only wrote out day 5. ESM2.4SQ_A_Control_1990_B5_topaz_nov052008_1x0m8d_660pe crashed CM2.4 + Topaz - to be rerun ESM2.4SQ_A_Control_1990_B5_topaz_nov052008_2x0m4d_660pe crashed CM2.4 + Topaz - to be rerun CM2.4SQ_A_Control-1990_B5_O_2008_03_topaz_1x0m2d_660pe.rerun ok rerun of CM2.4 + Topaz in omsk_2008_03 CM2.4SQ_A_Control-1990_B5_O_2008_03_topaz_3months_1x0m2d_660pe delete CM2.4SQ_A_Control-1990_B5_O_2008_03_topaz_3months crashed bscip OBGC IC created. ran 1 month, but no output ?? crashed (sat vap pres overflow) with dt_cpld etc = 1800. changed to 1200 and submitted 3 year run. (if crash, check obgc IC - glodap) (1 month run ran successfully with dt_cpld etc = 1200). Model blew up in TOPAZ in Sept of year 3. Save files and restart from July. Run 2 month segments. 2008/06/23 Model did not show problem in Sept. Still running. Stopped Nov of year 3. timed out and killed. Suspect model is blowing up and therefore running out of time. ============================================================================================================================================ COMPLETED/STOPPED JOBS ESM2.1 ------ ESM2.1U_Control-1860_baseline_nov152008 stopped 0015 extend Feb19 spinup: use lnd/ocn co2 data overrides at 286 Perth + co2 unit mods + wsink=20 + alpha_o2 mods + topaz diags with lm3v non_mosaic code 100 year run to check climate. co2 restore=false, co2 rad=false, qco2=286, rad_aerosols=control Need to check this code ESM2.1U_Control-1860_extend_feb19spinup (nov112008) stopped 0028 extend Feb19 spinup: use lnd/ocn co2 data overrides at 286 Perth + co2 unit mods + wsink=20 + topaz diags with lm3v non_mosaic code 100 year run to check climate. co2 restore=false, co2 rad=false, qco2=286, rad_aerosols=control ESM2.1U_Control-1860_perth_co2_units_lm3v_non_mosaic stopped 0028 Perth + co2 unit mods + topaz diags - lm3v non_mosaic code lm3v_mosaic giving different answers in SM2.1 (seed transport is affected by land area), so we will not be switching ESM2.1 to lm3v_mosaic (2008/11/17) ESM2.1U_Control-1860_perth_co2_units_lm3v_mosaic stopped 0012 Perth + co2 unit mods + topaz diags + lm3v_mosaic code lm3v_mosaic giving different answers in SM2.1 (seed transport is affected by land area), so we will not be switching ESM2.1 to lm3v_mosaic (2008/11/17) ESM2.1U_Control-1860_O_2008_03_stocks_test1 to be redone in Perth ESM2.1U_Control-1990_lm2_jun182008 crashed Equivalent to CM2.4 hi-res + topaz. passed regression tests (after fixing bug in topaz). production run failed: sat vap pressure overflow ESM2.1U_Control-1860_dec192007_accelerate_samesign crashed 0119 start from year 50. crashed year 60 - no space on device. Resubmitted. system failover stopped model at year 120. Try to resubmit from year 119. check script ESM2.1U_Control-1860_dec192007_accelerate_linear crashed 0090 start from year 50. crashed in year 90. restarted from 0089 - crashed in year 90 again. ESM2.1U_Control-1860_perth (jul312008) done test perth release. regression tests passed ESM2.1U_Control-1860_preP_jul182008 done prePerth + updates (lm3v update from 11/07 + other k_fw nml change) regression tests passed ESM2.1U_Control-1860_jun162008 done test co2_dry mods and minor change to topaz. fixed bug in topaz: regression tests passed. ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_spinup_feb192008 1000 done stopped at year 200, restarted with equilibrated soil C obgc changed at year 263. stopped at year 357 to recreate year 330 history file. ESM2.1U_Control-1860_dec192007_free_co2_v2_fixed2 done 0401 started from year 400. Run 1 year only with prescribed co2. Use year 400 restart from dec 19 spinup with no changes. compare to year 401 of dec19 spinup: identical ESM2.1U_Control-1860_dec192007_free_co2_v2_fixed done 0401 started from year 400. Run 1 year only with prescribed co2. Use year 400 restart from dec 19 spinup with changes for free co2 run (no qco2, atm co2 = 286/(1+q)) compare to year 401 of dec19 spinup: co2 and qco2 different. ESM2.1U_Control-1860_dec192007_free_co2_v2 done 0705 started from year 400. changed atmco2 IC = 286e-6/(1+q), added co2_dry diagnostic year 414 history file is bad. recreated from raw cpio year 518 history file is bad. could not recreate from raw cpio may 16: stopped at year 525 to recreate year 518 history file. may 22: crashed at year 530 - bad temperatures (sat vap press) year 528 history bad. restarted from beginning of year 528. year 529 history/restart did not compare. 2008/06/10 stopped run at year 0544. restarted from year 0518. ESM2.1U_Control-1860_dec192007_free_co2_plus_rad done 0427 started from year 400. stopped at year 427. Check atmco2 IC. co2 units likely incorrect (dry vs moist) ESM2.1U_Control-1860_dec192007_free_co2_qco2_diag done 0481 started from year 400. stopped at year 481. Check atmco2 IC. ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_free_co2 done 0416 started from year 400. stopped (atmos_co2 was not reset). ESM2.1U_Control-1860_dec192007_dup_qco2_diag done regression tests passed compared to dec19 run. 1 year production run: restart and hist files identical to dec19 run. ESM2.1U_Control-1860_apr242008 done identical to mar202008 - (updated obgc). passed regression tests. restart files identical to mar202008 tests. ESM2.1U_Control-1860_mar202008 done updated obgc. scaling test failed. scaling test resubmitted - passed. NOTE: repeat regression tests after upgrade. ESM2.1U_Control-1860_O_2007_10_historical_co2_phot_p done 1885 started from year 1860. (version1 stopped at year 1865) version2 stopped at year 1885 - check atmco2 IC. ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_spinup_feb142008 done 0024 stopped at year 25. Start new spinup. ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007 done 0504 stopped at year 504 ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_accelerate_ap10_af5 crashed 0110 started from year 50. crashed at start of year 110 fix pending - try different acceleration instead ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_accelerate_test4 done 0059 started from year 50. stopped at year 0059. ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_accelerate_test3 done 0139 started from year 50. Stopped at year 139. ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_anderson95 done 0100 stopped at year 100 ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_chl_true done 0100 stopped at year 100 ............................................................................................................................................................... ESM2M (ESM2.1M) --------------- ESM2M_Control-1860_generic (aug142008) 0008 Perth + opacity bugfix ESM2M_Control-1860_non_generic (aug142008) 0006 Perth + opacity bugfix ESM2M_Control-1860_generic_preP_jul212008 prePerth + updates: failed restart test ESM2M_Control-1860_non_generic_preP_jul212008 prePerth + updates: failed restart test ESM2M_Control-1860_preP_generic_jun232008 crashed passed regression tests (irr_mem restart quirk) unable to run 1 month (try fixing diag table) changed domains_stack_size to 5000000, but did not help problem is total SW < 0 -> fvis < 0 -> neg sq root in ocean_shortwave. temporary fix from Niki - need fix from Mike W in ice model. ESM2M-coarse check with Eric moved to testing tag. compilation failed. start from omsk_2008_03: compiled ok. regression tests ok. try to make debug executable: compile failed. got updated mkmf debug template - compile ok. ESM2M_Control-1860_preP_generic_jun232008_IC2 done use same IC as non_generic case, corrected diag table this was a check to see if IC was cause of crash ESM2M_Control-1860_preP_non_generic_jun232008 done passed regression tests (irr_mem restart quirk) can run for 3 months ESM2M_Control-1860_P_jun112008 crashed esmd test ESM2M generic topaz + cfc modify TOPAZ IC to include new tracers (eg sidet) om3_ecosystem_mdfl_generic crashed esmd test generic topaz. Nan in year 1, month 4. Nan in 1990/04/07. try changing ice_dhdt ESM2.1Mc_om1p7h_1860_O_2008_03_lm2_mar212008 failed restart test, but checksums okay. ESM2.1Mc_om1p7h_1860_O_2008_03_lm3v_mar212008 not run ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p1_jan172008 stopped 0008 (Suspended with frepriority 01/22/08) ............................................................................................................................................................... ESM2G ----- ESM2G_realepi_generic_no_diffuse 0002 1860. Perth + realepi. undef diffuse_ml_to_interior ESM2G_realepi_non_generic_no_diffuse 1860. Perth + realepi. undef diffuse_ml_to_interior ESM2G_generic_chl_true 0007 1860. prePerth + updates. read chl from file. jul152008 ESM2G_non_generic_chl_true 0009 1860. prePerth + updates. read chl from file. jul152008 ESM2G_generic done 0020 1860. land cold start. prePerth + updates fixed xml/runscript to pick up HIM.res.nc ESM2G_non_generic done 0020 1860. land cold start. prePerth + updates fixed xml/runscript to pick up HIM.res.nc ESM2G_Control-1860_preP_non_generic_jun242008 passed restart test. scaling test passed Note: TOPAZ fields incorrect problem is that HIM.res.nc IC is not picked up and model initializes internally. ESM2G_Control-1860_preP_generic_jun242008 passed restart test. scaling test failed. Note: TOPAZ fields incorrect problem is that HIM.res.nc IC is not picked up and model initializes internally. ESM2G_Control-1860_P_jun112008 crashed test ESM2G generic topaz + cfc ............................................................................................................................................................... OCEAN_ICE --------- mom4p0_sis_ciafv2_O_2007_10_diurnal_true_dec192007_extend done 2235 extend control run, use new ciaf forcing started from year 2281. mom4p0_sis_ciafv2_O_2007_10_diurnal_true_dec192007_pertco2_extend done 2235 extend perturbed run, use new ciaf forcing started from year 2281. mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_pertco2 done 2282 started from year 2136, stop at year 2280 problems with co2_history file. (time attributes) - fixed ice_dhdt=40 did not work. changed to 260, then 280. NOTE: time_interp crash - check on this. resubmitted job - ran despite time_interp crash model ran to year 2152, but system crashed 2008/04/17 model started over from year 2136 (see stdout file). mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_fed_flux_half done 2188 started from year 2097, ice_dhdt=260, stop at year 2188 mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_fed_flux_2x done 2188 started from year 2097, ice_dhdt=260, stop at year 2188 mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_fed_flux_5x done 2188 started from year 2097, ice_dhdt=260, stop at year 2188 mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_fed_flux_new done 2188 started from year 2097, ice_dhdt=260, stop at year 2188 mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_p2nstatic_true done 2320 stop at year 2320 crash in year 2321 - did not change dhdt to run longer mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007 done 2320 1959-2004, 2050, 2096, 2142, 2188, 2234, 2280, 2326 stopped Feb 11 at year 2321 (increase dhdt > 400 to keep going) om3_ecosystem1_jan142008 stopped 0100 8 model yrs/day stopped Feb 13 at year 100 mom4p1_cpld_om1p7_spinup ---- ============================================================================================================================================ CIAF model runs 46 year loops (skip 1958) sfc diags for last 7 years of loop (eg 1998-2004) - modify pp section of xml as needed ocean_topaz_sfc = yrend-6 ice_dhdt changes after crashes loop 1 1959-2004 1998 start at 160, change to 200 at 2004/06/30 loop 2 2005-2050 2044 change to 240 after 2045/06/30 loop 3 2051-2096 2090 change to 260 after 2091/06/30 loop 4 2097-2142 2136 change to 280 after 2137/06/30 loop 5 2143-2188 2182 change to 300 after 2183/06/30 loop 6 2189-2234 2228 change to 310 after 2229/06/30 loop 7 2235-2280 2274 change to 350 after 2238/10/28, change to 400 after 2275/06/30 loop 8 2281-2326 2320 crash in year 2321 - did not change dhdt to run longer. ============================================================================================================================================ lst $archive/omsk/ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_spinup_feb192008/history/ lst $archive/omsk/ESM2.1U_Control-1860_dec192007_free_co2_v2/history lst $archive/omsk/mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_pertco2/history/ myfreppcheck -A -x ~/omsk/rts/ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_feb192008.xml ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_spinup_feb192008 myfreppcheck -A -x ~/omsk/rts/ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_free_co2.xml ESM2.1U_Control-1860_dec192007_free_co2_v2 myfreppcheck -A -x ~/omsk/rts/ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007.xml ESM2.1U_Control-1860_dec192007_accelerate_samesign myfreppcheck -A -x ~/omsk/rts/ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007.xml ESM2.1U_Control-1860_dec192007_accelerate_linear myfreppcheck -A -x ~/omsk/rts/mom4p0_sis_topaz_ciaf_diurnal_O_2007_10_dec192007.xml mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_pertco2 myfreppcheck -A -x ~/omsk/rts/mom4p0_sis_topaz_ciaf_diurnal_O_2007_10_dec192007.xml mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_p2nstatic_true myfreppcheck -A -x mom4p1_cpld_jan142008_v2.xml om3_ecosystem1_jan142008 ============================================================================================================================================