Several tests fail in Jenkins after move to new NCSA build node

Comments (9)

1. 

   Ian Hinder reporter

   • removed comment

   The test machine "login.barrywardell.net" is configured in the same way as the Jenkins build node, and runs on the same hardware (at least it reports the same information in /proc/cpuinfo). I am focusing first on the GRHydro_test_shock_weno test. This fails with the standard ubuntu.cfg from simfactory. However, it passes with [https://bitbucket.org/ianhinder/cactusjenkins/src/d7021a52bd83448db589b2346c43441682eecabb/build.cfg?at=master build.cfg], which is an optionlist which was used before for the test system (I'm not 100% sure when it was changed to ubuntu.cfg; this may coincide with the move to the NCSA machine). There are several differences, and I am working my way through them.

   The possible differences responsible are:

   diff build.cfg ubuntu.cfg with irrelevant bits removed:

   33,36c30,33
   < CFLAGS   = -g3 -std=gnu99
   < CXXFLAGS = -g3 -std=gnu++0x
   < F77FLAGS = -g3 -fcray-pointer -ffixed-line-length-none
   < F90FLAGS = -g3 -fcray-pointer -ffixed-line-length-none -ffree-line-length-none
   ---
   > CFLAGS   = -g3 -march=native -std=gnu99
   > CXXFLAGS = -g3 -march=native -std=gnu++0x
   > F77FLAGS = -g3 -march=native -fcray-pointer -ffixed-line-length-none -fno-range-check
   > F90FLAGS = -g3 -march=native -fcray-pointer -ffixed-line-length-none -fno-range-check
   39,43d35
   < VECTORISE                  = yes
   < VECTORISE_ALIGNED_ARRAYS   = no
   < VECTORISE_INLINE           = no
   < VECTORISE_STREAMING_STORES = no
   < 
   49d40
   < # -check-uninit fails for asm output operands
   61,64c52,55
   < C_OPTIMISE_FLAGS   = -O2 #-ffast-math
   < CXX_OPTIMISE_FLAGS = -O2 #-ffast-math
   < F77_OPTIMISE_FLAGS = -O2 #-ffast-math
   < F90_OPTIMISE_FLAGS = -O2 #-ffast-math
   ---
   > C_OPTIMISE_FLAGS   = -O2 -ffast-math -fno-finite-math-only
   > CXX_OPTIMISE_FLAGS = -O2 -ffast-math -fno-finite-math-only
   > F77_OPTIMISE_FLAGS = -O2 -ffast-math -fno-finite-math-only
   > F90_OPTIMISE_FLAGS = -O2 -ffast-math -fno-finite-math-only
   91,93c82,87
   < 
   < HDF5_ENABLE_CXX     = no
   < HDF5_ENABLE_FORTRAN = no
   ---
   96,100d89
   < MPI_INC_DIRS = /usr/include/mpich2 /usr/include/mpich
   < MPI_LIB_DIRS = /usr/lib
   < MPI_LIBS     = mpich  mpl
   103,119d91
   < 
   < BLAS_DIR    = NO_BUILD
   < LAPACK_DIR  = NO_BUILD
   < LIBJPEG_DIR = NO_BUILD
   < 
   < ZLIB_DIR    = /usr
   < ZLIB_LIB_DIRS = /usr/lib/x86_64-linux-gnu
   < ZLIB_INC_DIRS = /usr/include
   

   Removing -march=native from ubuntu.cfg does not make the test pass.

   • 2017-03-13T16:18:16+00:00
2. 

   Ian Hinder reporter

   • removed comment

   Removing both native and fast-math makes GRHydro_test_shock_weno pass, but boostedpuncture still fails, suggesting that there is more than one problem here.

   I have created a couple of tables to keep track of results at [https://docs.einsteintoolkit.org/et-docs/Test_Failures Test failures].

   • 2017-03-13T17:28:04+00:00
3. 

   Frank Löffler

   • removed comment

   Action items from call on Mar 20th: * Roland: Try if increasing the resolution makes the hydro tests work (might use too low resolution anyway) * Erik: See how much fast math changes performance / reproducibility on a "modern Haswell" machine

   • 2017-03-20T10:09:41+00:00
4. 

   Ian Hinder reporter

   • removed comment

   From the wiki page, a conclusion might be that in newer versions of gcc, fast-math may be much more aggressive, and cause the GRHydro.GRHydro_test_shock_weno/1procs test to fail. Older versions with ubuntu.cfg (and hence fast-math) seem to work, whereas newer versions don't. Omitting fast-math on the newer version also works. We talked on the call about possibly making the optionlists more conservative to avoid all the time spent on these sorts of problems, but we would like to see benchmark results before making such a decision.

   • 2017-03-20T10:22:16+00:00
5. 

   Roland Haas

   • removed comment

   Pull request for GRHydro_test_shock_weno: https://bitbucket.org/einsteintoolkit/einsteinevolve/pull-requests/10/grhydro-regenerate-weno-shocktube-test/diff

   • 2017-06-28T23:00:00+00:00
6. 

   Frank Löffler

   • changed status to open
   • removed comment

   This changes weno_eps for the testsuite. That's ok for the release, so (review_ok), but should we maybe change the default as well (after this release), if this is indeed so prone to problems?

   • 2017-06-29T14:06:05+00:00
7. 

   Frank Löffler

   • changed status to open
   • removed comment

   • 2017-06-29T14:06:09+00:00
8. 

   Roland Haas

   • changed status to resolved
   • removed comment

   Applied as git hash 01b6e12 of einsteinevolve.

   I do not think we can easily change the default since the defaults are for backwards compatibility and not necessarily "good starting point" values. It is also not clear that 1e-26 is actually a bad value for typical problems that are not a shocktube test.

   • 2017-06-29T14:28:36+00:00
9. 

   Roland Haas

   • edited description
   • changed status to closed

   • 2019-02-21T19:59:18+00:00
10. Log in to comment