Modify

Opened 4 years ago

Closed 4 years ago

#1648 closed enhancement (fixed)

MPI thorn should auto configure

Reported by: Steven R. Brandt Owned by:
Priority: major Milestone: Cactus_4.3.0
Component: EinsteinToolkit thorn Version: development version
Keywords: Cc:

Description

The attached patch upgrades the MPI configuration thorn in a number of ways. First, and most importantly, it automatically configures mpi based on information obtained from mpicc when MPI_DIR isn't specified.

Configuration is only disabled by setting MPI_DIR = NONE.

In addition to MPI_INSTALL_DIR, it will look at CACTUS_EXT_INSTALL_DIR. Frank says that at one point a general external directory for applications was discussed.

My feeling is that having the install land in configs is usually not what people want. This version emits an error message if you set MPI_DIR to BUILD, but don't set one of the above install dirs. If you really want MPI under configs, you can set MPI_INSTALL_DIR=CONFIGS.

I anticipate pushback on my MPI_DIR=NONE, and MPI_INSTALL_DIR=CONFIGS suggestions, but I thought I'd suggest them regardless.

Attachments (4)

mpi.patch (17.7 KB) - added by Steven R. Brandt 4 years ago.
mpi2.patch (18.2 KB) - added by Steven R. Brandt 4 years ago.
mpi3.patch (18.6 KB) - added by Steven R. Brandt 4 years ago.
mpi4.patch (2.4 KB) - added by Steven R. Brandt 4 years ago.
Look for mpi compiler wrapper under MPI_DIR, if a valid directory is specified there.

Download all attachments as: .zip

Change History (25)

Changed 4 years ago by Steven R. Brandt

Attachment: mpi.patch added

comment:1 Changed 4 years ago by Ian Hinder

The auto-configuration is a nice idea.

I think the MPI thorn should behave like all the other external library thorns, and continue to install in configs by default if you set MPI_DIR to BUILD, without having to set any new variables. The fact that this is possibly not the best long-term solution is outweighed by the fact that this is how it works now (and everyone should not have to update their optionlists for this), and is consistent with the other thorns.

For reference, I believe that logic behind having the libraries in configs is that they are built using the Cactus build environment (compilers, flags, libs etc) and this is different for each configuration. They may not work with other configurations. I think Erik has a clearer picture of the issues relating to this than I do, though. Erik's proposal is that libraries are built outside of Cactus, e.g. using simfactory. I'm not sure whether that mechanism uses the same compilers as Cactus or not.

Changed 4 years ago by Steven R. Brandt

Attachment: mpi2.patch added

comment:2 Changed 4 years ago by Steven R. Brandt

This patch addresses Ian's comment. I believe this patch will make it easier to use MPI on new systems and/or people's laptops/desktops.

comment:3 Changed 4 years ago by Frank Löffler

Status: newreview

comment:4 Changed 4 years ago by Erik Schnetter

I agree with both of Steve's suggestions. However, I recommend that they should be implemented in the same way as is already done in other thorns:

The variable CACTUS_EXT_INSTALL_DIR is superfluous. If you want to install MPI somewhere outside Cactus, then you set MPI_INSTALL_DIR accordingly. If this variable is not set, MPI should be installed into the configs directory.

If you do not want MPI to be build, then one should set MPI_DIR to NO_BUILD. This should effectively disable the configuration logic, assuming that all necessary compiler flags are defined automatically.

comment:5 Changed 4 years ago by Steven R. Brandt

The idea of CACTUS_EXT_INSTALL_DIR is that you could set the directory once, and all external thorns would build there. Maybe that should be a separate ticket/discussion.

Changed 4 years ago by Steven R. Brandt

Attachment: mpi3.patch added

comment:6 Changed 4 years ago by Ian Hinder

I think my only remaining issue with the patch is that it introduces a configure script written in Perl, which is different to the other library configure scripts. While this makes me uncomfortable, and might make it harder to debug (I'm not very familiar with Perl), the additional feature of automatically probing the MPI directories makes it worth committing on balance. If Erik is now OK with the current patch, then so am I.

comment:7 Changed 4 years ago by Erik Schnetter

Status: reviewreviewed_ok

MPI_DIR=NO_BUILD should not search.

Apart from this change the patch seems fine. The Perl script has a much better structure than our Bash scripts.

comment:8 Changed 4 years ago by Steven R. Brandt

Fixed in revision 44.

comment:9 Changed 4 years ago by Frank Löffler

Resolution: fixed
Status: reviewed_okclosed

comment:10 Changed 4 years ago by Roland Haas

Priority: minormajor
Resolution: fixed
Status: closedreopened

The script fails if only MPI_DIR is given. It should then itself set MPI_INC_DIRS and MPI_LIB_DIRS based on that. Currently I get:

$ MPI_DIR=/usr/lib/openmpi perl arrangements/ExternalLibraries/MPI/configure.pl
BEGIN MESSAGE
MPI was manually configured.
END MESSAGE
BEGIN DEFINE
CCTK_MPI 1
HAVE_MPI 1
END DEFINE
BEGIN MAKE_DEFINITION
CCTK_MPI     = 1
HAVE_MPI     = 1
MPI_DIR      = /usr/lib/openmpi
MPI_INC_DIRS =
MPI_LIB_DIRS =
MPI_LIBS     =
END MAKE_DEFINITION
INCLUDE_DIRECTORY $(MPI_INC_DIRS)
LIBRARY_DIRECTORY $(MPI_LIB_DIRS)
LIBRARY           $(MPI_LIBS)

Changed 4 years ago by Steven R. Brandt

Attachment: mpi4.patch added

Look for mpi compiler wrapper under MPI_DIR, if a valid directory is specified there.

comment:11 Changed 4 years ago by Steven R. Brandt

Resolution: fixed
Status: reopenedclosed

comment:12 Changed 4 years ago by Ian Hinder

Resolution: fixed
Status: closedreopened

The result of this change has been to cause mpicxx to be run when you specify MPI_DIR. Why is this needed? If I specify MPI_DIR, then I don't think the compiler wrapper script needs to be run. Can't you deduce the default include and lib directories from this? If they don't exist, you can then try to run the compiler wrapper, but (see #1668) this is less likely to work reliably.

comment:13 Changed 4 years ago by Erik Schnetter

I implemented deducing defaults from MPI_DIR. However, MPI_LIBS cannot be auto-deduced.

One option would be to fail loudly.

Another option would be to set LD_LIBRARY_PATH and equivalents (for other operating systems) automatically based on the library directory for MPI_DIR.

We could also use pkg-info to look for the respective libraries.

What did the code do previously? How did it determine the libraries? Did it just assume that it was an OpenMPI installation?

comment:14 Changed 4 years ago by Ian Hinder

The previous version is here: http://git.barrywardell.net/?p=arrangements/ExternalLibraries/MPI.git;a=blob;f=configure.sh;h=6f32d25a601057e6ecb4076e7a89543357c4712c;hb=f6bf8a9340aaf18c11656e406ac016a5d097581f

It probes some places, and hard-codes the libs to mpi mpi_cxx.

I would say that guessing MPI_DIR/lib and MPI_DIR/include would be a fairly good guess. For the libraries, "mpi" sounds good. Does it make sense to put logic supporting each of the common MPI implementations into the script, or try to get the compiler wrapper approach to work?

comment:15 Changed 4 years ago by Erik Schnetter

The include and library directories are already guessed.

Guessing just "mpi" works only for OpenMPI. We either need to hard-code various MPI implementations, or need to make the wrapper approach work, or need to declare that trying to use an MPI implementation where one can't run "mpicxx" is sign of a broken implementation or a configuration error.

comment:16 Changed 4 years ago by Steven R. Brandt

Both OpenMPI and Mpich provide a mechanism to reveal their command line options when they are run with either --showme or -config_info, and most machines use one of these two. That was the rationale for the script. It means that most laptops and desktops can configure out of the box.

If those mechanisms fail, I invoke mpicxx with "sh -x" in the hopes that mpicxx is a shell script and that I can find the compile line in stderr. Obviously, all this will fail if there is an mpicxx and it doesn't work.

However, it doesn't seem surprising to me that there are some machines that fail all these attempts to auto configure. One can set everything manually for those machines, and mpicxx won't be run.

comment:17 in reply to:  12 Changed 4 years ago by Frank Löffler

Replying to hinder:

The result of this change has been to cause mpicxx to be run when you specify MPI_DIR. Why is this needed?

Apart from the question whether this is needed (which has been answered): is there a problem with this approach? If it does not work for some cases we obviously have to look into it.

comment:18 Changed 4 years ago by Erik Schnetter

I think the problem is that the new system fails in some corner cases, which are different from the corner cases in which the old system failed. In principle, using mpicxx is a sound approach, as it is "the" approach explicitly supported by the MPI implementation.

comment:19 Changed 4 years ago by Erik Schnetter

cmake also looks for mpicxx (and friends) to determine the MPI flags; see <https://github.com/Kitware/CMake/blob/master/Modules/FindMPI.cmake>. cmake is the modern cousin of autoconf.

comment:20 Changed 4 years ago by Ian Hinder

The approach of using mpicxx to automatically probe for the required settings is good. The problem arises because of simfactory, and workarounds to its problems, but it is there nonetheless.

comment:21 Changed 4 years ago by Ian Hinder

Resolution: fixed
Status: reopenedclosed

I think the problems mentioned above are just quirks of the new system, so I am closing this ticket (I was the one who reopened it in the first place). If the above discussion includes a report of a still-existing problem, it should be reported in a new defect ticket.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The ticket will remain with no owner.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.