Modify

Opened 2 years ago

Last modified 2 years ago

#1923 new defect

make utils and simfactory ignore HWLOC=BUILD and try to copy hwloc executables from /usr/bin

Reported by: vassilios.mewes@… Owned by:
Priority: unset Milestone:
Component: SimFactory Version: development version
Keywords: Cc:

Description

Hello!

Trying to install the Einstein toolkit on a new machine, we came across this seemingly wrong behaviour:

setting HWLOC=BUILD in the optionlist, the ET is built with the bundled hwloc successfully.

However, upon building the utilities, it gives an error when it tries to copy hwloc-ls from /usr/bin to /exe/sim

The error results from hwloc-ls being a broken symlink in the system, but the weird thing is that it actually copies the executables from /usr/bin when we have actually built hwloc using the bundled version.

The same behaviour can also be seen here, where some hwloc executables are copied from /usr/bin, while others are copied from the hwloc bundle:

http://lists.einsteintoolkit.org/pipermail/test/2014-January/000047.html

Copying hwloc-assembler from <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/configs/sim/scratch/external/hwloc/bin/hwloc-assembler> to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying hwloc-assembler-remote from <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/configs/sim/scratch/external/hwloc/bin/hwloc-assembler-remote> to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying hwloc-bind from /usr/bin/hwloc-bind to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying hwloc-calc from /usr/bin/hwloc-calc to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying hwloc-distances from <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/configs/sim/scratch/external/hwloc/bin/hwloc-distances> to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying hwloc-distrib from /usr/bin/hwloc-distrib to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying hwloc-info from /usr/bin/hwloc-info to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying hwloc-ls from /usr/bin/hwloc-ls to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying hwloc-ps from /usr/bin/hwloc-ps to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying lstopo from /usr/bin/lstopo to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>
Copying lstopo-no-graphics from <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/configs/sim/scratch/external/hwloc/bin/lstopo-no-graphics> to <https://build.barrywardell.net/job/EinsteinToolkitReleased/ws/exe/sim>

Is this wanted? I guess if one specifies HWLOC_DIR=BUILD, it should only use hwloc executables from the actual hwloc that comes bundled, right?

I have attached the optionlist as well as the config-info and a tar of the config-data folder for the build.

Attachments (3)

config-data.tar.gz (33.7 KB) - added by vassilios.mewes@… 2 years ago.
config-data
config-info (1.3 KB) - added by vassilios.mewes@… 2 years ago.
config-info
config.cfg (3.4 KB) - added by vassilios.mewes@… 2 years ago.
optionlist

Download all attachments as: .zip

Change History (6)

Changed 2 years ago by vassilios.mewes@…

Attachment: config-data.tar.gz added

config-data

Changed 2 years ago by vassilios.mewes@…

Attachment: config-info added

config-info

Changed 2 years ago by vassilios.mewes@…

Attachment: config.cfg added

optionlist

comment:1 Changed 2 years ago by Frank Löffler

This mix is certainly not wanted. Some observations:

For reference: the files that are copied from /usr/bin are also present in the built version. With that, the question is still why some of the files are taken from /usr/bin and some from the built version. Could it be that by default they are taken from /usr/bin, but in case some of them are not present there, the built version is used? To answer that, it would be interesting to know whether, e.g., /usr/bin/hwloc-assembler exists.

In the thorn itself I don't see why, e.g., hwloc-assembler and hwloc-ls should be treated any different. Both are mentioned as standard hwloc binaries.

Looking at https://build.barrywardell.net/job/EinsteinToolkitReleased/101/consoleFull I find "-L/usr/lib" in GENERAL_LIBRARIES, as well as -Wl,-rpath,/usr/lib. I wouldn't expect that there, but I am also not sure if this is related. Both LAPACK and BLAS define their dir to be /usr/lib, but both should be stripped in their configure.sh script (and from the output it looks like that happens).

GENERAL_LIBRARIES also looks odd because of paths for PETSc and hwloc to appear multiple times (but could be due to dependencies).

The option list adds LIBS=-L/usr/lib64, which looks not right, but since this is lib64 and not lib it might also not be related.

Also, /usr/lib appears in INC_DIRS_F, which is strange, and points to MPI (only difference between INC_DIRS and INC_DIRS_F are MPI related). Looking at the output from when PETSc is built, at this point:
MPI_LIB_DIRS=/usr/lib. That shouldn't happen.

Looking at MPI's configure script, it does seem to strip $ENV{MPI_LIB_DIRS}. However, I don't know why it does contain this line:

print "HWLOC_DIR       = $ENV{HWLOC_DIR}\n";

What does MPI have to do with HWLOC? But then, this isn't HWLOC_LIB_DIRS, and shouldn't cause the problem mentioned in this ticket.

One problem, I believe, is found in detect.pl of MPI:

sub strip_lib_dirs {
    my $dirlist = shift;
    my @dirs = split / /, $dirlist;
    map { s{//}{/}g } @dirs;
    @dirs = grep { !m{^/(usr/(local/)?)?lib(64?)/?$} } @dirs;
    return join ' ', @dirs;
}

The regex does not trigger on /usr/lib paths, but should. Thus, MPI adds /usr/lib to the -L options.

Now, whether and/or why this might trigger the hwloc-util problem I don't know. At this point I am probably at least as confused as you are reading this. Too many things look iffy:

  • MPI sets HWLOC_DIR
  • MPI doesn't strip paths properly
  • The option lists adds /usr/lib64 (shouldn't be necessary, but who knows)

comment:2 Changed 2 years ago by Frank Löffler

https://build.barrywardell.net/job/EinsteinToolkitReleased/101/consoleFull also mentions: "hwloc selected, but HWLOC_DIR not set.". HWLOC_DIR=build should not trigger this output. However, build 101 uses ubuntu.cfg that doesn't define HWLOC_DIR, while build 25 (the one references above) probably didn't (Jenkins switched options lists). On the other hand, even without HWLOC_DIR defined, since Jenkins didn't find a system installation, it built the bundled version in 101. Why it didn't find the system version (that apparently is installed) is another matter.

comment:3 in reply to:  1 Changed 2 years ago by Frank Löffler

Replying to knarf:

Looking at MPI's configure script, it does seem to strip $ENV{MPI_LIB_DIRS}. However, I don't know why it does contain this line:

print "HWLOC_DIR       = $ENV{HWLOC_DIR}\n";

What does MPI have to do with HWLOC? But then, this isn't HWLOC_LIB_DIRS, and shouldn't cause the problem mentioned in this ticket.

The reason for that seems to be that the build script makes use of HWLOC_DIR to point the MPI install to that directory in order to build using hwloc.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The ticket will remain with no owner.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from (none) to the specified user.
Next status will be 'confirmed'.
The owner will be changed from (none) to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.