Opened 4 years ago

Last modified 4 years ago

#1722 confirmed enhancement

hwloc could warn the user if a process spans more than one NUMA node

Reported by: Ian Hinder Owned by:
Priority: minor Milestone:
Component: EinsteinToolkit thorn Version: development version
Keywords: Cc:


If running on a NUMA architecture, performance could be very bad if a process allocates memory from one NUMA node, and also has threads running on cores of another NUMA node, as they will attempt to access the memory from the first node, which will be slow. It is probably better in that situation if there is one process per NUMA node, so that each process allocates memory it has fast access to.

hwloc could detect whether the threads in a process are all on the same NUMA node or not, and output a warning (or error?) if there are threads running on more than one NUMA node in the same process.

See also #1446 and #1528.

Attachments (0)

Change History (2)

comment:1 Changed 4 years ago by Ian Hinder

Status: newconfirmed

comment:2 Changed 4 years ago by Erik Schnetter

In the same spirit, hwloc should detect if the current MPI process started up with a sub-ideal memory affinity. Other libraries, in particular MPI, may be initialized before hwloc runs, and these would see and use the sub-ideal memory affinity. This is actually causing slow-downs on some systems.

Another remedy for this could be to run hwloc very early, before MPI is initialized. This would require passing in the current MPI rank, which may not be possible before initializing MPI.

Modify Ticket

Change Properties
Set your email in Preferences
as confirmed The ticket will remain with no owner.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from (none) to the specified user.
The owner will be changed from (none) to anonymous.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.