Modify

Opened 5 years ago

Last modified 5 years ago

#1548 reopened defect

stdout redirection in multithreading scenario

Reported by: gzheng@… Owned by:
Priority: major Milestone:
Component: Cactus Version: development version
Keywords: stdout redirection Cc:

Description

Current stdout redirection scheme implemented in CommandLine.c:CCTKi_CommandLineFinished() (which redirects stdout to a file or /dev/null based on myproc) does not work on multithreaded MPI (e.g AMPI), because it redirects all output (from all threads in a process).
That is, all MPI ranks in a process will be mistakenly directly to one file, or if redirecting to /dev/null, there will be simply no output.

It may be possible to implement a cctk_printf function, which checks myproc and manage the output for each thread.

Attachments (0)

Change History (4)

comment:1 Changed 5 years ago by Roland Haas

Resolution: fixed
Status: newclosed

I believe the bug reporter is confused about threads and processes. An MPI rank is the same as a process. It was always the case that all threads in a processes (or MPI rank) output to the same file. The usual fix to make output legible is to surround CCTK_Warn etc by an OMP CRITICAL section.

We may want to consider including this in the flesh code for CCTK_Warn and CCTK_Info (should be fine since IO is slow so the extra serialization hopefully does not hurt).

comment:2 Changed 5 years ago by Erik Schnetter

Resolution: fixed
Status: closedreopened

AMPI is an MPI implementation that creates threads for each MPI rank. Thus, any operation that acts on a Unix process (e.g. redirecting I/O) will affect multiple MPI ranks.

comment:3 Changed 5 years ago by Roland Haas

Interesting. Why did I close the ticket? That was not quite intended. Anyhow. Thank you for the clarification. I am not sure though what to do about it. We can certainly fairly easily create one output file per MPI rank and store the FILE* pointers inside of a threadprivate (or the equivalent thread local storage type of the compiler if we don't want to rely on OpenMP threads being the same as "threads" in general or that the only threads that we will encounter are OpenMP generated and not eg from pthread in HTTPD) static so that CCTK_Warn etc write to separate files per MPI rank. I am not aware of anything that would allow us to redirect output to stdout on a per-thread basis since he POSIX dup call that we use is per-process (and file ids are per-process entities as far as I know).

comment:4 Changed 5 years ago by Roland Haas

After Gengbing's talk today and studying #1618, would it be possible to do something as simple as this:

static AMPI_TLS FILE *CCTK_stdout = NULL;
...
CCTKi_CommandLineFinished(...)
{
...
if(CCTK_MyProc(cctkGH) != 0) {
  CCTK_stdout = fopen("CCTK_ProcX.out");
} else {
  // or just = stdout since we never close the file handle anyway
  CCTK_stdout = freopen(stdout); 
}
...
}

?

We would then use CCTK_stdout in CCTK_VWarn etc. We'd still keep the dup() code around for the benefit of Fortran code that does write(*,*) etc. and requires redirecting the OS level file descriptors (which may need some code to protect against multiple AMPI ranks trying to redirect the same file descriptor). Such Fortran code would not be compatible with AMPI (not much we can do about this I think unless we want to change the file descriptors each time the thread changes).

This would make this ticket a sub-ticket of #1618 (supporting AMPI).

Modify Ticket

Change Properties
Set your email in Preferences
Action
as reopened The ticket will remain with no owner.
Next status will be 'review'.
as The resolution will be set.
to The owner will be changed from (none) to the specified user.
The owner will be changed from (none) to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.