stdout redirection in multithreading scenario

Issue #1548 open
Gengbin Zheng created an issue

Current stdout redirection scheme implemented in CommandLine.c:CCTKi_CommandLineFinished() (which redirects stdout to a file or /dev/null based on myproc) does not work on multithreaded MPI (e.g AMPI), because it redirects all output (from all threads in a process). That is, all MPI ranks in a process will be mistakenly directly to one file, or if redirecting to /dev/null, there will be simply no output.

It may be possible to implement a cctk_printf function, which checks myproc and manage the output for each thread.

Keyword: stdout
Keyword: redirection

Comments (4)

  1. Roland Haas
    • changed status to resolved
    • removed comment

    I believe the bug reporter is confused about threads and processes. An MPI rank is the same as a process. It was always the case that all threads in a processes (or MPI rank) output to the same file. The usual fix to make output legible is to surround CCTK_Warn etc by an OMP CRITICAL section.

    We may want to consider including this in the flesh code for CCTK_Warn and CCTK_Info (should be fine since IO is slow so the extra serialization hopefully does not hurt).

  2. Erik Schnetter
    • changed status to open
    • removed comment

    AMPI is an MPI implementation that creates threads for each MPI rank. Thus, any operation that acts on a Unix process (e.g. redirecting I/O) will affect multiple MPI ranks.

  3. Roland Haas
    • removed comment

    Interesting. Why did I close the ticket? That was not quite intended. Anyhow. Thank you for the clarification. I am not sure though what to do about it. We can certainly fairly easily create one output file per MPI rank and store the FILE* pointers inside of a threadprivate (or the equivalent thread local storage type of the compiler if we don't want to rely on OpenMP threads being the same as "threads" in general or that the only threads that we will encounter are OpenMP generated and not eg from pthread in HTTPD) static so that CCTK_Warn etc write to separate files per MPI rank. I am not aware of anything that would allow us to redirect output to stdout on a per-thread basis since he POSIX dup call that we use is per-process (and file ids are per-process entities as far as I know).

  4. Roland Haas
    • removed comment

    After Gengbing's talk today and studying #1618, would it be possible to do something as simple as this:

    static AMPI_TLS FILE *CCTK_stdout = NULL;
    ...
    CCTKi_CommandLineFinished(...)
    {
    ...
    if(CCTK_MyProc(cctkGH) != 0) {
      CCTK_stdout = fopen("CCTK_ProcX.out");
    } else {
      // or just = stdout since we never close the file handle anyway
      CCTK_stdout = freopen(stdout); 
    }
    ...
    }
    

    ?

    We would then use CCTK_stdout in CCTK_VWarn etc. We'd still keep the dup() code around for the benefit of Fortran code that does write(*,*) etc. and requires redirecting the OS level file descriptors (which may need some code to protect against multiple AMPI ranks trying to redirect the same file descriptor). Such Fortran code would not be compatible with AMPI (not much we can do about this I think unless we want to change the file descriptors each time the thread changes).

    This would make this ticket a sub-ticket of #1618 (supporting AMPI).

  5. Log in to comment