1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

Switch Cactus to 64 bits

Issue #1610 new
Erik Schnetter created an issue

Although pointers can have 64 bits, integers in Cactus are de facto (since we use "int" in many places) restricted to 32 bits. This limit is easily reached, and this has in fact been a problem on at least two occasions.

One issue comes from Carpet's numbering of grid points. With 25 refinement levels, the coarse grid can be at most 127^3, and this limit was reached in production simulations years ago.

Another issue comes from large 1D arrays. It is easily possible to have a 1D array with 10M (10^7) elements that is replicated across processes (distrib=const). In this case, one can use at most 200 MPI processes.

We should devise a plan to switch Cactus to 64 bits. One option would be to replace most "int" by "CCTK_INT", so that those affected can switch these to 64 bits.

I want to note that using 64-bit loop counters is often slightly faster than using 32-bit loop counters (on 64-bit architectures), since the 32->64 bit conversion for array indexing can be omitted.

Keyword:

Comments (3)

  1. Erik Schnetter reporter
    • removed comment

    I suggest the following:

    • Correct all thorns to handle when CCTK_INT is different from int
    • Change a few core functions to use CCTK_INT instead of int, in particular lsh, gsh, CCTK_GFINDEX3D etc. -- anything that handles grid point counts
    • Update examples, documentation to use CCTK_INT for looping over grid points
    • Default CCTK_INT to be CCTK_INT8 on 64-bit platforms
  2. Roland Haas
    • removed comment

    I am not sure if this is really what we (should) want.

    It seems to me that CCTK_INT is the type of grid variables and parameters while "int" is the type of the index variables (or should be, admittedly right now this is all muddled). I think the cleanest solution would be one integer type for grid functions (things that come to mind would be the desire for HDF5 datasets to have the same type indep. on whether they are written on 32 or 64 bit platforms although HDF5 will probably do the translation from CCTK_INT-when-writing to CCTK_INT-when-reading for you if you ask for HDF5_NATIVE_INT when reading) and distinct integer type (something like size_t or ptrdiff_t for a signed type) for grid indices which can be 64/32bit depending on the machine or just always 64bit depending on a compile time flag.

  3. Erik Schnetter reporter
    • removed comment

    If you need an integer grid functions with a particular range, then you would be using CCTK_INT4 or CCTK_INT8. CCTK_INT is a "natural integer type", same as CCTK_REAL is a "natural floating-point type". If you run on a machine where ints have 64 bits, then CCTK_INT has 64 bits.

    Using ptrdiff_t doesn't interoperate well with Fortran. We can introduce a new type for this -- CCTK_PTRDIFF_T or some equivalent name --and make this type available e.g. for aliased functions etc.

    CCTK_INT is currently most used in aliased functions as well as in source code as an integer type e.g. for index variables and similar, and less for grid functions where a particular number of bits is expected. Thus introducing CCTK_PTRDIFF_T would require channging a lot of code, whereas the places where people use CCTK_INT in grid functions in a way that would break when CCTK_INT has 64 bits is very rare.

  4. Log in to comment
Assignee
–
Type
enhancement
Priority
minor
Status
new
Component
Cactus
Milestone
–
Version
development version
Votes
0
Watchers
0
Jira Software: the preferred issue tracker for Bitbucket. Join the team!