Want to define constants in parameter files

Issue #1554 closed
Erik Schnetter created an issue

I would like to define constants in parameter files, such as:

$rho = 1
grid::dx = 1.0/$rho
IOASCII::out_every = 10*$rho

The expansion of expressions is already implemented, but is limited to parameters. Being able to define constants would be very cool.

Keyword:

Comments (35)

  1. Steven R. Brandt
    • changed status to open
    • assigned issue to
    • removed comment

    Yes, this would be quite easy to implement. I think I may even have suggested this once, but concern was expressed over making the par files more complex. If people think this is a good idea, I will do it.

  2. Barry Wardell
    • removed comment

    An alternative which is already available is the rpar file format supported by SimFactory. That is, whenever SimFactory encounters a parameter file with the .rpar extension, it assumes it is executable and will produce a .par file with the same base name when executed. It then executes this file and runs Cactus with the generated .par file. This allows almost arbitrary flexibility, including constants and variables by writing rpar files in your language of choice (Python/Perl/etc.). There is an example of how this works in the sgw3d.rpar file in the McLachlan repository.

  3. Erik Schnetter reporter
    • removed comment

    The disadvantage is that .rpar files require a bit of boilerplate to write a Perl/Python/Bash script that generates a parameter file, and to quote existing dollar signs in the parameter file. In my case, this script would not need to do anything except define a variable (e.g. "rho=@RHO@"), and expand it.

  4. Roland Haas
    • removed comment

    Actually: do we really want this? Turning the parfiles into a small turing incomplete programming language might not be the best idea. Personally I do not find the overhead of rpar files to be onerous. If we introduce a variable (to set several parameters to the same value) then maybe a better solution is to have the C code have functionality to use-that-other-parameter.

    I would try and limit the variables to the top part of the file so that they are not scattered through the file (though that of course differs on whether one likes eg C++ conventions or C conventions).

  5. Erik Schnetter reporter
    • removed comment

    I want to write a parameter file that can run with multiple resolutions. Changing the resolution requires changes in multiple places, including grid size, time step size, and output frequency. Most of these values have a non-trivial dependency on the resolution.

  6. Ian Hinder
    • removed comment

    Another option would be to improve the Cactus parameters themselves so that this became easier. For example, you could specify a time interval between outputs, rather than the resolution-dependent iteration number. The time step could be specified as a fraction of a space step, as a Courant factor. With multipatch this is ambiguous, but there is probably a way around that. Alternatively, there could be a global "resolution scaling factor" which automatically rescaled certain quantities for convergence testing.

    I don't think the overhead of an rpar file is so much. See http://simfactory.org/info/documentation/userguide/use.html#parameter-file-scripts for an example; it's an additional 8 lines, which is very small in comparison to a typical parameter file. It also doesn't change from one parameter file to the next. And is quoting every $ sign by repeating it so onerous? Is there anything other than $parfile which needs to be quoted? And then you get the benefit of a complete programming language if you need it, and the parameter file as Cactus sees it is still easy to parse by other programs.

  7. Erik Schnetter reporter
    • removed comment

    I was looking for a solution without Simfactory. With Simfactory, I just use @RHO@, and set its value when I submit.

  8. Ian Hinder
    • removed comment

    I usually combine the two features. I specify the resolution with a simfactory definition such as @RHO@, which is then assigned to a python variable, and this is then used to compute the resolution-dependent parameters such as grid spacing, output frequency etc. This allows me to use the same parameter file for different resolutions, which wouldn't be possible with the script feature alone.

    To be complete, I would say that a solution without simfactory would be an rpar file which you run before submitting it. An rpar file generates a par file next to the rpar file, which can then be submitted. Yes, the extra step is annoying. I guess that like Roland and Wolfgang, I'm just a bit apprehensive about making the parameter file language too "clever". Does Cactus output a version of the parameter file with all these parameters expanded so you can easily see iteration numbers and grid sizes etc?

  9. Ian Hinder
    • removed comment

    Regarding the $ signs in the proposed solution in the summary. Why are they necessary? I believe that Cactus used to support "global parameters" which would not have a double-colon as they don't have a thorn, but these are not used, and I think the new parser doesn't support them anyway. So an assignment of the form x = y should be unambiguous. The $ signs seem unnecessary (and would have to be quoted if you used this feature from a python rpar script).

  10. Roland Haas
    • removed comment

    Replying to [comment:6 eschnett]:

    I want to write a parameter file that can run with multiple resolutions. Changing the resolution requires changes in multiple places, including grid size, time step size, and output frequency. Most of these values have a non-trivial dependency on the resolution.

    Independent of the topic of the current ticket: Cactus should allow the resolution to be all specifiable via a single parameter (eg ncells_[xyz]), shouldn't it? The timestep should be given by the Courant factor though Llama breaks this (I think I actually do have a patch for this somewhere). Output frequency can be specified by out_dt which does not depend on the iteration number (but may be prone to numerical roundoff issues). Search frequencies are likely fine if given as every_iteration (usually I want: everytime the second-to-finest level executes or at the the same time that the finest level enclosing the NS is updated). Other codes go to great length to provide output at times independent of the time step size and will interpolate the state vector to any desired output time if required which may be overkill (and very hard) for our AMR based infrastructure. If there are parts of the parfile (other than ncellss or dx) that change with resolution then I would try and add functionality to avoid them. Please keep in mind though that this is not really related to this ticket. Variables could also be used eg to produce a series of extraction radii whose size is all tied to the same variable.

  11. Steven R. Brandt
    • removed comment

    Currently there are variables which have them: $parfile, $ENV{name}, and $pi. The $ is useful to ensure there isn't a conflict with a keyword. However, it should be possible to get rid of them.

  12. Steven R. Brandt
    • removed comment

    If there are only a few use cases for these variables, we could just define a thorn called Vars and use Vars::scale as the scaling parameter (instead of $x).

  13. Erik Schnetter reporter
    • removed comment

    Another idea would be to move this functionality into Simfactory. Statements such as

    # @RHO=2@
    

    which are otherwise comments, would set RHO to a default value that can be overridden on Simfactory's command line.

  14. Frank Löffler
    • removed comment

    Simfactory isn't a good option, because even as regular simfactory user I do want to run the same parameter files from time to time by hand.

  15. Roland Haas
    • removed comment

    If we keep the variables directly in the flesh I would keep the '$' prefix. This makes it possible to use variables in strings as well (in particular [hint hint] if we also support ${varname} or $(varname) the way the shell or make do), eg as out_dir = "$var-output" or out_file = "${var}iable.asc".

  16. Bruno Mundim
    • removed comment

    I like the '$' prefix. Indeed it should be extended as Roland pointed out. For example it would be interesting to split output data directory according to data format or data size. I would set then out2D_dir = ${parfile}_2D or ${parfile}_asc. Or maybe we want to read an input file set as input_dir = $ENV{HOME}/path/to/input/file. It would be quite useful. Anyways, just adding a thought to this mix.

  17. Erik Schnetter reporter
    • removed comment

    ${var}text would be one syntax; possibly cleaner would be $var + "text" where the string concatenation is explicit.

  18. Steven R. Brandt
    • removed comment

    Actually, the attached patch already supports this

    $a = "foo"
    $b = "bar"
    TestPar::string1 = $a + $b + "xx"
    
  19. Roland Haas
    • changed status to open
    • removed comment

    No particularly strong objections from me. I tend to not want to turn par files into mini-codes but then, I don't have to use this feature. Someone who has to currently parse the parfiles eg to analyze simulation results may have stronger opinions. Possibly for this we could import parts of Formaline's parameter output into the flesh (either using the current format or something that makes the flesh write a file that could be read as a parfile again).

  20. Ian Hinder
    • removed comment

    I'd rather not apply the patch just because it works. We haven't come to a consensus on whether we want the feature, or the syntax that should be used. There is an obvious road here, which leads to writing our own home-grown programming language to be included in parameter files. Before we start down this road, I think we should consider if this is what we want, and if it is, whether there is a better way to accomplish it. For example, could we use Lua? If we add features like the proposed one, we will be constrained in what we can do in the future to maintain compatibility. I am also concerned about making the parameter files difficult to interpret by the end user: does Cactus provide an "expanded" version anywhere, so the user can see the final result? Cactus doesn't even write out a metadata file for the simulation. If it did, parsing the parameter file would not be necessary during analysis of the data. Maybe Formaline is the answer to this, but this automatically introduces an overhead (archiving of source files) which many people find onerous. I don't want to lose the capability of parsing parameter files to obtain fundamental information about the simulation.

  21. Erik Schnetter reporter
    • removed comment

    Ian, what you say is true. So what do you suggest concretely? Do you want to accept the current patch, propose an alternate syntax, or suggest to people to use Simfactory's replacement feature instead?

  22. Ian Hinder
    • removed comment

    After discussion in the ET call today, we decided that the proposed feature introduces only an incremental increase in the work needed to parse a parameter file, and that if we were to use a richer language with scripting features, it would likely be separate to the current parameter file mechanism. So the patch should be applied.

  23. Ian Hinder
    • removed comment

    Given that existing Cactus parameters are referenced as thorn::name, not $thorn::name, do we still want to have the requirement of introducing a $ for the user's variables? Maybe we should consider this feature as experimental, and allow the possibility to make backward-incompatible changes until the next release.

  24. Roland Haas
    • removed comment

    I had not even realized that parameter values could be used in expressions, is this documented? Requiring '$' may depend on whether we want to blur the distinction between Cactus parameters and the constants defined in the parfile (and $parfile and $ENV{...} which currently exit). I am not sure where I stand on this. Aesthetically I prefer "$" (since I associate it with "scripts"). There seems to be no reason for it though. Syntaxwise it should be fine since we no longer support global Cactus parameters (which don't have the "::" in the name) anymore, I suspect we can live with the ambiguity of ActiveThorns = "a b" not being the assignment of a constant named "ActiveThorns" but instead a directive.

  25. Roland Haas
    • removed comment

    Replying to [comment:28 rhaas]:

    I had not even realized that parameter values could be used in expressions, is this documented? Nevermind. It apparently is documented.

  26. Ian Hinder
    • removed comment

    Yes, ActiveThorns would be a reserved word in the language. The only objective reason for the $ that I can see in the above comments is that it allows other reserved words (e.g. sqrt) to be used as constant names (we shouldn't call them variables, they cannot be changed). String concatenation can apparently be handled as var1 + "string" + var2 so we don't need "${var1}string${var2}". Is there another reason for the $s, and do we really want to support users using reserved words as constant names?

  27. Roland Haas
    • removed comment

    There seems to be no remaining reason for keeping "$". Since the only reserved word that may conflict is ActiveThorns, I think we can afford now allowing its use. Regular parameter names all have the form "THORN::PARAMETER" and ":" is not allowed in constant names. With the current implementation the constants are actually changable, ie this

    ActiveThorns = IOUtil
    
    $const = 42
    
    Cactus::cctk_final_time = $const
    
    $const = $const / 2
    
    Cactus::cctk_initial_time = $const
    
    IO::parfile_write = "generate"
    IO::out_dir = "$parfile"
    

    works as expected if read as a script (ie the generated parfile contains 42 and 21 for cctk_final_time and cctk_initial_time). If we add a "computed goto" statement we are probably Turing complete in our parfiles :-P

  28. Roland Haas
    • removed comment

    Replying to [comment:31 rhaas]:

    Since the only reserved word that may conflict is ActiveThorns, I think we can afford now allowing its use. This was supposed to be "not allowing its use"

  29. Ian Hinder
    • removed comment

    I assumed that the built-in mathematical functions which Cactus supports in parameter file expressions, such as sqrt, would also be reserved words. It's possible that we have a different namespace for functions and constants, but I don't think that is a good idea. I also think that parameter files should not allow the same parameter or constant to be defined more than once, and that the ordering of statements in a parameter file should not matter. i.e. the above fragment should give an error about a multiply-defined constant.

  30. Steven R. Brandt
    • removed comment

    I don't think all of these assumptions are true. :)

    In particular, order does matter (at the moment) and variables can be reassigned. I can change this, of course.

  31. Log in to comment