- removed comment
Filter rules error when using rsync 2.6.9
Issue #461
closed
When I use rsync 2.6.9 (the default for Mac OS 10.6) on my local machine, I get the error
invalid modifier sequence at 'p' in filter rule: -p _darcs
when running sim sync. Probably this is an rsync 3 feature.
Keyword:
Comments (19)
-
-
- removed comment
Replying to [comment:1 knarf]:
Could you try updating rsync? rsync 3.0.0 was released more than three years ago and 2.6.9 in 2006...
The 'p' rule was added in 3.0.0, so updating to that version will fix the problem. Unfortunately 2.6.9 is included by default with the latest version of OSX so many people will likely encounter this.
Having said that - should we add a check for the version of rsync?
I agree that adding a check for the rsync version is a good idea.
-
- removed comment
Updating is not a general solution -- we have to work with whatever is available on the given systems, and many don't have rsync 3 yet. We will either have to disable this altogether, or test the rsync version and use a sensible fallback if rsync is too old.
Instead of explicitly testing rsync's version, we can just call rsync, and if it fails, call it again with a reduced set of options.
-
- removed comment
I agree that we could fall back in the case of the -p option. However I am also a bit hesitant when it comes to supporting software which is that old. Is this only affecting Macs, and is the -p option the only one affected? -p preserves the permissions - is the long option available in 2.6.9: --perms? If not we could also just not add this option for an old version of rsync, but warn about it.
-
- removed comment
Many HPC systems run older versions of rsync. Many also run Python 2.3. We need to support these. rsync 2.6.9 is a very common version. Mind that omitting the "-p" option etc. works fine -- the only drawback is that we see ten or twenty warning messages when we sync. That is a small price to pay for supporting these systems.
Remember also IBM AIX systems, where the patch command is from 1988... we're dealing with these systems as well.
-
- removed comment
I've attached a patch which only uses the p filter rule if we are using at least rsync 3.0.0. It prints a warning for older versions of rsync.
-
- removed comment
The patch seems fine, please apply. (I assume it's been tested.)
I'm slightly worried that checking the version number may not always be correct, if e.g. the feature has been introduced only in rsync 3.0.1. But I guess we'll see it if this is the case.
Does this only depend on the local rsync version, or also on the remote version? We have no control over the remote version.
-
- removed comment
Replying to [comment:7 eschnett]:
The patch seems fine, please apply. (I assume it's been tested.)
Yes, I tested with both 3.0.7 and 2.6.9 on my Mac, syncing to Damiana.
I'm slightly worried that checking the version number may not always be correct, if e.g. the feature has been introduced only in rsync 3.0.1. But I guess we'll see it if this is the case.
The particular feature we're worried about was added in 3.0.0 according to the rsync changelog. Patch committed.
Does this only depend on the local rsync version, or also on the remote version? We have no control over the remote version.
Unfortunately, yes. Both the remote and local versions of rsync need to be up-to-date. If the version on the remote machine is too old it fails with the error:
"filter rules are too modern for remote rsync."
We seem to mostly provide our own rsync on remote machines, though, don't we?
-
- removed comment
We provide an rsync on some machines, but not on all. A new rsync isn't a requirement.
That means that the current patch is incomplete. We either need to "ssh+rsync --version" to check the remote version as well, or call rsync and look for the error message.
-
reporter
- removed comment
ssh connections to HPC systems can be incredibly slow - maybe up to 10 seconds for something like Kraken. We shouldn't make more of these than are necessary. One option would be to use the ControlMaster feature if we wanted to do this. This would add yet another thing which could go wrong, but it would actually be quite nice because it means you could sync and submit in the same ssh connection.
-
- removed comment
Yes, ssh can be very slow. Hence my suggestion to first try to sync with the full feature set, without checking anything. If this fails, we analyse the return code or error message, and/or try with a reduced feature set. This is as fast as is possible on Kraken.
We could even skip the initial "ssh mkdir" unless rsync fails, since this won't be necessary in general.
-
- removed comment
What about adding an optional entry to the machine database for the rsync version. If it is specified and sufficiently recent, then we use the 'new' rules. If it is too old or not specified, we default to the 'old' rules. Anyone who uses a machine and is sufficiently annoyed by the .pyc files that the old rules sees will then be motivated to update that machine's entry.
-
- removed comment
I prefer MDB entries that are as small as possible. Things that can be done automatically should be done automatically. Only things that are impossible to detect at run time should be in the MDB.
I assume that rsync returns an error code. It should be easy to construct a command such as
rsync-with-permissions || rsync-without-permissions
which would do the "right thing" automatically.
Or maybe even
rsync-with-permissions || rsync-without-permissions || { ssh mkdir && { rsync-with-permissions || rsync-without-permissions } }
-
reporter
- removed comment
This is particularly serious as lonestar, the newest teragrid machine, has rsync 2.6.8. This means that we cannot currently sync to lonestar. I get the error:
filter rules are too modern for remote rsync.
You can work around this temporarily by using a version of rsync 3 that I just compiled by adding the following to your defs.local.ini:
[lonestar] rsynccmd = /home1/00915/hinder/software/rsync-3.0.8/bin/rsync
-
- removed comment
I've attached a patch which works around the problem by first trying the new rules and then falling back to the older ones if the first attempt fails with a protocol error. This is essentially implementing Erik's suggestion. OK to apply?
-
- removed comment
The patch is good to apply if it has been tested. (I have not tested it myself.) Essentially, we need to make sure that (a) it still uses the new features if available, and (b) it correctly falls back to the old features if they are not.
I would also reduce the warning to an information, since there are no correctness problems with an old rsync, and in many cases people won't care that much about sync performance. (And those who do will read the output, or should be punished for not doing so.)
-
- removed comment
Replying to [comment:16 eschnett]:
The patch is good to apply if it has been tested. (I have not tested it myself.) Essentially, we need to make sure that (a) it still uses the new features if available, and (b) it correctly falls back to the old features if they are not.
I tested it under 4 conditions:
- rsync 3.0.7 on my laptop, 2.6.8 on Kraken
- rsync 3.0.7 on my laptop, 3.0.3 on Kraken
- rsync 2.6.9 on my laptop, 2.6.7 on Kraken
- rsync 2.6.9 on my laptop, 3.0.3 on Kraken and it does the correct thing in all four cases. I will commit the patch now.
-
- changed status to resolved
- removed comment
-
- edited description
- changed status to closed
- Log in to comment
Could you try updating rsync? rsync 3.0.0 was released more than three years ago and 2.6.9 in 2006...
Having said that - should we add a check for the version of rsync?