1. Einstein Toolkit Avatar Einstein Toolkit
  2. Einstein Toolkit
  3. tickets
  4. Issues

OSX does not install trusted certificates, breaking GetComponents (svn)

Issue #1801 closed
Roland Haas created an issue

I just tried to check out the ET Hilbert release on my OSX laptop (OSX 10.10.4, subversion version 1.7.19) and get for the LSUThorns: --8<-- Could not checkout module LSUThorns/SummationByParts svn: E175002: Unable to connect to a repository at URL 'https://svn.cct.lsu.edu/repos/numrel/LSUThorns/SummationByParts/branches/ET_2015_05' svn: E175002: OPTIONS of 'https://svn.cct.lsu.edu/repos/numrel/LSUThorns/SummationByParts/branches/ET_2015_05': Server certificate verification failed: issuer is not trusted (https://svn.cct.lsu.edu) --8<-- https://www.sslshopper.com/ssl-checker.html#hostname=https://svn.cct.lsu.edu reports the certificate to be ok though (with the exception of it using SHA1 as a hash).

Doing a manual svn checkout https://svn.cct.lsu.edu/repos/numrel/LSUThorns/QuasiLocalMeasures/branches/ET_2015_05 asks me whether I would want to trust the certificate.

  1. I thought the LSU certificates were trusted by common OS by default by now (and OSX is certainly common)
  2. I thought we had special code in GetComponents to make it automatically not try and verify signatures because of this

  3. this is really becoming a nuisance (if indeed caused by an uncommon certificate issuer and not by something odd on my laptop) :-)

    openssl ssl_client -connect svn.cct.lsu.edu:443 </dev/null >ssl.out reports a self-signed certficate though this may well be the untrusted certificate.

Keyword:

Comments (41)

  1. Roland Haas reporter
    • removed comment

    Well this seems to be something either local to my mac or at least local to OSX. I can check out from my Linux box and the svn client also complains about the aei svn server (which tends to not happen normally).

  2. Erik Schnetter
    • removed comment

    I think we should move the LSUThorns thorns to Bitbucket: - LSUThorns/PeriodicCarpet - LSUThorns/QuasiLocalMeasures - LSUThorns/SummationByParts - LSUThorns/Vectors

  3. Roland Haas reporter
    • removed comment

    Suggestions:

    • PeriodicCarpet into Carpet
    • QuasiLocalMeasures into EinsteinAnalysis
    • SummationByParts into CactusNumerical (needs license to be LGPL) or Numerical if code does not adhere to the stricter Cactus rules (ie it really needs documenation for that)
    • Vectors into CactusUtils or Utils (same as SummationByParts)
  5. Roland Haas reporter
    • removed comment

    The suggestion of setting up a cert.pem in /System/Library/OpenSSL seems to no longer work. OpenSSL still complains and running it through dtruss (strace equivalent) reveals it is not even trying to open any cert.pem file.

  8. Frank Löffler
    • removed comment

    Replying to [comment:7 eschnett]:

    This is also a problem on my Raspberry PI (with all packages up to date).

    I can confirm that, using an up2date Raspbian. This is based on the old Debian wheezy, which points to the same problem: the ca-certificates are from Jan 2013. In theory this should be enough, since the root certificate was issued in 2010, but apparently it wasn't included in Raspbian until after Jan 2013. I would expect that to go away once Raspbian uses the Jessie release.

  9. Frank Löffler
    • removed comment

    I propose to close this ticket. The "offending" root cert is by now 5 years old. It shouldn't take that long to propagate to clients - 5 years is already a long time for a certificate, and 1/4 of its lifetime. Imho, this is an issue with old clients. We cannot do without certificates for write access, and using certificates requires to have your trusted certs somewhat updated.

  10. Roland Haas reporter
    • removed comment

    I would think that even if the offending client is old (and I think we all believe that it is) we still need to support it if the old client is very prevalent. The raspberry is probably not something that is common enough, however OSX would be I think. However for OSX since its OpenSSL seems to lack any root certificates it would seem to me that all attempts to use https transport with openssl (and thus svn) must fail. Is this correct?

  11. Erik Schnetter
    • removed comment

    The issue with certificates has been around for years, has not been solved yet, and continues to annoy people. What you probably mean (Frank) is that there is no way to solve this with our current repo infrastructure (hosting SVN repos at LSU). That doesn't mean that there isn't another way, or that the issue isn't annoying to people any more.

    We should address this; if we can't get certificates to work, and if the "solution" is to ignore them, then we should use a different mechanism and forego svn via https. For example plain http should work, or svn+ssh, or switching to git.

  12. Frank Löffler
    • removed comment

    Plain http wouldn't work for write access. We could use it for read-only, but then have the problem with people trying to use it to commit again. svn+ssh wouldn't work in this case (no ssh access), but also kind of circumvents the problem, as there is no trusted host-key. This would be similar to telling svn to trust all certificates (bad idea, but also a possibility - could be done for problematic certs only too). git isn't solving this at all. git also uses either ssh or https as transport protocol - both with the same issues as already discussed. This isn't a svn problem.

    The root problem is clients not updating trusted certs, and this isn't something we could fix as ET developers. I trust that all OSX users are loudly complaining to Apple? In this case (afaik), no certificate would work. Isn't there a version of subversion/openssl available for OSX that isn't broken (not shipping with trusted root certs counts for me as "broken")? We could then require (and test for) this.

  13. Erik Schnetter
    • removed comment

    Bitbucket and Github solved the problem. We can either do something similar, or switch to them (or another service provider).

  14. Frank Löffler
    • removed comment

    How did they solve this problem? I can only imagine that the git client shipped with OSX is not as broken as the svn client, in which case bitbucket/github didn't solve anything. What does the OSX client say to

    svn co https://github.com/ianhinder/Kranc

    Really - if Apple svn/openssl is broken on OSX, our reaction shouldn't be to abandon svn or https. What do we switch to if some OS breaks git next? We have to get svn working instead. Either users complain until it is fixed by the vendor (you do pay, don't you?), or they use a working version of the client.

    So, again my question: Is there a working version of subversion/openssl available for OSX? I don't have such a machine and cannot try.

  15. Erik Schnetter
    • removed comment

    [Long answer deleted.]

    I don't care about the details, but I expect downloading the code to work out of the box. I don't think we should close a ticket when there is no good resolution.

  16. Frank Löffler
    • removed comment

    I also expect the svn client to work out of the box. Apparently it doesn't. Is there working version available via a standard mechanism - homebrew maybe? Let's not close this ticket then, but let's find a solution on OSX. But for that I need someone to test and try. There seem to be plenty available: https://subversion.apache.org/packages.html#osx . If one of these works and is reasonably easy to install, all we need to do is blacklist the broken version.

  17. Roland Haas reporter
    • removed comment

    Replying to [comment:14 knarf]:

    How did they solve this problem? I can only imagine that the git client shipped with OSX is not as broken as the svn client, in which case bitbucket/github didn't solve anything. What does the OSX client say to svn co https://github.com/ianhinder/Kranc It says:

    queech4u:tmp rhaas$ /usr/bin/svn co https://github.com/ianhinder/Kranc kranc
Error validating server certificate for 'https://github.com:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: github.com
 - Valid: from Apr  8 00:00:00 2014 GMT until Apr 12 12:00:00 2016 GMT
 - Issuer: www.digicert.com, DigiCert Inc, (null), (null), US ((null))
 - Fingerprint: A0:C4:A7:46:00:ED:A7:2D:C0:BE:CB:9A:8C:B6:07:CA:58:EE:74:5E
(R)eject, accept (t)emporarily or accept (p)ermanently? ^Csvn: E200015: Unable to connect to a repository at URL 'https://github.com/ianhinder/Kranc'

    so svn also fails for bitbucket's svn transport (the svn that works for me is the homebrew one '''after''' I manually install a cert.pem file eg from curl).

    I think that for downloads we can live with http transports and the number of persons who commit is small enough (and expert enough) that we can ask those to use a fully fledged svn and openssl client.

    I also think that we must make every effort so that downloads work. Basically no matter how annoying this is for us.

  18. Frank Löffler
    • removed comment

    Ok, we could let GetComponents switch https to http if this client is detected. We might be able to see this from 'svn --version', but really svn isn't to blame here, but an incomplete openssl installation. 'openssl version' might give us a clue, but again - we really would have to check if svn uses an openssl without certificates. This seems to be quite complicated. Maybe we can simply blacklist the default OSX svn client (regardless of openssl configuration), and in that case switch to http?

  19. Roland Haas reporter
    • removed comment

    One more data point: I can get the default client to work by setting the SSL_CERT_FILE environment variable to the certificate bundle from curl. Unfortunately that file does not exist on a freshly installed mac it seems. However one can extract the keys out of OSX's key chain using:

    security export -k /System/Library/Keychains/SystemRootCertificates.keychain -t certs -f pemseq p

    so we '''could''' (when we detect we are running on OSX and that certificates are failing, eg by checking that svn info --non-interactive fails but svn info --non-interactive --trust-server-cert succeeds so that we don't have to rely on some English language string parsing) use the "security" command to write the certificates into a file (either in /tmp or in $HOME/.crl) and then set SSL_CERT_FILE before we call svn.

  20. Ian Hinder
    • removed comment

    You cannot compile the ET on a freshly installed OS X system without a lot of extra packages, usually provided by MacPorts or Homebrew. The subversion (/usr/bin/svn) that you are trying to use is installed by xcode (which is needed for compiling on OS X), and for some reason seems to be broken regarding SSL. We could blacklist this version and insist that users install a good subversion. The version in MacPorts has always worked for me. I have just tested the MacPorts and Homebrew versions on github, and they both work, whereas the xcode version does not. Since we only support using OS X with MacPorts or Homebrew (and people compiling everything from scratch are well-capable of building their own SVN), I suggest we just tell people to install svn from whichever package manager (macports or homebrew) they are using. This would be done when GetComponents detects that the svn version is the apple version.

    I would rather not add complicated fallback logic into GetComponents to try to work with buggy software, when it is a small extra step for users to add one more package to the list of recommended packages that they need to install from their package manager.

  21. Roland Haas reporter
    • removed comment

    If this failure happened anywhere other than the initial download I would agree that we can ask users to update their system. However since this is the very first thing a new user will do (in particular following the Download link on our website) this really should work, no matter what. Even with the old svn client (I think). If the solution of writing a "useful" set of certificates using OSX's built-in certificate store is not acceptable, then I think that at the very least we must add code to GetComponents that detects that svn is failing due to missing certificates and if it also finds that it runs on OSX it must output a warning explaining what is happening and how to fix it (by suggesting to install homebrew or macports). This warning must be collected with all the other warnings and output at the end of the GetComponents output. I really believe that if the initial download does not work, a new potential user will just give up and look somewhere else (unless they are in a group already using the ET in which case they have local support to sort this out).

  23. Zach Etienne
    • removed comment

    This is not just an OSX problem.

    I witnessed the same error message on RHEL6 (Stampede), preventing me from compiling. In the end, I just grabbed the tarball and added the LSUThorns package manually. Given how widespread RHEL6 is, increased priority is justified.

    I do not see this behavior on Ubuntu 14.04.3.

  26. Frank Löffler
    • removed comment

    This seems to show the message if on OSX and certificate validation fails. Isn't that a little broad? It could fail for a valid reason.

  27. Roland Haas reporter
    • removed comment

    True. Right now it shows the message about the stock svn client if svn info first fails (line 584) and then succeeds if one adds --trust-server-cert (line 587). The man page says "--tryst-server-cert}} that this tell svn to "accept SSL server certificates from unknown certificate authorities" so other causes of failure to verify (expired, faulty) should still trigger. If the failure is due to something other than the certificates then this would have to have changed in between the two svn calls for this check to become invalid. It is of course possible that the validation fails for another certificate related reason and --trust-server-cert accepts this. On could make the regex more explicit since svn actually outputs: Server certificate verification failed: issuer is not trusted. Would you think this to be better, it would at least avoid mis-classifying expired certificates?

    I am a little bit unhappy to look for an English language text to determine some success state but the text seems to stay the same even if I eg set LANG to "DE" to try and see if I can get German language error messages. A bigger concern may be subversion updating the error text in the future.

  28. Frank Löffler
    • removed comment

    It turns out that we probably need a similar fix also for clients that happen to not have updated root certs. Thus, I propose to make this change a little more general: in addition to disabling cert-checking and/or explicitly specifying the ca cert when this fails on OSX, do this on all systems where this fails - if the problem occurs with servers we know can fail. We can either hardcode these servers in GetComponents, or add something to the thornlist.

  29. Roland Haas reporter
    • removed comment

    Has there been any progress an a more comprehensive fix? If not I think we should include the warning and workaround in the release. I have updated the pull request to test not only on OSX but on all OS.

  30. Roland Haas reporter
    • removed comment

    Anything? Given that we are about to release I assume this is now working on OSX? Conceivably using subversion from macports/homebrew? Should we include a warning to this effect in the release notes?

  31. Frank Löffler
    • removed comment

    If clients are incorrectly installed, we have not really an option there. Disabling certs if their checking fails amounts to disabling certificates. At this point, I would opt for mentioning this in the release notes, however, I would need some current information about this from actual users.

  32. Barry Wardell
    • removed comment

    It still fails with the OS X system version of svn, but it works fine with the homebrew version installed.

  33. Frank Löffler
    • removed comment

    Thanks. I'll mention that Apple screwed this up, and you need homebrew for a working svn client.

  34. Roland Haas reporter
    • removed comment

    Just to repeat what I said 8 months ago (https://trac.einsteintoolkit.org/ticket/1801#comment:29) : --8<-- Has there been any progress an a more comprehensive fix? If not I think we should include the warning and workaround in the release. I have updated the pull request to test not only on OSX but on all OS. --8<--

    Given that there is no more comprehensive solution yet I would apply this patch which will at least give users the option of manually accepting all those certificates.

  35. Frank Löffler
    • removed comment

    It would be a step in the right direction. It wouldn't cover all cases (would be OSX only), but that's probably ok.

    Out of curiosity: did any of the OSX users file a bug report with Apple - since this is quite obviously a bug on their system?

  36. Roland Haas reporter
    • removed comment

    Does this mean "please apply"?

    Note that this is no longer specific to OSX --8<-- I have updated the pull request to test not only on OSX but on all OS. --8<-- so all OS display the warning (the one for OSX is just more verbose) and all OS will then allow users to manually accept the unknown certificates.

    I have not filed a bug nor would I expect this to help I lot I admit. I do not even know how to file bugs for OSX.

  39. Roland Haas reporter
    • removed comment

    Hah, seems as if Apple fixed this issue in at least the latest OSX release. Even after wiping $HOME/.subversion (and uninstalling subversion from homebrew) and on a fairly newly installed OSX laptop I do not get any failures for

    /usr/bin/svn info http://svn.cactuscode.org/projects/ExternalLibraries/hwloc/

    This also seems to happen to others. I do not know however in which OSX release this was fixed.

    Question now: do we want to keep the workaround with the always present danger of it introducing new unknown bugs (rather than old known ones)?

    The test still triggers though in case things are actually wrong, eg if one uses ssh to forward port 443 from ones laptop to svn.cactuscode.org then

    /usr/bin/svn info http://localhost/projects/ExternalLibraries/hwloc/

    fails due to a mismatch in the certificates (cn-mismatch).

  42. Log in to comment
Assignee
–
Type
bug
Priority
minor
Status
closed
Component
GetComponents
Milestone
–
Version
development version
Votes
0
Watchers
0
Jira Software: the preferred issue tracker for Bitbucket. Join the team!