svn+ssh problems

[eliben]

Hello,

I've been running SVN through SSH for months on my Bluehost domain, but last week it just spontaneously stopped working. Investigating the cause, I realized that it doesn't even work if I try to login through SSH from the server itself, so the client/TortoiseSVN isn't to blame.

I've discovered some strange things. The error is:


bash: svnserve: command not found

Even if I just ask:


ssh user@server "which svnserve"


I get:


which: no svnserve in (/usr/bin:/bin)

Although I've specifically set up PATH in my .bashrc to point to $HOME/bin where svnserve is installed. I've also added the path to ~/.ssh/environment without luck.

Curiously when running:


ssh user@server "echo $PATH"

It does print my updated path.

Any idea what might be going on here?

[elfoak]

I am having the exact same problem.

If I login to the server with "ssh username@domian" then everything is fine.
but ssh username@domian "witch svnserve" doesn't seem to be able to pick up the $PATH I set up in bash profile.

[happycactus]

Hi,

I am also facing the same problem, and I am stucked.
I think I have found the problem, since the "svnserve missing" message can be due to the fact that while in a non-interactive ssh session the bashrc/bashprofile is not read and so the svnserve path is not appended.

it *could* be caused by a reconfiguration of sshd, if they have disabled the "PermitUserEnvironment" option in sshd_config (but I could not check since sshd_config is out of sight from the virtual session).

I have opened a ticket - maybe they'll respond - I'll post here the result as soon as I'll have a response.

Keep in touch...
Regards

HappyCactus

[eliben]

happycactus

Yes, I came to the same conclusion re their configuration of sshd, and also opened a ticket to which, alas, there's still no reply. Bluehost are really starting to piss me off.

So far I'm handling it by using commands in my SSH keys (
~/.ssh/authorized_keys), as explained here: http://serverfault.com/questions/80650/ssh-svnserve-t-command-while-still-allowing-shell-access

[happycactus]

The solution you posted can solve this issue, but prevents you to access ssh interactively with key authentication unless you use different keys for interactive logins and subversion logins.
This is not good if you need to do both things (interactive and svn logins) from the same machine.
Of course, the solution is very simple for the administration side (either restore the original sshd configuration or append ˜/bin to the default path), but what is doing bluehost support now? I am thinking about close the account and find another provider (and asking a refund, of course).

[eliben]

The solution you posted can solve this issue, but prevents you to access ssh interactively with key authentication unless you use different keys for interactive logins and subversion logins.
This is not good if you need to do both things (interactive and svn logins) from the same machine.


It's not really a problem - I use two different keys, and configure Putty to use the one without svnserve for the shell, and TortoiseSVN to use the one with svnserve, and it works OK.

[happycactus]

It's not really a problem - I use two different keys, and configure Putty to use the one without svnserve for the shell, and TortoiseSVN to use the one with svnserve, and it works OK.

In my humble opinion, it is not a solution. I can't manage 2*x keys (two for each installation I have, linux, windows, macos...)... it is just a workaround.
The solution should be:

1) restore the original sshd_configuration, or
2) put ˜/bin into the default path

I have received a reply from bluehost customer cares - pardon, customer's care - and they addressed me to that solution, so they are reading this thread... I hope that they'll find the solution soon, I am a bit disappointed for this issue... I just bought a subversion hosting, and I am migrating the sensitive projects there... I wouldn't want to think that they're indifferent...

[eliben]

In my humble opinion, it is not a solution. I can't manage 2*x keys (two for each installation I have, linux, windows, macos...)... it is just a workaround.
The solution should be:

1) restore the original sshd_configuration, or
2) put ˜/bin into the default path

I have received a reply from bluehost customer cares - pardon, customer's care - and they addressed me to that solution, so they are reading this thread... I hope that they'll find the solution soon, I am a bit disappointed for this issue... I just bought a subversion hosting, and I am migrating the sensitive projects there... I wouldn't want to think that they're indifferent...

I agree that this isn't an optimal solution. And I've just received a completely irrelevant answer from Bluehost's support for my ticket :(

[TonyS]

According to the scripting team: customers using programs that connect to the server over SSH to run services that they installed in their local account are currently failing. The reason is that with the recent openssh upgrade the .bashrc is no longer run for non-interactive sessions. There is no workaround at this time, but we hope to have it resolved soon. The issue will most frequently present as “Command not found”-type errors when trying to use tools like git or svn remotely.

[eliben]

According to the scripting team: customers using programs that connect to the server over SSH to run services that they installed in their local account are currently failing. The reason is that with the recent openssh upgrade the .bashrc is no longer run for non-interactive sessions. There is no workaround at this time, but we hope to have it resolved soon. The issue will most frequently present as “Command not found”-type errors when trying to use tools like git or svn remotely.

TonyS: thanks for this information. I hope the issue will be resolved soon.

[Streamlet]

I received a reply from Jeff, which says:

At this point, it is difficult to say for sure that it will or will not be
fixed. The problem seems to stem from a difference in the way the shell is
called by openssh 4 (what we had before) and openssh 5. We had to make the
upgrade to openssh 5 for security reasons, and won't be able to roll back to
the old openssh. At the moment, the best advice we can give is to attempt to
use workarounds that are specific to each version control system that you want
to use. Each workaround involves specifying an absolute path to the program
that needs to be executed on the remote end. For example, with git, the
following fails with "bash: git-upload-pack: command not found":

git clone example.com:test.git

But the following would work (assuming the path is in fact where your own
git-upload-pack exists):

git clone --upload-pack=/home/examplec/.local/bin/git-upload-pack example.com:test.git

It's definitely not an ideal solution, but it's the only type of workaround we
have at the moment.

Another workaround is to use one ssh key for each type of version control
system you have, and use the "command" option in your ~/.ssh/authorized_keys
file.

We are still hopeful that we'll be able to find a way to get openssh5 to launch
your shell in such a way that your .bashrc file is honored the same way that
openssh4 did. A more likely outcome will be the enabling of the
PermitUserEnvironment, so the PATH variable can be hard coded in
.ssh/environment.


Thank you,

Jeff

[eliben]

I've finally received a good response from BH, here it is:



A recent upgrade of openssh from 4 to 5 has changed the behavior of the way commands are called. This has affected anyone who has custom software installed into their own home directory, and call the commands via ssh directly, rather than entering a shell session first.

The first reaction to a problem like this is to suggest that the upgrade be rolled back. Unfortunately, this change was implemented due to a security issue in the older version. Rolling back would expose our servers and invite attackers to exploit the older version of OpenSSH. At this time, we are still researching ways to make this new unwanted behavior go away, but still retain the newer version of OpenSSH.

We only have a set of workarounds for the most commonly used programs that are affected. In the future, we hope to have a workaround that will work for every program, or maybe even completely revert the behavior.

There are two categories of workarounds. The first one is forcing the full path to the command needed in the program you are using. The second method is forcing the full path to the command to be executed by using ssh keys.


Solution 1

To get your custom path to the svnserve binary to be used, you need to create a wrapper script on your local machine:

#!/bin/sh
ssh $1 /path/to/svnserve -t

You can find the correct /path/to/svnserve by logging into your Bluehost account via SSH and typing 'which svnserve' . Name this script anything you want, such as "svnssh", make it executable, and place it in a convenient location. Now, in your .subversion/config file, place the following:

[tunnels]
ssh=/path/to/svnssh

The /path/to/svnssh should match the location that you put your svnssh wrapper script.


Solution 2

Another solution for svn is to create a public/private ssh keypair that will be used only for svn. When you add the public key to your .ssh/authorized_keys file, put command="/path/to/svnserve -t", at the beginning of the line, before the key itself.

Any time you connect using that corresponding private key, you will be forced to use the snvserve command, so make sure you don't use that key always. One way to use that key only when you're using svn is to rename the private key file to something other than default. For example, rename it from ~/.ssh/id_rsa to ~/.ssh/id_svn_rsa. To get svn to use this key, put the following in your .subversion/config file:

[tunnels]
ssh=ssh -i ~/.ssh/id_svn_rsa

[masterm1nd]

I have been searching for the fix for this. .bashrc is not loaded.

I found a fix for git error:

$ git push web
bash: git-receive-pack: command not found
fatal: The remote end hung up unexpectedly

The solution is to

Here's a temporary fix for those using git

$ git pull --upload-pack /path/to/git/git-upload-pack remote_repo_name

$ git push --receive-pack /path/to/git/git-receive-pack remote_repo_name

you can find the path to git-upload-pack & git-receive-pack by

$ which git-receive-pack

AND/OR

$ which git-upload-pack

/path/to/git/ is the path on the REMOTE server not the one which pulls or pushes!

Do not forget that.
It's a temporary fix, but at least a fix otherwise you cannot work around that.

[140fulton]

I'm trying to use the shell wrapper workaround with a Tortoise SVN client under XP. I've created the wrapper using a cygwin bash script, but now repro am seeing:
'Cannot create tunnel %1 is not a valid Win32 executable'

It's right, of course. The script is, well, a script (albeit executable). The script appears to work from the command line.

Is there any hope?

Jack

[TheFlashGuy]

Can anyone confirm that they have got Solution 1 or 2 working?
I've been trying for hours, but I still get the same 'bash: svnserve: command not found'

[zetlan]

So the root cause of the problem is that the ssh login that Capistrano uses does not run a login or interactive shell, and only a limited environment is set. In your Capistrano recipe (deploy.rb), you can define:

default_run_options[:shell] = "/bin/bash -l"

which will cause all commands to run as if they were in a login shell, which will invoke /etc/profile, ~/.bash_profile, ~/.bash_login, and then ~/.profile (files that don't exist are skipped). Simply put your environment and other commands in .bash_profile, or have .bash_profile source .bashrc, and you should be good to go.