## 51.7. Server and Client Problems

Q: How do I get sshd to recognize a new configuration file?

A: You can terminate and restart sshd, but there's quicker way: send the "hangup" signal (SIGHUP) to sshd with kill -HUP.

Q: I changed the sshd config file and sent SIGHUP to the server. But it didn't seem to make any difference.

A: sshd may have been invoked with a command-line option that overrides that keyword. Command-line options remain in force and take precedence over configuration file keywords. Try terminating and restarting sshd.

Q: A feature of ssh or scp isn't working, but I'm sure I'm using it correctly.

A: The feature might have been disabled by a system administrator, either when the SSH software was compiled (Chapter 4) or during serverwide configuration (Chapter 5). Compile-time flags cannot be checked easily, but serverwide configurations are found in the files /etc/sshd_config (SSH1, OpenSSH) or /etc/ssh2/sshd2_config (SSH2). Ask your system administrator for assistance.

Q: ssh or scp is behaving unexpectedly, using features I didn't request.

A: The program might be responding to keywords specified in your client configuration file (Section 7.1). Remember that multiple sections of the config file apply if multiple `Host` lines match the remote machine name you specified on the command line.

Q: My SSH1 .ssh/config file doesn't seem to work right.

A: Remember that after the first use of a "Host" directive in the config file, all statements are inside some `Host` block, because a `Host` block is only terminated by the start of another `Host` block. The ssh1 manpage suggests that you put defaults at the end of the config file, which is correct; when looking up a directive in the config file, ssh1 uses the first match it finds, so defaults should go after any `Host` blocks. But don't let your own indentation or whitespace fool you. The end of your file might look like:

>     # last Host block
>     Host server.example.com
>      User linda
>     # defaults
>     User smith

You intend that the username for logging into server.example.com is "linda", and the default username for hosts not explicitly listed earlier is "smith". However, the line "User smith" is still inside the "Host server.example.com" block. And since there's an earlier `User` statement for server.example.com, "User smith" doesn't ever match anything, and ssh appears to ignore it. The right thing to do is this:

>     # last Host block
>     Host server.example.com
>      User linda
>     # defaults
>     Host *
>      User smith

Q: My .ssh2/ssh2_config file doesn't seem to work right.

A: See our answer to the previous question for SSH1. However, SSH2 has the opposite precedence rule: if multiple configurations match your target, the last, not the first, prevails. Therefore your defaults go at the beginning of the file.

Q: I want to suspend ssh with the escape sequence, but I am running more than two levels of ssh (machine to machine to machine). How do I suspend an intermediate ssh?

A: One method is to start each ssh with a different escape character; otherwise, the earliest ssh client in the chain interprets the escape character and suspends.

Or you can be clever. Remember that if you type the escape character twice, that's the meta-escape: it allows you to send the escape character itself, circumventing its usual special function. So, if you have several chained ssh sessions, all using the default escape character ~, you can suspend the nth one by pressing the Return key, then n tildes, then Control-Z.

Q: I ran an ssh command in the background on the command line, and it suspended itself, not running unless I "fg" it.

A: Use the -n command-line option, which instructs ssh not to read from stdin (actually, it reopens stdin on /dev/null instead of your terminal). Otherwise, the shell's job-control facility suspends the program if it reads from stdin while in the background.

Q: ssh prints "Compression level must be from 1 (fast) to 9 (slow, best)" and exits.

A: Your `CompressionLevel` is set to an illegal value for this host, probably in your ~/.ssh/config file. It must be an integer between 1 and 9, inclusive.

Q: ssh prints "rsh not available" and exits.

A: Your SSH connection attempt failed, and your client was configured to fall back to an rsh connection. However, the server was compiled without rsh fallback support or with an invalid path to the rsh executable.

If you didn't expect your SSH connection to fail, run the client in debug mode and look for the reason. Otherwise, the SSH server is just not set up to receive rsh connections.

Q: ssh1 prints "Too many identity files specified (max 100)" and exits.

A: SSH1 has a hardcoded limit of 100 identity files (private key files) per session. Either you ran an ssh1 command line with over 100 -i options, or your configuration file ~/.ssh/config has an entry with over 100 `IdentityFile` keywords. You should never see this message unless your SSH command lines and/or configuration files are being generated automatically by another application, and something in that application has run amok. (Or else you're doing something really funky.)

Q: ssh1 prints "Cannot fork into background without a command to execute" and exits.

A: You used the -f flag of ssh1, didn't you? This tells the client to put itself into the background as soon as authentication completes, and then execute whatever remote command you requested. But, you didn't provide a remote command. You typed something like:

>     # This is wrong
>     $ ssh1 -f server.example.com

The -f flag makes sense only when you give ssh1 a command to run after it goes into the background:

>     $ ssh1 -f server.example.com /bin/who

If you just want the SSH session for port-forwarding purposes, you may not want to give a command. You have to give one anyway; the SSH1 protocol requires it. Use sleep 100000. Don't use an infinite loop like the shell command while true; do false; done. This gives you the same effect, but your remote shell will eat all the spare CPU time on the remote machine, annoying the sysadmin and shortening your account's life expectancy.

Q: ssh1 prints "Hostname or username is longer than 255 characters" and exits.

A: ssh1 has a static limit of 255 characters for the name of a remote host or a remote account (username). You instructed ssh1, either on the command line or in your configuration file, to use a hostname or username that's longer than this limit.

Q: ssh1 prints "No host key is known for <server name> and you have requested strict checking (or `cannot confirm operation when running in batch mode')," and exits.

A: The client can't find the server's host key in its known-hosts list, and it is configured not to add it automatically (or is running in batch mode and so can't prompt you about adding it). You must add it manually to your per-account or systemwide known-hosts files.

Q: ssh1 prints "Selected cipher type . . . not supported by server" and exits.

A: You requested that ssh1 use a particular encryption cipher, but the SSH1 server doesn't support it. Normally, the SSH1 client and server negotiate to determine which cipher to use, so you probably forced a particular cipher by providing the -c flag on the ssh1 command line or by using the `Cipher` keyword in the configuration file. Either don't specify a cipher and let the client and server work it out, or select a different cipher.

Q: ssh1 prints "channel_request_remote_forwarding: too many forwards" and exits.

A: ssh1 has a static limit of 100 forwardings per session, and you've requested more.

Q: scp printed an error message: "Write failed flushing stdout buffer. write stdout: Broken pipe" or "packet too long".

A: Your shell startup file (e.g., ~/.cshrc, ~/.bashrc), which is run when scp connects, might be writing a message on standard output. These interfere with the communication between the two scp1 programs (or scp2 and sftp-server). If you don't see any obvious output commands, look for stty or tset commands that might be printing something.

Either remove the offending statement from the startup file or suppress it for noninteractive sessions:

>     if ($?prompt) then
>       echo 'Here is the message that screws up scp.'
>     endif

The latest versions of SSH2 have a new server configuration statement, `AllowCshrcSourcingWithSubsystems`, which should be set to `no` to prevent this problem.

Q: scp printed an error message, "Not a regular file."

A: Are you trying to copy a directory? Use the -r option for a recursive copy. Otherwise, you may be trying to copy a special file that it doesn't make sense to copy, such as a device node, socket, or named pipe. If you do an ls -l of the file in question and the first character in the file description is something other than `-` (for a regular file) or `d` (for a directory), this is probably what's happening. You didn't really want to copy that file, did you?

Q: Why don't wildcards or shell variables work on the scp command line?

A: Remember that wildcards and variables are expanded by the local shell first, not on the remote machine. This happens even before scp runs. So if you type:

>     $ scp server.example.com:a* .

the local shell attempts to find local files matching the pattern `server.example.com:a*`. This is probably not what you intended. You probably wanted files matching `a*` on server.example.com to be copied to the local machine.

Some shells, notably C shell and its derivatives, simply report "No match" and exit. Bourne shell and its derivatives (sh, ksh, bash), finding no match, will actually pass the string `server.example.com:a*` to the server as you'd hoped.

Similarly, if you want to copy your remote mail file to the local machine, the command:

>     $ scp server.example.com:$MAIL .

might not do what you intend. `$MAIL` is expanded locally before scp executes. Unless (by coincidence) `$MAIL` is the same on the local and remote machines, the command won't behave as expected.

Don't rely on shell quirks and coincidences to get your work done. Instead, escape your wildcards and variables so the local shell won't attempt to expand them:

>     $ scp server.example.com:a\* .
>     $ scp 'server.example.com:$MAIL' .

Q: I used scp to copy a file from the local machine to a remote machine. It ran without errors. But when I logged into the remote machine, the file wasn't there!

A: By any chance, did you omit a colon? Suppose you want to copy the file myfile from the local machine to server.example.com. A correct command is:

>     $ scp myfile server.example.com:

but if you forget the final colon:

>     # This is wrong!
>     $ scp myfile server.example.com

myfile gets copied locally to a file called server.example.com. Check for such a file on the local machine.

Q: How can I give somebody access to my account by scp to copy files but not give full login permissions?

A: Bad idea. Even if you can limit the access to scp, this doesn't protect your account. Your friend could run:

>     $ scp evil_authorized_keys you@your.host:.ssh/authorized_keys

Oops, your friend has just replaced your authorized_keys file, giving himself full login permissions. Maybe you can accomplish what you want with a clever forced command, limiting the set of programs your friend may run in your account.

Q: scp -p preserves file timestamps and modes. Can it preserve file ownership?

A: No. Ownership of remote files is determined by SSH authentication. Suppose user smith has accounts on local computer L and remote computer R. If the local smith copies a file by scp to the remote smith account, authenticating by SSH, the remote file is owned by the remote smith. If you want the file to be owned by a different remote user, scp must authenticate as that different user. scp has no other knowledge of users and uids, and besides, only root can change file ownership (on most modern Unix variants, anyway).

Q: Okay, scp -p doesn't preserve file ownership information. But I am the superuser, and I'm trying to copy a directory hierarchy between machines (scp -r) and the files have a variety of owners. How can I preserve the ownership information in the copies?

A: Don't use scp for this purpose. Use tar and pipe it through ssh. From the local machine, type:

>     # tar cpf - local_dir | (ssh remote_machine "cd remote_dir; tar xpf -")

Q: sftp2 reports "Cipher <name> is not supported. Connection lost."

A: Internally, sftp2 invokes an ssh2 command to contact sftp-server. It searches the user's PATH to locate the ssh2 executable rather than a hardcoded location. If you have more than one version of SSH2 installed on your system, sftp2 might invoke the wrong ssh2 program. This can produce the error message shown.

For example, suppose you have both SSH2 and F-Secure SSH2 installed. SSH2 is installed in the usual place, under /usr/local, whereas F-Secure is installed under /usr/local/f-secure. You ordinarily use SSH2, so /usr/local/bin is in your PATH, but /usr/local/f-secure isn't. You decide to use the F-Secure version of scp2 because you want the CAST-128 cipher, which SSH2 doesn't include. First, you confirm that the SSH server in question supports CAST-128:

>     $ /usr/local/f-secure/bin/ssh2 -v -c cast server
>       ...
>     debug: c_to_s: cipher cast128-cbc, mac hmac-sha1, compression none
>     debug: s_to_c: cipher cast128-cbc, mac hmac-sha1, compression none

Satisfied, you try scp2 and get this:

>     $ /usr/local/f-secure/bin/scp2 -c cast foo server:bar
>     FATAL: ssh2: Cipher cast is not supported.
>     Connection lost.

scp2 is running the wrong copy of ssh2 from /usr/local/bin/ssh2, rather than /usr/local/f-secure/bin/ssh2. To fix this, simply put /usr/local/f-secure/bin earlier in your PATH than /usr/local/bin, or specify the alternative location of ssh2 with scp2 -S.

The same problem can occur in other situations where SSH programs run other programs. We have run afoul of it using host-based authentication with both 2.1.0 and 2.2.0 installed. The later ssh2 ran the earlier ssh-signer2 program, and the client/signer protocol had changed, causing it to hang.

Q: sftp2 reports "ssh_packet_wrapper_input: invalid packet received."

A: Although this error appears mysterious, its cause is mundane. A command in the remote account's shell startup file is printing something to standard output, even though stdout isn't a terminal in this case, and sftp2 is trying to interpret this unexpected output as part of the SFTP packet protocol. It fails and dies.

You see, sshd uses the shell to start the sftp-server subsystem. The user's shell startup file prints something, which the SFTP client tries to interpret as an SFTP protocol packet. This fails, and the client exits with the error message; the first field in a packet is the length field, which is why it's always that message.

To fix this problem, be sure your shell startup file doesn't print anything unless it's running interactively. tcsh, for example, sets the variable `$interactive` if stdin is a terminal. This problem has been addressed in SSH 2.2.0 with the `AllowCshrcSourcingWithSubsystems` flag, which defaults to `no`, instructing the shell not to run the user's startup file.

Q: I'm trying to do port forwarding, but ssh complains: "bind: Address already in use."

A: The port you're trying to forward is already being used by another program on the listening side (the local host if it's a -L forwarding or the remote host if it's a -R ). Try using the netstat -a command, available on most Unix implementations and some Windows platforms. If you see an entry for your port in the LISTEN state, you know that something else is using that port. Check to see whether you've inadvertently left another ssh command running that's forwarding the same port. Otherwise, just choose another, unused port to forward.

This problem can occur when there doesn't appear to be any other program using your port, especially if you've been experimenting with the forwarding feature and have repeatedly used the same ssh to forward the same port. If the last one of these died unexpectedly (you interrupted it, or it crashed, or the connection was forcibly closed from the other side, etc.), the local TCP socket may have been left in the TIME_WAIT state (you may see this if you used the netstat program as described earlier). When this happens, you have to wait a few minutes for the socket to time out of this state and become free for use again. Of course, you can just choose another port number if you're impatient.

Q: How do I secure FTP with port forwarding?

A: This is a complex topic. FTP has two types of TCP connections, control and data. The control connection carries your login name, password, and FTP commands; it is on TCP port 21 and can be forwarded by the standard method. In two windows, run:

>     $ ssh -L2001:name.of.server.com:21 name.of.server.com
>     $ ftp localhost 2001

Your FTP client probably needs to run in passive mode (execute the `passive` command). FTP data connections carry the files being transferred. These connections occur on randomly selected TCP ports and can't be forwarded in general, unless you enjoy pain. If firewalls or NAT (network address translation) are involved, you may need additional steps (or it may not be possible).

Q: X forwarding isn't working.

A: Use ssh -v, and see if the output points out an obvious problem. If not, check the following.

Make sure you have X working before using SSH. Try running a simple X client such as xlogo or xterm first. Your local DISPLAY variable must be set, or SSH doesn't attempt X forwarding.

X forwarding must be turned on in the client and server, and not disallowed by the target account (that is, with `no-X11-forwarding` in the authorized_keys file).

sshd must be able to find the xauth program to run it on the remote side. If it can't, this should show up when running ssh -v. You can fix this on the server side with the `XAuthLocation` directive (SSH1, OpenSSH), or by setting a PATH (that contains xauth) in your remote shell startup file.

Don't set the DISPLAY variable yourself on the remote side. sshd automatically sets this value correctly for the forwarding session. If you have commands in your login or shell startup files that unconditionally set DISPLAY, change the code to set it only if X forwarding isn't in use.

OpenSSH sets the remote XAUTHORITY variable as well, placing the xauth credentials file under /tmp. Make sure you haven't overridden this setting, which should look like:

>     $ echo $XAUTHORITY
>     /tmp/ssh-maPK4047/cookies

Some flavors of Unix actually have code in the standard shell startup files (e.g., /etc/bashrc, /etc/csh.login) that unconditionally sets XAUTHORITY to ~/.Xauthority. If that's the problem, you must ask the sysadmin to fix it; the startup file should set XAUTHORITY only if the variable is unset.

If you are using an SSH startup file (/etc/sshrc or ~/.ssh/rc), sshd doesn't run xauth for you on the remote side to add the proxy key; one of these startup files must do it, receiving the proxy key type and data on standard input from sshd

-- SP

Copyright © 2003 O'Reilly & Associates. All rights reserved.