So I found out about a need little shell called rssh by Derek Martin. You grant the users shell access, and set their shell to /usr/local/bin/rssh. But rssh is no normal shell; it will allow users to execute only one command, /usr/libexec/openssh/sftp-server (you can also allow other commands — scp, cvs, rdist and rsync — if you want). So that's two security problems eliminated (insecure FTP and shell access for users). With rssh it will be very difficult for users to obtain escalated privileges. The server now becomes a jewellery store in which authorized people can visit but only with their hands tied behind their back.

Total filesystem access

But the story doesn't end there. The problem with the OpenSSH implementation of SFTP is that it does not restrict users to their home directories. Once logged in, users can browse the entire file system, or at least the parts of which for which they have access. It is true that your file system permissions should be set in a way that prevents users from accessing stuff that they shouldn't have access to (like sensitive configuration files, logs which could reveal information that would be useful to an attacker looking for a weakness, or other people's email) but even so allowing free reign to explore the file system seems to be unnecessarily liberal and is simply asking for trouble.

When I was on FreeBSD I addressed this by using the commercial SFTP server from ssh.com, which allows you to "chroot" users to their directories upon logging in, thus preventing them from moving outside their home directories. When you "chroot" you are effectively "changing the root" of the file system. The user's home directory might be at /home/johnsmith/ but when logged in via SFTP that folder becomes the new "root" of the file system (in other words, to the user, /home/johnsmith/ looks to be / and he cannot cd "up" and out of that directory). For this reason this configuration is sometimes referred to as a "chroot jail"; the idea is that the user can't escape from it.

Now that I'm using RHEL I wanted to stick with the default OpenSSH install rather than resort to third-party tools; RHEL really manifests its strength, security and ease of maintenance when you keep things as close to the standard base installation as possible.

The good news is that rssh allows you to set up a chroot jail and confine users to it. The bad news is that it can be tricky to set up. But once it is set up, your jewellery store is even more secure: it's now a jewellery store in which each customer has their own private booth and it's physically impossible for them to break out of it and look at other customers' booths.

Why is setting up a chroot jail so hard? The main reason is that applications running inside the jail (in our case, /usr/libexec/openssh/sftp-server) often dynamically link to library files and those files need to be inside the jail too otherwise the applications cannot access them and will fail to run. When I run the ldd tool on the sftp-server application it gives me a list of libraries that will need to be made available inside the jail:

libresolv.so.2
libutil.so.1
libz.so.1
libnsl.so.1
libcrypto.so.4
libkrb5.so.3
libk5crypto.so.3
libcom_err.so.3
libc.so.6
libgssapi_krb5.so.2
libdl.so.2
ld-linux.so.2

I could copy these files to the jail but that would take up a lot of disk space if I had to do it for every single jail on the system. I could alternatively put all the users inside a single jail to save a bit of disk space; that would stop them from searching for sensitive configuration and log files but it still wouldn't prevent them from at worst seeing each other's data or at best finding out what other users have accounts on the system.

The solution is to use "hard links" ("symbolic" links won't work across the jail boundary). Inside the jail you can set up a "hard link" reference to each of the necessary library files. The reference takes up effectively no space on the disk and to the jailed user it looks, feels and acts just like the original file. If the user were to delete the link, the original file would be untouched. On my system the use of hard links saves about 3.6MB of disk space per jail (compared with simply copying the required files).

In addition to hard-linking the library files listed by ldd, you also need to provide hard links to the sftp-server application itself and the rssh_chroot_helper executable (along with all the libraries to which the latter links). If either of these executables makes use of the dynamic linker you also should provide links to the dynamic linker support files, /etc/ld.so.cache and /etc/ld.so.conf. Finally, you'll find that you'll need an appropriate /etc/passwd inside the jail itself in order for authentication to succeed.

There are some technical hurdles to be overcome in doing all this:

1. It's time consuming to manually set this all up; an automated method for setting up a chroot jail needs to be developed. That means we need to write a shell script.
2. Prior to creating any links we need to ensure that the appropriate directories are in place; for example, if we need to link to libraries in /usr/lib/ then each chroot jail must contain usr/lib/ subdirectories.
3. Most of the library files listed by ldd are actually symbolic links themselves, and if you try to create a hard link that points to a symbolic link you'll get a warning and it won't work properly.
4. It can be cumbersome to find out in an automated way what the name of the real file (the target of the symbolic link) really is.
5. Once you know the name of the target file and the name of the symbolic link you'll need to create a hard link to the real target file and a relative symbolic link to your newly created hard link.
6. One of the goals of using the chroot jail is to prevent users from knowing what other user accounts are on the system: this means that instead of just hard-linking to the /etc/passwd file you should create a new one for each user inside the user's jail which contains only the entry for that user.
7. You should be able to run the script and then re-run it at a later date (for example, let's say you update your OpenSSH or rssh installations and want to "refresh" the jails).
8. You need to set directory and permissions carefully so that a user logged in to a chroot jail can't inadvertently delete, move or rename any of these important files, break the jail and effectively lock themselves out.
9. The script needs to run with root privileges to do all this, and giving that it is both automatic and making lots of changes to the file system it needs to implement lots of error checking, bail out on the first sign that things might be going wrong, and be used with extreme caution.
10. Bonus points for automatically appending per-user configuration entries to the /usr/local/etc/rssh.conf file.

The rssh distribution includes a script called mkchroot.sh which does a lot of this set-up for you if you're running the right kind of Red Hat distribution. I wanted to write my own script from scratch, however, because there are a few things I felt I could do better. For example: mkchroot.sh doesn't create hard links, it copies all of the libraries, taking up a lot of disk space unnecessarily; it also copies tools (for example scp) and libraries (such as name service resolution libraries) which I don't want or need inside my jails; finally, it copies the entire /etc/passwd file rather than a subset of it.

Now, I'm by no means a shell scripting expert. In fact, I feel very uncomfortable writing shell scripts, but I've managed to come up with something that seems to work. It meets all of the goals I listed above and addresses the issues I have with the mkchroot.sh script. It's main distinguishing features are: the use of variables to identify commandline executables (this is a habit of mine which ensures that the script will work regardless of what the PATH setting is, and it allows me to change all references to a tool in the script by changing a single variable); and the definition of a number of functions — die(), createdir(), checkbase() and link() — to handle commonly performed operations and make the code more readable

Here's the first version of my script which has performed fine in my own testing on RHEL3. If you choose to use this script or any part of it you do so at your own risk:

#!/bin/sh
# mkrsshjail.sh
# Copyright 2006 Wincent Colaiuta
# Version 1.0 (20 January 2006)


#
# Constants
#


# Tools/executables:
AWK="/usr/bin/awk"
BASENAME="/bin/basename"
CAT="/bin/cat"
CHOWN="/bin/chown"
DIRNAME="/usr/bin/dirname"
EGREP="/bin/egrep"
FIND="/usr/bin/find"
GREP="/bin/grep"
LDD="/usr/bin/ldd"
LN="/bin/ln"
LS="/bin/ls"
MKDIR="/bin/mkdir"
RSSHHELPER="/usr/local/libexec/rssh_chroot_helper"
SFTPSERVER="/usr/libexec/openssh/sftp-server"
WC="/usr/bin/wc"
WHOAMI="/usr/bin/whoami"


# Support files:
LDCACHE="/etc/ld.so.cache"
LDCONF="/etc/ld.so.conf"
PASSWD="/etc/passwd" 
RSSHCONF="/usr/local/etc/rssh.conf"


#
# Functions
#


die ()
{
  SCRIPT=`"${BASENAME}" "$0"`
  echo "${SCRIPT}: error: $1" >&2
  exit 1
}


createdir ()
{
  if [ $# -ne 1 ]; then
    die "createdir() requires exactly one argument"
  fi 
  if [ ! -d "$1" ]; then
   if [ -e "$1" ]; then
     die "createdir(): $1 already exists and is not a directory"
   else 
    "${MKDIR}" -p -v "$1" || die "createdir() failed for path: $1"
   fi
  fi
}


checkbase ()
{
  if [ $# -ne 1 ]; then
    die "checkbase() requires exactly one argument"
  fi
  BASE="$1"
  COUNT=`echo "${BASE}" | "${EGREP}" -c "\/$"`
  if [ $COUNT -ne 0 ]; then
    die "checkbase() requires base directory to have no trailing slash"
  fi
}


link ()
{
  if [ $# -ne 2 ]; then
    die "link() requires exactly two arguments"
  fi
  FILE="$1"
  BASE="$2"
  DIR=`"${DIRNAME}" "${FILE}"`
  checkbase "${BASE}"
  createdir "${BASE}${DIR}"
  if [ ! -f "${FILE}" ]; then
    die "link(): ${FILE} is not a regular file"
  fi
  if [ -L "$1" ]; then
    NAME=`"${BASENAME}" "${FILE}"`
    TARGET=`"${FIND}" "${DIR}" -name "${NAME}" -printf "%l\n"`
    TRUENAME="${DIR}/${TARGET}"
    echo "${FILE} is a symbolic link with target ${TRUENAME}"
    "${LN}" -v -f "${TRUENAME}" "${BASE}${TRUENAME}"  || \
      die "link() failed for file: ${TRUENAME}" 
    pushd "${BASE}${DIR}"
    "${LN}" -v -f -s "${TARGET}" "${NAME}" || \
      die "link() (symbolic) failed for file: ${BASE}${FILE}"
    popd
  else
    "${LN}" -v -f "${FILE}" "${BASE}${FILE}" || \
      die "link() failed for file: ${FILE}"
  fi
}


#
# Main
#


if [ `"${WHOAMI}"` != "root" ]; then
  die "this tool must be run as root"
fi


if [ $# -ne 2 ]; then
  die "exactly two arguments required ('path to chroot' and 'username')"
fi


BASE="$1"
USER="$2"
checkbase "${BASE}"


# create relevant /etc/passwd entry
DIR=`"${DIRNAME}" "${PASSWD}"`
createdir "${BASE}${DIR}"
"${CAT}" "${PASSWD}" | "${EGREP}" "^${USER}:" > "${BASE}${PASSWD}"
COUNT=`"${WC}" -l "${BASE}${PASSWD}" | "${AWK}" ' { print $1 } '`
if [ $COUNT -ne 1 ]; then
  die "error creating ${BASE}${PASSWD} from ${PASSWD}"
fi 


# Get an up-to-date list of libraries linked to by sftp-server:
LIBS=`"${LDD}" "${SFTPSERVER}" | "${AWK}" ' { print $3 } '`
for LIB in ${LIBS}; do
  link "${LIB}" "${BASE}"
done


LIBS=`"${LDD}" "${RSSHHELPER}" | "${AWK}" ' { print $3 } '`
for LIB in ${LIBS}; do
  link "${LIB}" "${BASE}"
done


# hardlink support files
link "${LDCACHE}" "${BASE}"
link "${LDCONF}" "${BASE}"


# hardlink tools
link "${RSSHHELPER}" "${BASE}"
link "${SFTPSERVER}" "${BASE}"


# append to rssh config
CONFIG="user=${USER}:022:00010:\"${BASE}\""
COUNT=`"${CAT}" "${RSSHCONF}" | "${EGREP}" -c "${CONFIG}"`
if [ $COUNT -eq 0 ]; then
  echo "Appending line to ${RSSHCONF}"
  echo "${CONFIG}"
  echo "${CONFIG}" >> "${RSSHCONF}"
fi


# tighten ownership on chroot jail dir
"${CHOWN}" -v root:root "${BASE}"


exit 0

The only shortcoming of the per-user chroot jail approach is that logging via the syslog facility is broken. It is possible to modify your start-up scripts to start syslog with additional "FIFOs" inside each jail, but there are problems with this approach; as the rssh CHROOT documentation says:

... in order to log chroot activity, you need to create a socket for syslog to log to (i.e. /dev/log, see above) in each of the chroot jails. However, the number of sockets syslog can listen to is generally limited. On most Linux systems, that limit is a maximum of 20, though it can be changed by editing the source code to syslogd. On other platforms, this may not even be possible. So if you go with individual chroot jails, you either need to be satisfied without logging for most of your users, or recompile syslogd and manually configure it to open a socket in each of your users' jails. And while I haven't done it, I suspect creating large numbers of sockets for syslogd to listen to will make logging to syslog somewhat painful.

For the time being, I'll just have to be "satisfied without logging", although the truth is that I am not sure exactly what kind of stuff would be logged by sftp-server and rssh anyway...

Hardening Apache CGI and PHP

So the combination of SFTP, rssh and per-user chroot provides quite a good degree of security. We now have a jewellery store in which customers can only enter with their hands tied behind their backs, they are confined to their own private booths, and they have no way of knowing what other customers have booths at the store.

That leaves us with the "shared server" security problems which have long plagued Apache. These are not a security "hole" in the formal sense but rather a consequence of the fact that Apache generally runs as user "apache" (or "nobody" or "www") and that means that everybody with a virtual host on the server ends up running with same privileges. That means that a user can write a CGI or PHP script, and that script will be able to read/write any file for which the webserver has privileges. Malicious users could sniff for database passwords and other sensitive information. This is like providing all of our jewellery store customers with equally-capable remote control robots which can move around outside of their private booths.

There are ways around this but they are generally not all that compatible with the idea of mass shared/virtual hosting. For example, you could run one Apache process for every user on the system but that's not viable for large numbers of virtual hosts. You could also try to set up a private virtual server for each user using software like Xen but again scalability is a problem.

When I was on FreeBSD I used the inbuilt jail subsystem (see also) which allowed me to create relatively lightweight virtual private servers. In theory no user could break out of his or her jail nor see or touch other users' data. But, still, there were limits to how many I could comfortably fit on a single host machine. Each one required its own IP address and considerable disk space. Furthermore, every time that I had to patch the host operating system I had to patch it inside each of the jailed systems as well. Even with automation it was a time consuming and complicated process.

With the move to Rackspace I was moving to a system with a single IP address and with no OS-level support for anything like FreeBSD's jail solution. So I had to do the best I could to lock things down without resorting to virtualization.

CGI scripts are fairly elegantly handled by the Apache's suEXEC support. With suEXEC, when "johnsmith" uploads a CGI script to the server Apache will run that script as user "johnsmith", not as user "apache". Another user, "henrykrinkle", will find that his scripts run as user "henrykrinkle". When this is combined with appropriate file permissions, the is no way that "johnsmith" can use a CGI script to read or alter the files of "henrykrinkle" and vice versa.

PHP is not so easily handled. It has a thing called "Safe Mode" which is designed to partially address the shared server security problem. I say partially because the PHP documentation itself admits that the solution is not ideal:

It is architecturally incorrect to try to solve this problem at the PHP level, but since the alternatives at the web server and OS levels aren't very realistic, many people, especially ISP's, use safe mode for now.

The problem is that PHP is generally used as an Apache module so by definition PHP scripts run with the same user, group and privileges as the webserver itself. You can try installing PHP as a CGI binary but you may find that not all existing PHP software plays nicely with it, it may have security risks of its own, and in any case PHP-as-cgi is not a standard part of RHEL so you lose the benefit of any automatic updates provided by Red Hat.

Safe Mode is pretty much your only option in this case. It won't make your scripts run with the user and group of the user that owns the scripts, but it will perform additional checks before executing code and opening files, it allows you to restrict access to certain parts of the file system (such as a user's home directory) and it allows you to disallow the use of certain functions. It is not perfect, impossible-to-circumvent security, but it is better than none at all.

To turn on suEXEC and Safe Mode protection for "johnsmith" you'd add something like this to his VirtualHost entry in your Apache configuration:

SuexecUserGroup johnsmith johnsmith
php_admin_value safe_mode "1"
php_admin_value open_basedir "/home/johnsmith/public_html/"

One of the most annoying things about Safe Mode is that files created by PHP scripts will be owned by the Apache process itself and not by the user running the scripts. This can make it difficult for scripts to delete files which they themselves created. There is also a lot of PHP software out there that doesn't work properly with Safe Mode and in those cases you may have to turn off Safe Mode on a per-directory basis. If you do this, try to disable Safe Mode in the smallest possible region of the file system that will enable the software to work (in other words, turn it off in the sub-directory where the software is installed rather than turning it off for the user's entire home directory). You can also often leave the open_basedir directive in effect, at least providing some degree of protection from casual snooping.

So to take our jewellery store analogy to its ultimate, contorted extreme, what we now have is a jewellery store in which persons who provide the appropriate identification are allowed to enter into their own private booths with their hands tied behind their backs, and with no way of knowing what other customers might have private booths of their own. Each customer has a remote control robot that could conceivably be used to snoop around outside their booth, but it's a disabled robot that's programmed to not move outside certain boundaries and is unlikely to be of much use to all but the most skillful and determined attacker. And if that's the kind of user you're giving accounts to perhaps you should think twice about it.