By Rainer Wichmann email@example.com (last update: Dec 29, 2009)
Scenarios for break-ins / intrusions commonly assume that the intruder will gain superuser (root) privileges, and will manage to completely subvert the host after an intrusion.
However, usually the intruder does not have root privileges initially, either because:
- the break-in was performed by exploiting a vulnerability in some network deamon running as a non-root user, or
- the break-in happened by social engeneering or by (brute-force) guessing the password of a non-root user.
- your machines are well patched, and/or
- the intruder (only) uses some outdated exploits that are ineffective against up-to-date patched hosts.
While the last point sounds a bit optimistic, experience shows that a sizeable fraction of intruders are not researchers equipped with bleeding-edge tools, but rather people with somewhat limited knowledge, relying on a small set of programs/exploits that are known to them, but may not work on well patched hosts.
What will the intruder do?
Even without superuser privileges, the machine is quite useful for an intruder, e.g. to:
- perform scanning in order to locate other vulnerable hosts on the network,
- install a file sharing server (serving from a nonstandard, non-privileged port) to distribute illegal content (e.g. pirated software or movies),
- use the host as a spam relay, or
- establish a web server (again on a nonstandard, non-privileged port) to be used for some phishing scam.
How do you determine the presence of a nonprivileged intruder?
Having gained superuser privileges, intruders often start to fuss about the machine in order to install some rootkit, with the aim to hide their presence and secure their access. This, however, modifies the operating system, and thus actually helps to detect them. Sometimes the activity of an intruder even may botch up the operating system (e.g. by a buggy rootkit), leading to prompt detection of the break-in.
On the other hand, legitimate nonprivileged shell users notoriously tend to do lots of random and unpredictable things, often with no useful pattern. Having users that travel around the world and ssh into your hosts at random times doesn't help either. I.e. there is a lot of 'noise' that helps an intruder to 'blend in', and can make it difficult to detect an intrusion.
Things to watch out for include (but may not be limited to):
Suspicious outbound traffic
Usually, an intruder will try to use your machine(s) in order to gain access to other machines, e.g. by running portscans and/or network attacks. Unless the intruder is cautious and performs these activities very slowly, this will generate very unusual and suspicious outbound network traffic that can be captured with a network sniffer such as e.g. snort.
Suspicious open ports
Intruders may use your hosts as a depot for illegal content like e.g. "pirated" copies of movies. Distributing such content requires some network application (e.g. an ftp server or a peer-to-peer application) that opens a port to listen for incoming connection attempts.
Information on open TCP ports may be collected with the command
netstat -pant | grep LISTEN
lsof -i tcp | grep LISTEN
Likewise, information on open UPD ports can be collected with the
netstat -panu | grep -v ESTABLISHED
Unfortunately, automated evaluation of such information may get complicated by user applications which open ports for more or less legitimate reasons. This doesn't only apply to notorious applications like skype - there are quite a few ill-coded application suites that rely on sockets for interprocess communication, and by default open a TCP socket for all interfaces, even if in most cases a Unix Domain Socket would suffice. For this reason, you may want to monitor open ports in order to learn how the system looks normally.
Suspicious established connections
There are file sharing programs which can log in to an IRC channel, wait for commands, and then initiate a connection to a designated IP address in order to deliver content (let's say, pirated movies or software).
As you may notice, this scheme completely avoids any open ports. All connections are outbound connections from the hijacked machine. Checking for open ports is useless if an intruder has installed such software.
For this reason, it is highly recommended to run regular checks
for established connections, and familiarize
yourself with the 'normal' pattern of software and/or outbound
connections of legitimate users, such that you are able to
spot anomalies, like established connections involving new and/or
unusual software. To check for established TCP connections, you
netstat -pant | grep ESTABLISHED
Some intruders manage to botch the system by exhausting the diskspace available for /tmp or /var/tmp. This may lead to obvious problems if other applications cease to function properly.
Suspicious file names
You may want to look out for files or directories like ".. "
(note trailing whitespace) or " " in /tmp and
in user home directories.
With GNU ls, using
ls -Q is helpful (this encloses filenames
in double quotes).
Crontab, ~/.profile, ~/.bashrc
If an intruder runs some service from your host(s), then he/she is probably interested in the service surviving a reboot. Since an unprivileged user cannot start services during the boot sequence, the intruder has to resort to other methods. One possibility would be to install a cron job that checks at regular intervals whether the service is running, and restarts it if required.
Another option would be to modify the ~/.profile or ~/.bashrc file of the user (if the intruder has gained access to a shell account). In this case, whenever the legitimate user starts a shell, the code implanted by the intruder would run.
Recovering the actions of the intruder
Finding out what an intruder has done is important for assessing the damage, and cleaning up after detection of the intrusion. The following is a (probably incomplete) list of things that will help you in this task.
The shell history
I will only discuss the bash shell here, but most Bourne-style shells should be similar. Usually, the intruder will try to hide his actions by preventing the shell from writing the history file. This can be done by any of the following means:
- Remove the history file ($HOME/.bash_history), then create a directory with the same name (or a link to /dev/null).
- Unset the HISTFILE shell environment variable, or set it to /dev/null.
- Set the HISTSIZE and/or HISTFILESIZE shell environment variable(s) to 0 (zero).
It is not possible to fully prevent a user from evading the shell command history. However, the following steps will serve to protect against the most commonly used methods (i.e. those shown above):
- Use file flags to make the history file of your users append-only.
On Linux, set the 'append-only' attribute on the history file with
chattr +a ~username/.bash_history
(may not be supported by all filesystem types). See 'man chattr/lsattr' for details.
Not feasible for NFS-mounted home directories (no protocol support). Read/write operations would be impossible.
On FreeBSD, set the 'append-only' file flag with
chflags sappnd ~username/.bash_history
See 'man chflags' for details. Probably will also not work for NFS-mounted home directories.
- On Linux, set the 'append-only' attribute on the history file with
- Use the readonly shell builtin to make the relevant shell
environment variables readonly. E.g. you could include the following
lines in /etc/profile:
readonly HISTFILE readonly HISTFILESIZE readonly HISTIGNORE readonly HISTSIZE
Process accounting allows an administrator to track resource usage by users as well as the commands executed by them. The primary advantage is that process accounting cannot be disabled by unprivileged users (even more, they can't even determine whether it takes place).
There are however also significant disadvantages:
- It may generate large logfiles, requiring a lot of diskspace,
- only the command itself is recorded, not the arguments (thus e.g. you can learn that a user has run rm, but you won't know which file has been removed), and
- process accounting information gets written only when a process exits, i.e. as long as a process still runs, it will not show up.
On Debian/Ubuntu Linux, 'apt-get acct' will install the userspace accounting utilities and enable accounting. On Fedora/RHEL, the package is named 'psacct', and after installation, you need to start process accounting manually with:
# chkconfig psacct on # /etc/init.d/psacct start
For FreeBSD, to enable process accounting use:
# touch /var/account/acct # accton /var/account/acct # echo 'accounting_enable="YES"' >> /etc/rc.conf
After running your system for some time (to let a bit of logging
information accumulate), you can then use the 'lastcomm' command
to get information about user activity. E.g.:
may yield information like:
bash F rainer stderr 0.00 secs Wed Dec 27 12:12 sed rainer stderr 0.01 secs Wed Dec 27 12:12 bash F rainer stderr 0.00 secs Wed Dec 27 12:12 bash F rainer stderr 0.00 secs Wed Dec 27 12:12 tr rainer stderr 0.01 secs Wed Dec 27 12:12 wc rainer stderr 0.00 secs Wed Dec 27 12:12 ...
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 Germany License.