04- Logging, SSH, & Situational Awareness in Linux

Logging in Linux

Introduction

Linux systems use the syslog standard as the preferred method for system logs. An informal standard for many years, these were codified in RFC 3164 and then updated in RFC 5425. Various tools have been used to implement the syslog standard; for many years the most common was called syslog. That tool had a number of shortcomings, which led to the development of alternative implementations, including syslog-ng and rsyslog. The latter of these is used on all of the images provided to the class, though in slightly different versions. CentOS 6.2 uses rsyslog 4.6.2, Ununtu 10.04 uses rsyslog 4.2.0 while Mint 11 uses rsyslog 4.6.4.

Programs running in a Linux environment can dispatch a log message using syslog. Each message is assigned two values- a facility, which identifies the type of message, and a severity, which determines its importance. The facilities are

  • auth: Formerly used for security/authorization messages, but deprecated in favor of authpriv.
  • authpriv: Used for security/authorization messages.
  • cron: Used for time based services, including the clock daemon, cron.
  • daemon: Used for system daemons without separate facility values
  • ftp: Used for the ftp server.
  • kern: Used solely for kernel messages.
  • local0, local1, …, local7: Available for local system use.
  • lpr: Used for the print subsystem.
  • mail: Used for the mail server.
  • news: Used for the news server (NNTP; see e.g. RFC 977).
  • syslog: Used for messages generated by the syslog server itself.
  • user: Default facility; used for generic messages.
  • uucp: Used for the (now obsolete) Unix to Unix Copy system (UUCP).

The priorities are, in decreasing order of importance:

  • emerg (emergency)
  • alert
  • crit (critical)
  • err (error)
  • warning
  • notice
  • info (informational)
  • debug

Open the configuration file /etc/rsyslog.conf from your CentOS 6.2 machine, and take a look at the section marked rules.

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Before we even look at the documentation for rsyslog, can you determine where most log messages are sent? Now do you know why when we were setting up BIND, we looked through /var/log/messages for error reports?

Though this is how our Linux system was set up, we do not have to keep it this way. Create the file /var/log/testlog; then based on your reading of the rsyslog.conf file, modify it so that all messages get sent to that file. Save your changes, and restart the rsyslog service by running

[root@proclas log]# /etc/init.d/rsyslog restart
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]

When this is complete, take a look at your file /var/log/testlog; it should contain (at least) the log message saying that rsyslog was restarted.

[root@proclas log]# tail /var/log/testlog 
Feb  9 08:14:34 proclas kernel: imklog 4.6.2, log source = /proc/kmsg started.
Feb  9 08:14:34 proclas rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" 
x-pid="3009" x-info="http://www.rsyslog.com"] (re)start

Also examine the file /var/log/messages to see if any messages were sent to both files. What conclusions do you draw?

Sending Custom Log Messages

Take a look at the man page for the tool logger. Using this tool, we can craft log messages with specified facilities and priorities. Modify your /etc/rsyslog.conf file so that only messages from local1 of priority err or higher are sent to the file /var/log/testlog. Restart rsyslog and test your settings by sending messages with various facilities and priorities.

Hey, wait a minute. You mean any user can send any log message they want to the log server on a Linux system? Demonstrate that you can use logger to send a message that makes it appear as if the named server has shut down.

Though rsyslog is quite complex, it is also solidly documented. Take a look at the file /usr/share/doc/rsyslog-4.6.2/manual.html

If you want, you can also write your own program to interface directly with the logging system. Documentation to do so is provided in the man subsystem; take a look at man 3 syslog to see the details.

Consider the following C program

#include<syslog.h>

int main(int argc, char* argv[])
{
   const char log_ident[] = "Log Testing Program";
   const int log_option = LOG_NDELAY | LOG_PID;
   const int log_facility = LOG_LOCAL3;
   openlog(log_ident, log_option, log_facility);
   
   syslog(LOG_CRIT, "I just sent a critical error!");

   closelog();

   return(0);
}

It compiles and runs in the usual fashion

[seldon@proclas ~]$ gcc log.c -o log -Wall -pedantic
[seldon@proclas ~]$ ./log 

Interestingly, though the user seldon can write to the logs, that user cannot read the logs:

[seldon@proclas ~]$ tail /var/log/messages
tail: cannot open `/var/log/messages' for reading: Permission denied

However, checking the logs as root shows that the message was sent

[root@proclas log]# tail /var/log/messages
Feb  9 08:29:02 proclas kernel: imklog 4.6.2, log source = /proc/kmsg started.
Feb  9 08:29:02 proclas rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" 
x-pid="3009" x-info="http://www.rsyslog.com"] (re)start
Feb  9 08:39:34 proclas Log Testing Program[7037]: I just sent a critical error!
[root@proclas log]# 

Suppose we modify the program slightly

#include<syslog.h>

int main(int argc, char* argv[])
{
   const char log_ident[] = "named [31337]";
   const int log_option = LOG_NDELAY ;
   const int log_facility = LOG_SYSLOG;
   openlog(log_ident, log_option, log_facility);
   
   syslog(LOG_CRIT, "The name server just sent a critical error!");

   closelog();

   return(0);
}

Now when this is run, it will look to the sysadmin that the named process, with PID 31337, just had a critical error:

[root@proclas log]# tail -n1 /var/log/messages
Feb  9 09:06:14 proclas named [31337]: The name server just sent a critical 
error!

Just think of the laughs you can have!

Aggregating Multiple Logs

Let’s learn how to centralize the logs from multiple sources on a single host. Before we learn how to do this, let us consider why. Suppose that you are managing a real network; then you are going to have many different systems. In the case of an error or an intrusion and the absence of a centralized logging system, then you would need to visit the logs for each system individually; moreover if the issue crosses multiple systems (as might be expected from a real intrusion) then you would have multiple files to examine and correlate. Putting the logs from multiple systems in a single place gives us much better insight into our network. Further, if an attacker does compromise a system, one thing that they want to do is to erase their tracks from the system logs. If these are kept on a different system, then the attacker’s job is much more difficult.

All of the different syslog implementations allow for remote logging, however the structure of the configuration files vary with the precise distribution.

The CentOS 6.2 configuration is probably the simplest, as it is all contained in the single file /etc/rsyslog.conf.

Mint 11 and Ubuntu 10.04 end the file /etc/rsyslog.conf with a directive to include all of the files in the directory /etc/rsyslog.d

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

That directory contains two files, 50-default.conf which determines where log messages of a given priority and facility are sent, and the file 20-ufw.conf that sends all log messages that begin with the text "[UFW " to a separate file. UFW is the firewall configuration tool for Ubuntu machines. Note also that to start, stop, restart, or check the status of a service, the preferred method is to use the service command in the form

seldon@ubuntu:~$ sudo service rsyslog restart
rsyslog start/running, process 2805

Note also that the these systems store most messages in the file /var/log/syslog.

For definiteness int his example, we will configure a CentOS system; you should be able to make the necessary modifications for the other provided distributions. In this example, our CentOS system is named proclas.cosc.tu and is located at 10.0.2.13. We will be sending our logs to a Linux system located at 10.0.2.3.

The original standard for syslog only allowed for sending and receiving remote logs via UDP on port 514. To configure our syslog server to do so, instead of providing a file name as the destination for our logs, we provide the ip address, preceded by “@”; so if we add the line

*.*     @10.0.2.3

to our rsyslog.conf file, we are sending all messages, regardless of facility or priority to the host 10.0.2.3 via UDP on port 514.

We want to test this setup, but because we are now going to be sending network traffic, we will use a packet sniffer like Wireshark to actually see the packets. Wireshark is not set up on the class images for the CentOS or the Ubuntu systems (you can install it if you wish!), but it is available on the Backtrack systems. Since VMWare internally uses a hub (rather than a switch) for network traffic, a Backtrack VM can sniff the traffic to/from any other VM on the same physical host.

Here is the packet capture for traffic observed after we restarted the log server on proclas.cosc.tu; note that we used a Wireshark filter so that we only see the traffic that interests us.

Wireshark-Out

Looking at the destination log server (capella.cosc.tu, 10.0.2.3), we see that nothing has appeared in the logs. Of course, this was expected after we saw the packet capture, as we saw the ICMP messages telling 10.0.2.13 that the UDP port on 10.0.2.3 was closed.

To configure the server that will receive the log messages, we must load the required module to receive UDP traffic, then tell it to listen on port 514. In the CentOS 6.2, Mint 11, or Ubuntu 10.04 /etc/rsyslog.conf file, these lines already exist and simply need to be uncommented:

# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514

Restart the rsyslog server on the listening host, and be sure to also open the proper port in the firewall. At this point, you will see all log messages on the first host arrive on the listening host. Here is what you might see on the listening host

seldon@capella:~$ sudo tail -n 6 /var/log/messages
Feb  9 10:11:14 capella kernel: [   77.934187] __ratelimit: 9 callbacks suppressed
Feb  9 10:11:14 capella kernel: [   77.934189] type=1503 audit(1360433474.949:15):  operation="capable" pid=1651 parent=1642 profile="/usr/sbin/cupsd" name="sys_admin"
Feb  9 10:19:30 proclas kernel: Kernel logging (proc) stopped.
Feb  9 10:19:30 proclas rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="7775" x-info="http://www.rsyslog.com"] exiting on signal 15.
Feb  9 10:19:30 proclas kernel: imklog 4.6.2, log source = /proc/kmsg started.
Feb  9 10:19:30 proclas rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="7875" x-info="http://www.rsyslog.com"] (re)start

You can see the log entries from the local host (capella) as well as the log entries from the remote host (proclas) all together in the same file.

There is a downside to receiving logs though. We have already seen how to spoof logs, at least if we can run programs on the system. If the system is accepting logs remotely though, we can do more.

Try the following on your Backtrack box, where the IP address is the address of your log server.

root@bt:~# nc -w1 -u 10.0.2.3 514 <<< "This should be fun"

What entries (if any) appear in your log?

nc is the netcat command- it is well worth learning what we can do
with this command. If you want to specify the facility and/or priority via netcat, take a look at
http://linux.byexamples.com/archives/412/syslog-sending-log-from-remote-servers-to-syslog-daemon

There are a few disadvantages to using UDP for logs, the first of which is that it does not guarantee delivery. We can use rsyslog to send logs via TCP; to do so add the entry

facility.priority @@ip.address:port

to /etc/rsyslog.conf. Now we use the "@@" instead of "@" to indicate TCP.

To enable rsyslog to receive remote logs (TCP), include the lines

# Provides TCP syslog reception
$ModLoad imtcp.so  
$InputTCPServerRun 514

in your rsyslog configuration files. (These lines already exist (commented out) for the provided distributions). Note that we have selected TCP port 514 to receive the logs, but we could have selected other ports.

Again, open the firewall, and restart rsyslog; check that your log messages arrive.

What does a packet capture of TCP syslog traffic show?

What happens if you try

root@bt:~# nc 192.168.1.60 514 <<< "message"

where the ip address (and port) of the log server is 192.168.1.221 (514)?

Log Rotation

The logs generated cannot be kept indefinitely; as they continue to expand in size they will eventually consume all system resources. The logrotate tool is used on linux systems to compress, archive, rotate and delete log files.

Read the man page for logrotate.

Take a look at the file /etc/logrotate.conf, and answer the following:

  • How often are the log files rotated?
  • How long are log files kept as archives?
  • Are the log files compressed?
  • Are their any special features for the log rotation of the DNS server logs? Did you look in the logroatate.d subdirectory?
  • What about the web server logs?

Splunk

Splunk is a useful tool to manage the logs for large numbers of systems. Though we have seen how to combine the logs from multiple Linux systems into one location, we are still left with the problem of extracting useful information from those logs. Splunk is one of the tools that will let us do exactly that.

On CentOS, choose the right rpm (32 bit or 64 bit) and then install as usual:

[root@proclas ~]# rpm -ivh /home/seldon/Desktop/splunk-5.0.2-149561-
linux-2.6-x86_64.rpm 
warning: /home/seldon/Desktop/splunk-5.0.2-149561-linux-2.6-x86_64.rpm: 
Header V3 DSA/SHA1 Signature, key ID 653fb112: NOKEY
Preparing...                ########################################### [100%]
   1:splunk                 ########################################### [100%]
-------------------------------------------------------------------------
Splunk has been installed in:
        /opt/splunk

To start Splunk, run the command:
        /opt/splunk/bin/splunk start


To use the Splunk Web interface, point your browser to:
    http://proclas.cosc.tu:8000


Complete documentation is at http://docs.splunk.com/Documentation/Splunk
-------------------------------------------------------------------------
[root@proclas ~]# 

On a Mint or Ubuntu system, we use the appropriate .deb

seldon@nexon ~ $ sudo dpkg -i Desktop/splunk-5.0.2-149561-linux-2.6-intel.deb 
[sudo] password for seldon: 
Selecting previously deselected package splunk.
(Reading database ... 139950 files and directories currently installed.)
Unpacking splunk (from .../splunk-5.0.2-149561-linux-2.6-intel.deb) ...
Setting up splunk (5.0.2-149561) ...
-------------------------------------------------------------------------
Splunk has been installed in:
        /opt/splunk

To start Splunk, run the command:
        /opt/splunk/bin/splunk start


To use the Splunk Web interface, point your browser to:
    http://nexon.cosc.tu:8000


Complete documentation is at http://docs.splunk.com/Documentation/Splunk
-------------------------------------------------------------------------
Disk Space

One problem has been reported in previous versions of this class using Splunk on the CentOS system. That system was created with the default disk size and partition structure from VMWare; however consider the size of the partitions:

[root@proclas ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             3.9G  2.7G  975M  74% /
tmpfs                 498M  708K  498M   1% /dev/shm
/dev/sda1              49M   42M  4.1M  92% /boot
/dev/sda5              14G  239M   13G   2% /home

The root partition (/) has only 975M free, while most of the free space on the disk has been allocated to the nearly empty partition for /home. [Don’t even ask about how tight the boot partition is; trust me, you can’t update the kernel!]

The situation for the Mint system (for example) is different:

seldon@nexon ~ $ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              19G  3.5G   15G  20% /
none                  492M  648K  491M   1% /dev
none                  501M  148K  501M   1% /dev/shm
none                  501M  352K  501M   1% /var/run
none                  501M     0  501M   0% /var/lock

Here /boot, /home and /opt are all contained in the root partition, which has plenty of space.

If you use Splunk on the CentOS system, you may run out of disk space, as the partition containing /opt is the root partition. In general, Splunk likes to have at least 2G of free space. Past students have recommended copying the Splunk install to a convenient directory inside /home if it is to collect a significant amount of data. This may work if it is done as part of the install; I couldn’t make it happen. You can however, edit the file /opt/splunk/etc/splunk-launch.conf to change the location of the Splunk database; I set it to read

SPLUNK_DB = /home/splunk

It whined less.

Starting Splunk

To start splunk for the first time, run (as root, or via sudo)

[root@proclas ~]# /opt/splunk/bin/splunk start 

It does take some time to start the first time, as it needs to generate certain cryptographic keys. Once it starts, you use the tool by pointing your browser to your local host, on port 8000. The first time you do so, you will be presented with a screen like the following.

Screenshot-Login - Splunk - Mozilla Firefox

Be sure to change the password appropriately when prompted.

If you want to use a browser from other than the host, you need to open the firewall. The decision to open the firewall to allow connections to the Splunk server is a trade-off between security and convenience. Hey- I wonder if the data passing to and from the server is encrypted. What do you think?

Splunk launches with a default license valid for 60 days and up to 500 MB of data per day. It also comes with a free license that allows it to be used provided certain conditions are met. You must accept the license agreement the first time you run the tool. To change the license (after the system is running) from the trial license to the free license, first start Splunk, and visit the Splunk web page. Navigate Manager → License (near the bottom third of the page).

Screenshot-Splunk Manager - Splunk 5.0.2 - Mozilla Firefox

From the resulting page, select Change License Group, then choose the Free License. You will need to restart Splunk. One thing though- once you change to the Free License, you no longer need to authenticate to gain access to the Administrator page, so if you opened up your firewall to allow remote access to the Splunk web page, you may be in for a real treat. You may want to read these Splunk comments.

Begin by visiting the Getting Started Tutorial, and read the pages for the various tabs.

Screenshot-(getstarted) - Getting started - Splunk 5.0.2 - Mozilla Firefox

Let’s start by collecting some data about the host on which we installed Splunk. Following the instructions in the tutorial, navigate Manager → Data inputs → Files & Directories on the web interface.

Screenshot-Splunk Manager - Splunk 5.0.2 - Mozilla Firefox-1

Since this is a Mint system, let’s add the file /var/log/syslog; if this were CentOS we could use /var/log/messages instead. Since this is a core log file, we will ask Splunk to continuously index the file.

It may take a few moments to a few minutes for the file to be completely indexed.

When the process is complete though, you can search through the logged data using a Google like interface. Indeed, suppose we want to know whenever the term "seldon" appears in the logs. From the "app" menu in the top right of the web interface select Search. Then enter your search term(s) into the box, and his return. You obtain a result like the following

Screenshot-Search - Search - Splunk 5.0.2 - Mozilla Firefox

Here we see that "seldon" appeared twice, both in su commands at different times.

The real power of this search capability becomes more evident as we add more data sources to our Splunk system and successfully collect data from multiple systems on the same server.

Splunk Apps

Splunk can be extended with a number of apps. One valuable app for us is an app designed specifically to monitor the health and status of Unix and Linux hosts. If you are running your guest on a network with an Internet connection, you can find it and install it by navigating the Splunk Manager web page, App → Find more apps, then select Splunk for Unix and Linux.

In the classroom laboratory, select the App menu item from Splunk
Manager, then Manage Apps. On that page, select Install App from File and provide it with the app. It is called unix.tar.gz. Though these apps come as packaged .tar.gz files and do not need to be uncompressed.

Screenshot-Splunk Manager - Splunk 5.0.2 - Mozilla Firefox

The installation of a Splunk app will require a Splunk restart; this can be done from within the Splunk Manager.

Mint 11- Nexon-2013-02-09-20-56-34

Once Splunk is restarted, navigate Splunk Manager web page → app → *nix 4.6. From there, you can configure the data sources that you want the app to use.

Consider carefully the data sources- too many may not be helpful. However, once can clearly see the security value in knowing the various bash histories, or the collection of ports that were open on the system.

Experiment with the data available on your server, both from the log data source you originally specified as well as from the *nix app. Here is a sample search through the logs to see where the term "Google" appears; you can see it in seldon’s bash history; this was created during one of the system tests to check the networking
UnixSplunk

Starting Splunk on Boot

If you want splunk to automatically start on boot, run

seldon@nexon /var/log $ sudo /opt/splunk/bin/splunk enable boot-start
[sudo] password for seldon: 
 Adding system startup for /etc/init.d/splunk ...
   /etc/rc0.d/K20splunk -> ../init.d/splunk
   /etc/rc1.d/K20splunk -> ../init.d/splunk
   /etc/rc6.d/K20splunk -> ../init.d/splunk
   /etc/rc2.d/S20splunk -> ../init.d/splunk
   /etc/rc3.d/S20splunk -> ../init.d/splunk
   /etc/rc4.d/S20splunk -> ../init.d/splunk
   /etc/rc5.d/S20splunk -> ../init.d/splunk
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.
Collecting Data from Multiple Machines

Splunk can also be used to collect data from multiple machines and present all of the data in a coherent whole. To do so, we need to set up Splunk Forwarders (which send the data to the central server(s)) and Receivers (the central location(s) that record the results).

To set up a receiver, visit the Splunk Manager web page for that machine, and select Manager from the menu at the top right. Select Forwarding and Receiving, then Configure receiving.
Splunk Receiver
The receiver can listen on any unused TCP port, though TCP/9997 is the canonical choice. Choose your port(s) and select Save.

You then need to restart Splunk (Splunk Manager web page → Manager → Server controls → Restart Splunk) for the change(s) to take effect. This is a good time to ensure that the Firewall allows inbound traffic on the selected port(s).

On the forwarder, configure Splunk in essentially the same fashion as before; in particular be sure to select the data sources that you wish Splunk to collect. Then navigate Splunk Manager web page → Manager → Forwarding and Receiving → Configure Forwarding. [There is an option to use “Light Forwarding” where the full Splunk server is not running on the forwarder; we do not have time to cover this option.]
Be sure that you select the proper App context; you may wish to select "All".

Screenshot-Splunk Manager - Splunk 5.0.2 - Mozilla Firefox

Select New, the specify the host IP and listening port of the Splunk receiver. Save; then restart Splunk.

Screenshot-Splunk Manager - Splunk 5.0.2 - Mozilla Firefox-1

Test out the result, and notice that you can now see the logs from both machines from one location.

Disk Space, Again

Don’t use a CentOS system as your receiver- the small space allotted to the root partition will cause you no end of trouble. However, CentOS appears fine as a forwarder, and the Mint systems fine as receiver and/or forwarder.

SSH

Starting the SSH Server.

On CentOS, the SSH server is already running and already allowed through the firewall; it is running version 5.3p1.

On Ubuntu systems (Desktop or Server), the server is not present. For a Desktop system, grab the package openssh-server_5.3p1-1ubuntu7_i386.deb from the labshare or online. Install it in the usual way:

seldon@capella:~$ sudo dpkg -i Desktop/openssh-server_5.3p1-3ubuntu7_i386.deb 
(Reading database ... 124588 files and directories currently installed.)
Preparing to replace openssh-server 1:5.3p1-3ubuntu3 (using .../openssh-server_
5.3p1-3ubuntu7_i386.deb) ...
Unpacking replacement openssh-server ...
Setting up openssh-server (1:5.3p1-3ubuntu7) ...
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
ssh start/running, process 4684

Processing triggers for ureadahead ...
Processing triggers for ufw ...
Processing triggers for man-db ...

As noted earlier, the Ubuntu firewall is off by default; if you have enabled it, then you must open the necessary port.

Getting the SSH server onto the Ubuntu Server system is complicated by the fact that it has no GUI, so the usual trick of dragging a file from the host and dropping it on the desktop of the guest will not allow us to copy the files. We could set up a web server or an ftp server, and use client software like wget on the server to get the files. We could also take the server out out of the lab to an Internet connection and use ftp and/or wget.

There is another approach. From VMWare, select VM, then Settings. Under the Options tab, select Shared Folders. Select Add; then you can choose a folder on the host that will be mapped to a folder on the guest.

HGFS

In this example, I mapped the host directory F:\Software to the directory "Software". That directory will now appear in the guest, in the location /mnt/hgfs/Software

[Did he just say "hgfs"? Yep. Host-Guest-File-System. hgfs. It does make sense. Sort of.]

We can grab the files from the guest on the Ubuntu server by copying them directly from their location on the host. Watch out though- you don’t have write permissions to /mnt/hgfs or any of its subdirectories, unless you specifically granted that permission when you created the share. Here we copy everything back to /home/seldon.
Ubuntu 10.04 Server- Hesperus-2013-02-09-22-54-13

To actually install the SSH server, we need two packages- the server itself (openssh-server_5.3p1-3ubuntu7_i386.deb) and the package libwrap0_7.6.q-18_i386.deb. Install them both via dpkg.

Ubuntu 10.04 Server- Hesperus-2013-02-09-22-56-16

Mint 11 also does not come with an SSH server by default. It is handled in the same way as Ubuntu Desktop, but now the package is openssh-server_5.8p1-1ubuntu3_i386.deb.

seldon@nexon ~ $ sudo dpkg -i Desktop/openssh-server_5.8p1-1ubuntu3_i386.deb 
[sudo] password for seldon: 
Selecting previously deselected package openssh-server.
(Reading database ... 144835 files and directories currently installed.)
Unpacking openssh-server (from .../openssh-server_5.8p1-1ubuntu3_i386.deb) ...
Setting up openssh-server (1:5.8p1-1ubuntu3) ...
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
ssh start/running, process 23609
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Processing triggers for ufw ...
Processing triggers for man-db ...

SSH PKI
SSH can use public keys to authenticate users; this is often more convenient than passwords. To set up the PKI in CentOS 6.2, modify /etc/ssh/sshd_config to uncomment out the following lines

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

The situation in Mint is similar, though it writes the location of the Authorized Keys File as

#AuthorizedKeysFile	%h/.ssh/authorized_keys

Restart sshd; on CentOS 6.2 you can use

[root@proclas ~]# service sshd restart
Stopping sshd:                                             [  OK  ]
Starting sshd:                                             [  OK  ]

while on Mint 11 & Ubuntu 11.04 you can use

seldon@nexon ~ $ sudo service ssh restart
ssh start/running, process 3805

Each user that wants to log in to this server using SSH PKI needs to set the proper permissions on the ~/.ssh directory, create the file ~/.ssh/authorized_keys, and ensure that file also has the proper permissions. Indeed, we require

  • ~/.ssh is chmod 700 (rwx — —)
  • ~/.ssh/authorized_keys is chmod 644 (rw- r– r–)

We can do this manually now, but there is a method to do this at the same time the keys are copied to the server.

Now each user on each client system that will connect to the server must first generate the keys that will be used for the connection. Here is the user seldon on the host proclas.cosc.tu generating the SSH keys

seldon@proclas ~]$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/seldon/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/seldon/.ssh/id_rsa.
Your public key has been saved in /home/seldon/.ssh/id_rsa.pub.
The key fingerprint is:
96:b6:c8:db:72:62:13:c1:8d:68:8a:e0:1c:c1:5f:3a seldon@proclas.cosc.tu
The key's randomart image is:
+--[ RSA 2048]----+
|.                |
| o   .           |
|  o oo o         |
|.. Eo + ..       |
|+..o.  .S        |
|.o.  ..+ .       |
|      o..        |
|      =o.        |
|     ..=.        |
+-----------------+

Note that seldon did not use a passphrase to secure the keys.

What results are a pair of keys, a private key and a public key, stored in the directory /home/seldon/.ssh:

[seldon@proclas ~]$ ls -l .ssh
total 8
-rw-------. 1 seldon seldon 1675 Feb 10 07:00 id_rsa
-rw-r--r--. 1 seldon seldon  404 Feb 10 07:00 id_rsa.pub

To use SSH PKI, we need to copy the public key (ONLY) to the server; the contant of that key needs to then be appended to the file ~/.ssh/authorized_keys on the server. We can do this manually a number of ways, including using SSH with a password. However, there is a tool that will do this automatically for us, and will even ensure that the server’s directories and files have the correct permissions.

[seldon@proclas ~]$ ssh-copy-id seldon@nexon.cosc.tu
seldon@nexon.cosc.tu's password: 

Once this is complete, seldon on proclas can SSH into nexon (as seldon) without using a password.

Note that the user names do not have to agree! Suppose that the root user on proclas copied their public keys to the authorized key file for the user seldon on nexon

[root@proclas ~]# ssh-copy-id seldon@nexon.cosc.tu
seldon@nexon.cosc.tu's password:

Then the root user can SSH as seldon on nexon without a password

[root@proclas ~]# ssh seldon@nexon
Welcome to Linux Mint 11 Katya (GNU/Linux 2.6.38-8-generic i686)

Though this example used the root user, any user (A) on a client (C) can connect as any other user (B) on a server (S) provided the public key for (A) is saved in the authorized_keys file for the user (B) on the host (S).

You can do a lot of surprising thing with SSH. Suppose you want to just execute a single command on a remote system- say cat /etc/hostname. Well, run it as follows:

[seldon@proclas ~]$ ssh nexon cat /etc/hostname
nexon.cosc.tu

Since PKI has been setup already, no password was required, and the result of the command is simply dropped to our screen. If we had not set up PKI, then we would need to enter our password each time we wanted to execute a command.

Suppose you wanted to copy the file /home/seldon/text to the remote host. Then you can use scp, which used ssh as its transport mechanism:

[seldon@proclas Documents]$ scp text seldon@nexon:/home/seldon/copied_text
text                                          100%   20     0.0KB/s   00:00 

We can bring that file back just as easily

[seldon@proclas Documents]$ scp seldon@nexon:/home/seldon/copied_text ./copy
copy                                          100%   20     0.0KB/s   00:00    
[seldon@proclas Documents]$ ls -l
total 56
-rw-r--r--. 1 seldon seldon   20 Feb 10 07:23 copy
-rw-rw-r--. 1 seldon seldon   20 Feb 10 07:21 text
[seldon@proclas ~]$ 

Suppose that there is a graphical program like firefox on your remote host. You can run it remotely, provided you pass the "-Y" flag. Here is a screenshot of the user seldon on the CentOS system proclas calling Firefox from the Mint 11 system nexon.
CentOS 64-bit- Proclas (Logs)-2013-02-10-10-30-09

SSH can also be used to create tunnels, but we don’t have time to discuss SSH tunneling in the detail it deserves.

Securing SSH
There are a number of things that we can do to try to secure our SSH server. First, we can prevent root from logging in directly. If someone needs root privileges from a remote site, they can log in as a regular user and then use either sudo or su. To make this change, add the line

PermitRootLogin no

to the sshd_config file. Note that the default state is to allow root logins; in the CentOs 6.2 default configuration file a commented out line allowing root logins is present, while on the Mint 11 / Ubuntu 10.04 default configuration files the line is present and explicitly allows root logins.

Speaking of users, you may want to allow only certain users to be allowed to use SSH to log in to the box. The line

AllowUsers seldon hardin mallow

will allow only the listed users to be able to SSH into the machine; other users (e.g.riose) would not be able to log in. Though using a white list of allowed users is generally preferred, you could create a black list of users not allowed to log in via the DenyUsers directive.

Be sure to use only SSH protocol 2, as the older SSH-1 has a number of vulnerabilities and should be considered obsolete. Though SSH-2 is the default, your sshd_config file should contain the line

Protocol 2

to ensure that the system does not fall back to SSH-1.

You can also use the files /etc/hosts.allow and /etc/hosts.deny to restrict access. There is a three step process-

  • If a service / host combination is in the file hosts.allow, then access to the service is granted
  • If the combination is not in hosts.allow, then hosts.deny is checked; if the service / host combination matches then access is denied
  • If neither has occurred, then access is granted

Each line has the form

 service : host(s)

For our purpose the service can be either sshd or ALL. The host(s) can be specified by name, by IP address, or by IP Address Range. Multiple hosts can be separated by commas. Suppose we want to allow SSH from only the hosts 10.0.2.3 and 10.0.2.13; then we can set up the hosts.allow file as

# /etc/hosts.allow
sshd : 10.0.2.3, 10.0.2.13

while the hosts.deny file becomes

# /etc/hosts.deny
sshd: ALL

Then any ssh connection attempt from other than 10.0.2.3 or 10.0.2.13 will not even receive a response.

You are encouraged to read the manual page for hosts_access to learn more about the syntax for these files, and to learn other options.

Situational Awareness

By running a service like SSH, you are now explicitly allowing other people to access your system. To do so safely, you need to know what those users are doing on your system; this is the concept of situational awareness.

Linux provides a number of tools to give you, the system administrator, information about what is occurring on your system.

psacct

There is a package that allows for much more detailed logging of what is being done on a Linux system called psacct or acct depending on the particular distribution

The tool is already installed on your CentOS virtual machine, though it is not enabled. Called psacct it is controlled in the usual fashion. The command

[root@proclas ~]# /etc/init.d/psacct start
Starting process accounting:                               [  OK  ]

will start and stop the service, and we can ensure it starts on boot either via chkconfig or by navigating System → Administration → Services.
Screenshot-Service Configuration

For Ubuntu, the package is called acct, and for Ubuntu 10.04 we use version 6.5.1-1ubuntu1. You can get the software from the labshare or directly from the Ubuntu software repository.
Install it in the usual fashion.

seldon@capella:~$ sudo dpkg -i Desktop/acct_6.5.1-1ubuntu1_i386.deb 
[sudo] password for seldon: 
Selecting previously deselected package acct.
(Reading database ... 124588 files and directories currently installed.)
Unpacking acct (from .../acct_6.5.1-1ubuntu1_i386.deb) ...
Setting up acct (6.5.1-1ubuntu1) ...
Starting process accounting: Turning on process accounting, file set to '/var/log/account/pacct'.
acct.

Processing triggers for doc-base ...
Processing 1 added doc-base file(s)...
Registering documents with scrollkeeper...
Processing triggers for install-info ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot

For Mint the package is also named acct, though now the appropriate version in 6.5.4-2.

Ubuntu and Mint set up the package so that it will start and will start on the system’s next boot.

Using psacct

The psacct package keeps track of a number of pieces of system information. For example, the command ac can be used to determine how may hours of connect time have been used.

[root@proclas ~]# ac
	total       34.87

This shows that users have been connected to the system for 114.07 hours. If we want to see that number broken down by user, we can run

[root@proclas ~]# ac -p
	seldon                              34.18
	Bill                                 0.04
	vimes                                0.65
	total       34.86

to see that essentially all of the time has been from the user seldon, with the other users contributing essentially nothing to the total. We can use ac -d

[root@proclas ~]# ac -d
Jan  1	total        0.28
Jan  2	total        0.04
Jan 28	total        0.10
Jan  5	total        0.29
Jan 17	total        1.78
Jan 28	total        0.04
Feb  9	total       26.16
Today	total        6.19

to get a breakdown by date; we can even combine them as ac -dp

[root@proclas ~]# ac -dp
	vimes                                0.28
Jan  1	total        0.28
	vimes                                0.04
Jan  2	total        0.04
	vimes                                0.10
Jan 28	total        0.10
	seldon                               0.03
	Bill                                 0.04
	vimes                                0.23
Jan  5	total        0.29
	seldon                               1.78
Jan 17	total        1.78
	seldon                               0.04
Jan 28	total        0.04
	seldon                              26.16
Feb  9	total       26.16
	seldon                               6.20
Today	total        6.20

Another useful feature of this package is the ability to determine the commands that have been run on the system. For example, to find out each time the command su has been run, you can execute

[root@proclas ~]# lastcomm su
su                S     seldon   pts/0      0.12 secs Sun Feb 10 09:43
su                S     seldon   pts/0      0.17 secs Sun Feb 10 09:00

From this we see that the su command has been run only twice since accounting started. We can also sort by user. Running

[root@proclas ~]# lastcomm seldon
gnome-screensav         seldon   __         0.00 secs Sun Feb 10 09:44
sleep                   seldon   __         0.00 secs Sun Feb 10 09:43
xauth             S     seldon   pts/0      0.00 secs Sun Feb 10 09:43
su                S     seldon   pts/0      0.12 secs Sun Feb 10 09:43
gnome-screensav         seldon   __         0.00 secs Sun Feb 10 09:43
sleep                   seldon   __         0.00 secs Sun Feb 10 09:42
xauth             S     seldon   pts/0      0.00 secs Sun Feb 10 09:43
su                S     seldon   pts/0      0.17 secs Sun Feb 10 09:00
gnome-screensav         seldon   __         0.00 secs Sun Feb 10 09:42
sleep                   seldon   __         0.00 secs Sun Feb 10 09:41

we find the command recently executed by the user seldon; you can even see the two su commands we saw in the output of lastcomm. You can read the man page for lastcomm to determine the significance of the flags (e.g. the S that occasionally appears.)

If you want to find out information about previously run commands, you can use the sa tool; it is able to give you information on the cpu usage of all of the commands run on the system. See the man page for more details.

Who is Logged on the System?

The command w can be used to determine who is logged into the system. For example, here is the output of that command on our SSH server

seldon@nexon ~ $ w 
 13:07:05 up 26 min,  5 users,  load average: 0.00, 0.04, 0.14
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
seldon   tty7     :0               12:41   26:19   3.26s  0.24s gnome-session -
seldon   pts/0    proclas.cosc.tu  13:00    6:25   0.92s  0.06s sshd: seldon [p
mallow   pts/1    capella.cosc.tu  13:03    3:59   0.33s  0.33s -bash
seldon   pts/2    :0               13:03    0.00s  0.16s  0.00s w
hardin   pts/3    aurora.cosc.tu   13:04    2:49   0.49s  0.49s -bash

Here we see three users logged in (seldon, mallow, hardin) with three SSH logins from three different hosts.

Similar information is provided by the command who:

seldon@nexon ~ $ who
seldon   tty7         2013-02-10 12:41 (:0)
seldon   pts/0        2013-02-10 13:00 (proclas.cosc.tu)
mallow   pts/1        2013-02-10 13:03 (capella.cosc.tu)
seldon   pts/2        2013-02-10 13:03 (:0)
hardin   pts/3        2013-02-10 13:04 (aurora.cosc.tu)

The data these commands rely on is stored in the file /var/run/utmp.

You may also want to know not who is currently logged in, but rather the users that have logged in recently. This information is available through the command last

seldon@nexon ~ $ last
hardin   pts/3        aurora.cosc.tu   Sun Feb 10 13:04   still logged in   
seldon   pts/2        :0               Sun Feb 10 13:03   still logged in   
mallow   pts/1        capella.cosc.tu  Sun Feb 10 13:03   still logged in   
seldon   pts/0        proclas.cosc.tu  Sun Feb 10 13:00   still logged in   
seldon   tty7         :0               Sun Feb 10 12:41   still logged in   
reboot   system boot  2.6.38-8-generic Sun Feb 10 12:41 - 13:11  (00:30)    
seldon   pts/1        proclas.cosc.tu  Sun Feb 10 11:44 - 11:54  (00:10)    
seldon   pts/1        proclas.cosc.tu  Sun Feb 10 10:26 - 10:27  (00:00)    
seldon   pts/1        proclas.cosc.tu  Sun Feb 10 10:08 - 10:08  (00:00)    
seldon   pts/1        proclas.cosc.tu  Sun Feb 10 10:08 - 10:08  (00:00)    
seldon   pts/1        proclas.cosc.tu  Sun Feb 10 10:07 - 10:07  (00:00)    
seldon   pts/1        proclas.cosc.tu  Sun Feb 10 10:06 - 10:06  (00:00)    
seldon   pts/0        :0               Sun Feb 10 09:37 - down   (03:03)    
seldon   tty7         :0               Sun Feb 10 09:31 - down   (03:09)    
reboot   system boot  2.6.38-8-generic Sun Feb 10 09:30 - 12:40  (03:09)    
seldon   pts/0        :0               Sat Feb  9 10:27 - 23:08  (12:40)    

wtmp begins Sat Feb  9 10:27:55 2013

This data is stored in the file /var/log/wtmp, which is why attackers often clobber that file before departing a system.

Remember that is commands are executed via a bash shell, then the list of past commands for a user is stored in the file ~/.bash_history. Thus, if you want to see the last 15 bash commands executed by the user seldon, you can simply check the list

seldon@nexon ~ $ tail -n 15 /home/seldon/.bash_history 
ifconfig
ssh 10.0.2.13
cd
sudo dpkg -i Desktop/openssh-server_5.8p1-1ubuntu3_i386.deb 
firefox &
ssh localhost
gedit /etc/ssh/sshd_config &
sudo ls
sudo gedit /etc/ssh/sshd_config &
service sshd reload
sudo service ssh reload
sudo service ssh restart
man hosts_access
sudo dpkg -i Desktop/acct_6.5.4-2.1ubuntu1_i386.deb 
sudo reboot
What Processes are Running on the System?

If you want a real-time list of processes running on your system you can use the command top; here is the output of that program at a particular moment.

top - 13:20:39 up 39 min,  5 users,  load average: 0.09, 0.09, 0.10
Tasks: 148 total,   1 running, 146 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.7%us,  1.0%sy,  0.0%ni, 98.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1024776k total,   674588k used,   350188k free,    37112k buffers
Swap:  1046524k total,        0k used,  1046524k free,   369544k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 1645 root      20   0 81180  35m 8940 S  1.0  3.5   0:05.56 Xorg               
 5608 seldon    20   0 90584  13m  10m S  0.7  1.4   0:02.24 gnome-terminal     
 1778 root      20   0 94796  33m 9.8m S  0.3  3.3   0:28.49 splunkd            
 1805 seldon    20   0 50896  14m  12m S  0.3  1.5   0:06.04 vmtoolsd           
 7766 seldon    20   0  2632 1152  864 R  0.3  0.1   0:00.04 top                
    1 root      20   0  3044 1844 1260 S  0.0  0.2   0:01.51 init               
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd           
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.14 ksoftirqd/0        
    5 root      20   0     0    0    0 S  0.0  0.0   0:00.50 kworker/u:0        
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0        
    7 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 cpuset             
    8 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 khelper            
    9 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 netns              
   10 root      20   0     0    0    0 S  0.0  0.0   0:00.00 sync_supers        
   11 root      20   0     0    0    0 S  0.0  0.0   0:00.00 bdi-default  

The processes are listed in order, with the processes using most CPU listed at the top. If your system is slow or sluggish due to heavy load, this is the place to start diagnosing the problem.

If you want to know more about what your processes are up to, then you want to use the ps command. This is full of command switches that can give you some very different behaviors. If you want to know all of the processes running on a system, run

seldon@nexon ~ $ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1   3044  1844 ?        Ss   12:40   0:01 /sbin/init
root         2  0.0  0.0      0     0 ?        S    12:40   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    12:40   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S    12:40   0:00 [kworker/u:0]
root         6  0.0  0.0      0     0 ?        S    12:40   0:00 [migration/0]
root         7  0.0  0.0      0     0 ?        S<   12:40   0:00 [cpuset]
root         8  0.0  0.0      0     0 ?        S<   12:40   0:00 [khelper]
root         9  0.0  0.0      0     0 ?        S<   12:40   0:00 [netns]
root        10  0.0  0.0      0     0 ?        S    12:40   0:00 [sync_supers]
root        11  0.0  0.0      0     0 ?        S    12:40   0:00 [bdi-default]
root        12  0.0  0.0      0     0 ?        S<   12:40   0:00 [kintegrityd]
root        13  0.0  0.0      0     0 ?        S<   12:40   0:00 [kblockd]
root        14  0.0  0.0      0     0 ?        S<   12:40   0:00 [kacpid]
root        15  0.0  0.0      0     0 ?        S<   12:40   0:00 [kacpi_notify]
root        16  0.0  0.0      0     0 ?        S<   12:40   0:00 [kacpi_hotplug]
root        17  0.0  0.0      0     0 ?        S<   12:40   0:00 [ata_sff]
root        18  0.0  0.0      0     0 ?        S    12:40   0:00 [khubd]
root        19  0.0  0.0      0     0 ?        S<   12:40   0:00 [md]
root        23  0.0  0.0      0     0 ?        S    12:40   0:00 [khungtaskd]
root        24  0.0  0.0      0     0 ?        S    12:40   0:00 [kswapd0]
root        25  0.0  0.0      0     0 ?        SN   12:40   0:00 [ksmd]
root        26  0.0  0.0      0     0 ?        S    12:40   0:00 [fsnotify_mark]
root        27  0.0  0.0      0     0 ?        S<   12:40   0:00 [aio]
root        28  0.0  0.0      0     0 ?        S    12:40   0:00 [ecryptfs-kthre]
root        29  0.0  0.0      0     0 ?        S<   12:40   0:00 [crypto]
root        33  0.0  0.0      0     0 ?        S<   12:40   0:00 [kthrotld]
root        35  0.0  0.0      0     0 ?        S    12:40   0:00 [scsi_eh_0]
root        36  0.0  0.0      0     0 ?        S    12:40   0:00 [scsi_eh_1]
root        37  0.0  0.0      0     0 ?        S    12:40   0:00 [kworker/u:3]
root        40  0.0  0.0      0     0 ?        S<   12:40   0:00 [kmpathd]
root        41  0.0  0.0      0     0 ?        S<   12:40   0:00 [kmpath_handler]
root        42  0.0  0.0      0     0 ?        S<   12:40   0:00 [kondemand]
root        43  0.0  0.0      0     0 ?        S<   12:40   0:00 [kconservative]
root       224  0.0  0.0      0     0 ?        S<   12:40   0:00 [mpt_poll_0]
root       225  0.0  0.0      0     0 ?        S<   12:40   0:00 [mpt/0]
root       266  0.0  0.0      0     0 ?        S    12:40   0:00 [scsi_eh_2]
root       347  0.0  0.0      0     0 ?        S    12:40   0:01 [jbd2/sda1-8]
root       348  0.0  0.0      0     0 ?        S<   12:40   0:00 [ext4-dio-unwri]
root       398  0.0  0.0   2412   908 ?        S    12:40   0:00 upstart-udev-bri
root       444  0.0  0.1   2984  1248 ?        S<s  12:40   0:00 udevd --daemon
root       543  0.0  0.1   3140  1296 ?        S<   12:40   0:00 udevd --daemon
root       544  0.0  0.1   3140  1240 ?        S<   12:40   0:00 udevd --daemon
root       683  0.0  0.0      0     0 ?        S<   12:40   0:00 [vmmemctl]
root       707  0.0  0.0      0     0 ?        S<   12:40   0:00 [kpsmoused]
root       737  0.0  0.2   5652  2256 ?        Ss   12:40   0:00 /usr/sbin/sshd -
root       762  0.0  0.0   2412   604 ?        S    12:40   0:00 upstart-socket-b
root       770  0.0  0.4  16628  4180 ?        Ss   12:40   0:00 smbd -F
syslog     776  0.0  0.1  28016  1840 ?        Sl   12:40   0:00 rsyslogd -c4
102        800  0.0  0.1   3612  1884 ?        Ss   12:40   0:00 dbus-daemon --sy
avahi      833  0.0  0.1   3016  1448 ?        S    12:40   0:00 avahi-daemon: ru
avahi      834  0.0  0.0   3016   432 ?        S    12:40   0:00 avahi-daemon: ch
root       839  0.0  0.4  17636  4576 ?        Ssl  12:40   0:00 NetworkManager
root       841  0.0  0.2   4660  2500 ?        S    12:40   0:00 /usr/sbin/modem-
root       844  0.0  0.3  24892  3892 ?        Sl   12:40   0:00 /usr/lib/policyk
root       849  0.0  0.1   5172  1764 ?        S    12:40   0:00 /sbin/wpa_suppli
root       870  0.0  0.2   6924  2956 ?        Ss   12:40   0:00 /usr/sbin/cupsd
root       885  0.0  0.1  16628  1320 ?        S    12:40   0:00 smbd -F
root       961  0.0  0.0      0     0 ?        S    12:40   0:00 [flush-8:0]
root      1307  0.0  0.1   9420  1720 ?        Ss   12:40   0:00 nmbd -D
root      1389  0.0  0.0  30472   572 ?        Ssl  12:41   0:00 /usr/sbin/vmware
root      1427  0.1  0.3  25312  3696 ?        S    12:41   0:03 /usr/sbin/vmtool
root      1454  0.0  0.0   1872   580 tty4     Ss+  12:41   0:00 /sbin/getty -8 3
root      1459  0.0  0.0   1872   580 tty5     Ss+  12:41   0:00 /sbin/getty -8 3
root      1471  0.0  0.0   1872   576 tty2     Ss+  12:41   0:00 /sbin/getty -8 3
root      1472  0.0  0.0   1872   576 tty3     Ss+  12:41   0:00 /sbin/getty -8 3
root      1474  0.0  0.0   1872   580 tty6     Ss+  12:41   0:00 /sbin/getty -8 3
root      1477  0.0  0.0   1864   676 ?        Ss   12:41   0:00 acpid -c /etc/ac
root      1484  0.0  0.0   2268   856 ?        Ss   12:41   0:00 cron
daemon    1485  0.0  0.0   2132   348 ?        Ss   12:41   0:00 atd
root      1529  0.0  0.1  12596  1512 ?        Ss   12:41   0:00 tpvmlpd2
root      1537  0.0  0.3  18388  3704 ?        Ssl  12:41   0:00 gdm-binary
root      1543  0.0  0.3  26168  3420 ?        Sl   12:41   0:00 /usr/sbin/consol
root      1634  0.0  0.3  19716  3760 ?        Sl   12:41   0:00 /usr/lib/gdm/gdm
root      1645  0.3  3.5  46760 36436 tty7     Ss+  12:41   0:10 /usr/bin/X :0 -n
root      1657  0.0  0.3  20184  3680 ?        Sl   12:41   0:00 /usr/lib/gdm/gdm
seldon    1666  0.0  0.7  35328  7240 ?        Ssl  12:41   0:00 gnome-session --
seldon    1699  0.0  0.0   3368   192 ?        Ss   12:41   0:00 /usr/bin/ssh-age
seldon    1702  0.0  0.0   3456   572 ?        S    12:41   0:00 /usr/bin/dbus-la
seldon    1703  0.0  0.1   4372  1644 ?        Ss   12:41   0:00 //bin/dbus-daemo
seldon    1708  0.0  0.4   9568  4332 ?        S    12:41   0:00 /usr/lib/libgcon
seldon    1714  0.0  1.1  99780 11492 ?        Ssl  12:41   0:00 /usr/lib/gnome-s
seldon    1716  0.0  0.3  42180  3244 ?        Sl   12:41   0:00 /usr/bin/gnome-k
seldon    1724  0.0  0.2   7632  2240 ?        S    12:41   0:00 /usr/lib/gvfs/gv
seldon    1729  0.0  0.2  31084  2496 ?        Ssl  12:41   0:00 /usr/lib/gvfs//g
seldon    1733  0.0  1.0 107476 11248 ?        Sl   12:41   0:00 metacity
seldon    1738  0.0  0.4  95728  4952 ?        S<sl 12:41   0:00 /usr/bin/pulseau
rtkit     1740  0.0  0.1  18888  1212 ?        SNl  12:41   0:00 /usr/lib/rtkit/r
seldon    1754  0.0  1.8  95408 19372 ?        Sl   12:41   0:01 nautilus
seldon    1756  0.0  0.8  45072  8328 ?        Sl   12:41   0:00 bluetooth-applet
seldon    1757  0.0  0.3  20788  3336 ?        Sl   12:41   0:00 /usr/lib/pulseau
seldon    1759  0.0  0.4  49080  4632 ?        Sl   12:41   0:00 zeitgeist-datahu
seldon    1760  0.0  0.3   9568  3680 ?        S    12:41   0:00 /usr/bin/python
seldon    1762  0.0  1.2 107388 12364 ?        Sl   12:41   0:00 gnome-panel
seldon    1767  0.0  0.0   1912   512 ?        S    12:41   0:00 sh -c /usr/lib/l
seldon    1768  0.0  2.2 110856 23312 ?        Sl   12:41   0:00 python /usr/lib/
root      1778  1.1  3.3  94796 33976 ?        Sl   12:41   0:35 splunkd -p 8089
seldon    1784  0.0  0.8  52948  8352 ?        Sl   12:41   0:00 gnome-power-mana
root      1785  0.0  0.1  25936  2040 ?        Ss   12:41   0:00 [splunkd pid=177
seldon    1793  0.0  1.2  31476 13020 ?        Sl   12:41   0:00 /usr/bin/python
seldon    1804  0.0  1.1 103336 11488 ?        Sl   12:41   0:00 /usr/lib/policyk
seldon    1805  0.2  1.4  50896 15328 ?        S    12:41   0:07 /usr/lib/vmware-
seldon    1831  0.0  1.2 122180 13196 ?        Sl   12:41   0:00 nm-applet --sm-d
seldon    1858  0.0  0.6  27088  6940 ?        Sl   12:41   0:00 /usr/lib/notify-
seldon    1860  0.0  0.3   9040  3324 ?        S    12:41   0:00 /usr/lib/gvfs/gv
root      1863  0.0  0.3  14020  3296 ?        Sl   12:41   0:00 /usr/lib/udisks/
root      1865  0.0  0.3  16720  3648 ?        Sl   12:41   0:00 /usr/lib/upower/
root      1872  0.0  0.0   5564   724 ?        S    12:41   0:00 udisks-daemon: p
seldon    1886  0.0  0.2   8512  2256 ?        S    12:41   0:00 /usr/lib/gvfs/gv
seldon    1891  0.0  0.0   3912   244 ?        S    12:41   0:00 /bin/cat
seldon    1895  0.0  0.3  50552  3600 ?        Ssl  12:41   0:00 /usr/lib/bonobo-
seldon    1898  0.0  0.2  18204  2252 ?        Sl   12:41   0:00 /usr/lib/gvfs/gv
seldon    1901  0.0  0.3  27624  3236 ?        Ss   12:41   0:00 gnome-screensave
seldon    1922  0.0  0.3   8212  3212 ?        S    12:41   0:00 /usr/lib/gvfs/gv
seldon    1945  0.0  0.0      0     0 ?        Z    12:41   0:00 [zeit] 
seldon    1960  0.0  2.9 102388 30496 ?        Sl   12:41   0:01 python /usr/lib/
seldon    1962  0.0  1.2  80140 12404 ?        Sl   12:41   0:00 /usr/lib/gnome-p
seldon    1964  0.0  1.2  86604 13308 ?        Sl   12:41   0:00 /usr/lib/gnome-p
seldon    1966  0.0  1.1  95112 11564 ?        Sl   12:41   0:00 /usr/lib/indicat
seldon    1968  0.0  0.8  76112  8836 ?        Sl   12:41   0:00 /usr/lib/gnome-p
seldon    2006  0.0  0.6  19672  6656 ?        S    12:41   0:00 /usr/lib/gnome-d
seldon    2008  0.0  0.1   7440  1984 ?        S    12:41   0:00 /usr/lib/gvfs/gv
seldon    2027  0.0  0.2   7764  2496 ?        S    12:41   0:00 /usr/lib/gvfs/gv
root      2039  0.2  1.9  89036 19740 ?        Sl   12:41   0:08 python -O /opt/s
seldon    2055  0.0  0.4  47672  4120 ?        Sl   12:41   0:00 /usr/lib/indicat
seldon    2056  0.0  0.5 112276  5812 ?        Sl   12:41   0:00 /usr/lib/indicat
root      2413  0.0  0.0   1872   576 tty1     Ss+  12:41   0:00 /sbin/getty -8 3
seldon    2419  0.0  1.6  33680 16884 ?        S    12:41   0:00 /usr/bin/python
root      4155  0.0  0.3  10876  3580 ?        Ss   13:00   0:00 sshd: seldon [pr
seldon    4242  0.0  0.1  10876  1640 ?        S    13:00   0:00 sshd: seldon@pts
seldon    4243  0.0  0.6   9736  6416 pts/0    Ss   13:00   0:00 -bash
root      4347  0.0  0.1   7268  1988 pts/0    S    13:00   0:00 sudo su -
root      4349  0.0  0.1   7168  1784 pts/0    S    13:00   0:00 su -
root      4356  0.0  0.6   9732  6356 pts/0    S+   13:00   0:00 -su
root      4555  0.0  0.2  13944  3052 ?        Sl   13:01   0:00 /usr/sbin/system
root      4558  0.0  1.0  12896 10920 ?        S    13:01   0:00 /usr/bin/perl /u
root      5422  0.0  0.3  10876  3588 ?        Ss   13:03   0:00 sshd: mallow [pr
mallow    5485  0.0  0.1  10876  1780 ?        S    13:03   0:00 sshd: mallow@pts
mallow    5486  0.0  0.6   9736  6356 pts/1    Ss+  13:03   0:00 -bash
seldon    5608  0.3  1.4  91440 14996 ?        Rl   13:03   0:05 gnome-terminal
seldon    5611  0.0  0.0   2068   720 ?        S    13:03   0:00 gnome-pty-helper
seldon    5612  0.0  0.3   7124  3836 pts/2    Ss   13:03   0:00 bash
root      5888  0.0  0.3  10876  3576 ?        Ss   13:04   0:00 sshd: hardin [pr
hardin    5951  0.0  0.1  10876  1780 ?        S    13:04   0:00 sshd: hardin@pts
hardin    5952  0.0  0.6   9736  6360 pts/3    Ss+  13:04   0:00 -bash
root      7767  0.0  0.0      0     0 ?        S    13:20   0:00 [kworker/0:1]
root      8331  0.0  0.0      0     0 ?        S    13:25   0:00 [kworker/0:2]
root      8836  0.0  0.0      0     0 ?        S    13:30   0:00 [kworker/0:0]
seldon    8929  0.0  0.1   4708  1192 pts/2    R+   13:31   0:00 ps aux

This gives so much information, it is often hides what you want to know. Suppose you want to find out if there are any SSH processes running; then you can pipe the output of the ps command directly to a grep to to some filtering:

seldon@nexon ~ $ ps aux | grep ssh
root       737  0.0  0.2   5652  2256 ?        Ss   12:40   0:00 /usr/sbin/ssh
seldon    1699  0.0  0.0   3368   192 ?        Ss   12:41   0:00 /usr/bin/ssh-ag
root      4155  0.0  0.3  10876  3580 ?        Ss   13:00   0:00 sshd: seldon [pr
seldon    4242  0.0  0.1  10876  1640 ?        S    13:00   0:00 sshd: seldon@pt
root      5422  0.0  0.3  10876  3588 ?        Ss   13:03   0:00 sshd: mallow [pr
mallow    5485  0.0  0.1  10876  1780 ?        S    13:03   0:00 sshd: mallow@pt
root      5888  0.0  0.3  10876  3576 ?        Ss   13:04   0:00 sshd: hardin [pr
hardin    5951  0.0  0.1  10876  1780 ?        S    13:04   0:00 sshd: hardin@pt
seldon    9244  0.0  0.0   4156   852 pts/2    S+   13:34   0:00 grep --colour=a

Now we see SSH related processes from root, seldon, mallow, and hardin.

If you want to see the structure, showing which process spawned another, you can use either the -H or --forest switches:

seldon@nexon ~ $ ps aux --forest
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S    12:40   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [ksoftir]
root         5  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [kworker]
root         6  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [migrati]
root         7  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [cpuset]
root         8  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [khelper]
root         9  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [netns]
root        10  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [sync_su]
root        11  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [bdi-def]
root        12  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kintegr]
root        13  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kblockd]
root        14  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kacpid]
root        15  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kacpi_n]
root        16  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kacpi_h]
root        17  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [ata_sff]
root        18  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [khubd]
root        19  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [md]
root        23  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [khungta]
root        24  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [kswapd0]
root        25  0.0  0.0      0     0 ?        SN   12:40   0:00  \_ [ksmd]
root        26  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [fsnotif]
root        27  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [aio]
root        28  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [ecryptf]
root        29  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [crypto]
root        33  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kthrotl]
root        35  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [scsi_eh]
root        36  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [scsi_eh]
root        37  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [kworker]
root        40  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kmpathd]
root        41  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kmpath_]
root        42  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kondema]
root        43  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kconser]
root       224  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [mpt_pol]
root       225  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [mpt/0]
root       266  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [scsi_eh]
root       347  0.0  0.0      0     0 ?        S    12:40   0:01  \_ [jbd2/sd]
root       348  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [ext4-di]
root       683  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [vmmemct]
root       707  0.0  0.0      0     0 ?        S<   12:40   0:00  \_ [kpsmous]
root       961  0.0  0.0      0     0 ?        S    12:40   0:00  \_ [flush-8]
root      7767  0.0  0.0      0     0 ?        S    13:20   0:00  \_ [kworker]
root      8836  0.0  0.0      0     0 ?        S    13:30   0:00  \_ [kworker]
root      9332  0.0  0.0      0     0 ?        S    13:35   0:00  \_ [kworker]
root         1  0.0  0.1   3044  1844 ?        Ss   12:40   0:01 /sbin/init
root       398  0.0  0.0   2412   908 ?        S    12:40   0:00 upstart-udev-
root       444  0.0  0.1   2984  1248 ?        S<s  12:40   0:00 udevd --daemo
root       543  0.0  0.1   3140  1296 ?        S<   12:40   0:00  \_ udevd --d
root       544  0.0  0.1   3140  1240 ?        S<   12:40   0:00  \_ udevd --d
root       737  0.0  0.2   5652  2256 ?        Ss   12:40   0:00 /usr/sbin/ssh
root      4155  0.0  0.3  10876  3580 ?        Ss   13:00   0:00  \_ sshd: sel
seldon    4242  0.0  0.1  10876  1640 ?        S    13:00   0:00  |   \_ sshd:
seldon    4243  0.0  0.6   9736  6416 pts/0    Ss   13:00   0:00  |       \_ -
root      4347  0.0  0.1   7268  1988 pts/0    S    13:00   0:00  |           
root      4349  0.0  0.1   7168  1784 pts/0    S    13:00   0:00  |           
root      4356  0.0  0.6   9732  6356 pts/0    S+   13:00   0:00  |           
root      5422  0.0  0.3  10876  3588 ?        Ss   13:03   0:00  \_ sshd: mal
mallow    5485  0.0  0.1  10876  1780 ?        S    13:03   0:00  |   \_ sshd:
mallow    5486  0.0  0.6   9736  6356 pts/1    Ss+  13:03   0:00  |       \_ -
root      5888  0.0  0.3  10876  3576 ?        Ss   13:04   0:00  \_ sshd: har
hardin    5951  0.0  0.1  10876  1780 ?        S    13:04   0:00      \_ sshd:
hardin    5952  0.0  0.6   9736  6360 pts/3    Ss+  13:04   0:00          \_ -
root       762  0.0  0.0   2412   604 ?        S    12:40   0:00 upstart-socke
root       770  0.0  0.4  16628  4180 ?        Ss   12:40   0:00 smbd -F
root       885  0.0  0.1  16628  1320 ?        S    12:40   0:00  \_ smbd -F
syslog     776  0.0  0.1  28016  1840 ?        Sl   12:40   0:00 rsyslogd -c4
102        800  0.0  0.1   3612  1884 ?        Ss   12:40   0:00 dbus-daemon -
avahi      833  0.0  0.1   3016  1448 ?        S    12:40   0:00 avahi-daemon:
avahi      834  0.0  0.0   3016   432 ?        S    12:40   0:00  \_ avahi-dae
root       839  0.0  0.4  17636  4576 ?        Ssl  12:40   0:00 NetworkManage
root       841  0.0  0.2   4660  2500 ?        S    12:40   0:00 /usr/sbin/mod
root       844  0.0  0.3  24892  3892 ?        Sl   12:40   0:00 /usr/lib/poli
root       849  0.0  0.1   5172  1764 ?        S    12:40   0:00 /sbin/wpa_sup
root       870  0.0  0.2   6924  2956 ?        Ss   12:40   0:00 /usr/sbin/cup
root      1307  0.0  0.1   9420  1720 ?        Ss   12:40   0:00 nmbd -D
root      1389  0.0  0.0  30472   572 ?        Ssl  12:41   0:00 /usr/sbin/vmw
root      1427  0.1  0.3  25312  3696 ?        S    12:41   0:03 /usr/sbin/vmt
root      1454  0.0  0.0   1872   580 tty4     Ss+  12:41   0:00 /sbin/getty -
root      1459  0.0  0.0   1872   580 tty5     Ss+  12:41   0:00 /sbin/getty -
root      1471  0.0  0.0   1872   576 tty2     Ss+  12:41   0:00 /sbin/getty -
root      1472  0.0  0.0   1872   576 tty3     Ss+  12:41   0:00 /sbin/getty -
root      1474  0.0  0.0   1872   580 tty6     Ss+  12:41   0:00 /sbin/getty -
root      1477  0.0  0.0   1864   676 ?        Ss   12:41   0:00 acpid -c /etc
root      1484  0.0  0.0   2268   856 ?        Ss   12:41   0:00 cron
daemon    1485  0.0  0.0   2132   348 ?        Ss   12:41   0:00 atd
root      1529  0.0  0.1  12596  1512 ?        Ss   12:41   0:00 tpvmlpd2
root      1537  0.0  0.3  18388  3704 ?        Ssl  12:41   0:00 gdm-binary
root      1634  0.0  0.3  19716  3760 ?        Sl   12:41   0:00  \_ /usr/lib/
root      1645  0.3  3.5  46724 36408 tty7     Ss+  12:41   0:13      \_ /usr/
root      1657  0.0  0.3  20184  3680 ?        Sl   12:41   0:00      \_ /usr/
seldon    1666  0.0  0.7  35328  7240 ?        Ssl  12:41   0:00          \_ g
seldon    1699  0.0  0.0   3368   192 ?        Ss   12:41   0:00              
seldon    1733  0.0  1.0 107476 11248 ?        Sl   12:41   0:00              
seldon    1754  0.0  1.8  95408 19372 ?        Sl   12:41   0:01              
seldon    1756  0.0  0.8  45072  8328 ?        Sl   12:41   0:00              
seldon    1759  0.0  0.4  49080  4632 ?        Sl   12:41   0:00              
seldon    1760  0.0  0.3   9568  3680 ?        S    12:41   0:00              
seldon    1767  0.0  0.0   1912   512 ?        S    12:41   0:00              
seldon    1768  0.0  2.2 110856 23312 ?        Sl   12:41   0:00              
seldon    1762  0.0  1.2 107388 12364 ?        Sl   12:41   0:00              
seldon    1784  0.0  0.8  52948  8352 ?        Sl   12:41   0:00              
seldon    1804  0.0  1.1 103336 11488 ?        Sl   12:41   0:00              
seldon    1831  0.0  1.2 122180 13196 ?        Sl   12:41   0:00              
seldon    2006  0.0  0.6  19672  6656 ?        S    12:41   0:00              
seldon    2419  0.0  1.6  33680 16884 ?        S    12:41   0:00              
root      1543  0.0  0.3  26168  3420 ?        Sl   12:41   0:00 /usr/sbin/con
seldon    1702  0.0  0.0   3456   572 ?        S    12:41   0:00 /usr/bin/dbus
seldon    1703  0.0  0.1   4372  1644 ?        Ss   12:41   0:00 //bin/dbus-da
seldon    1708  0.0  0.4   9568  4332 ?        S    12:41   0:00 /usr/lib/libg
seldon    1714  0.0  1.1  99780 11492 ?        Ssl  12:41   0:00 /usr/lib/gnom
seldon    1716  0.0  0.3  42180  3244 ?        Sl   12:41   0:00 /usr/bin/gnom
seldon    1724  0.0  0.2   7632  2240 ?        S    12:41   0:00 /usr/lib/gvfs
seldon    1729  0.0  0.2  31084  2496 ?        Ssl  12:41   0:00 /usr/lib/gvfs
seldon    1738  0.0  0.4  95728  4952 ?        S<sl 12:41   0:00 /usr/bin/puls
seldon    1757  0.0  0.3  20788  3336 ?        Sl   12:41   0:00  \_ /usr/lib/
rtkit     1740  0.0  0.1  18888  1212 ?        SNl  12:41   0:00 /usr/lib/rtki
root      1778  1.1  3.3  94796 33828 ?        Sl   12:41   0:40 splunkd -p 80
root      1785  0.0  0.1  25936  2040 ?        Ss   12:41   0:00  \_ [splunkd 
seldon    1793  0.0  1.2  31476 13020 ?        Sl   12:41   0:00 /usr/bin/pyth
seldon    1891  0.0  0.0   3912   244 ?        S    12:41   0:00  \_ /bin/cat
seldon    1945  0.0  0.0      0     0 ?        Z    12:41   0:00  \_ 
seldon    1805  0.2  1.4  50896 15328 ?        S    12:41   0:08 /usr/lib/vmwa
seldon    1858  0.0  0.6  27088  6940 ?        Sl   12:41   0:00 /usr/lib/noti
seldon    1860  0.0  0.3   9040  3324 ?        S    12:41   0:00 /usr/lib/gvfs
root      1863  0.0  0.3  14020  3296 ?        Sl   12:41   0:00 /usr/lib/udis
root      1872  0.0  0.0   5564   724 ?        S    12:41   0:00  \_ udisks-da
root      1865  0.0  0.3  16720  3648 ?        Sl   12:41   0:00 /usr/lib/upow
seldon    1886  0.0  0.2   8512  2256 ?        S    12:41   0:00 /usr/lib/gvfs
seldon    1895  0.0  0.3  50552  3600 ?        Ssl  12:41   0:00 /usr/lib/bono
seldon    1898  0.0  0.2  18204  2252 ?        Sl   12:41   0:00 /usr/lib/gvfs
seldon    1901  0.0  0.3  27624  3236 ?        Ss   12:41   0:00 gnome-screens
seldon    1922  0.0  0.3   8212  3212 ?        S    12:41   0:00 /usr/lib/gvfs
seldon    1960  0.0  2.9 102388 30496 ?        Sl   12:41   0:01 python /usr/l
seldon    5608  0.3  1.4  91440 14996 ?        Sl   13:03   0:07  \_ gnome-ter
seldon    5611  0.0  0.0   2068   720 ?        S    13:03   0:00      \_ gnome
seldon    5612  0.0  0.3   7124  3836 pts/2    Ss   13:03   0:00      \_ bash
seldon    9755  0.0  0.1   4720  1100 pts/2    R+   13:39   0:00          \_ p
seldon    1962  0.0  1.2  80140 12404 ?        Sl   12:41   0:00 /usr/lib/gnom
seldon    1964  0.0  1.2  86604 13308 ?        Sl   12:41   0:00 /usr/lib/gnom
seldon    1966  0.0  1.1  95112 11564 ?        Sl   12:41   0:00 /usr/lib/indi
seldon    1968  0.0  0.8  76112  8836 ?        Sl   12:41   0:00 /usr/lib/gnom
seldon    2008  0.0  0.1   7440  1984 ?        S    12:41   0:00 /usr/lib/gvfs
seldon    2027  0.0  0.2   7764  2496 ?        S    12:41   0:00 /usr/lib/gvfs
root      2039  0.2  1.9  89036 19740 ?        Sl   12:41   0:09 python -O /op
seldon    2055  0.0  0.4  47672  4120 ?        Sl   12:41   0:00 /usr/lib/indi
seldon    2056  0.0  0.5 112276  5812 ?        Sl   12:41   0:00 /usr/lib/indi
root      2413  0.0  0.0   1872   576 tty1     Ss+  12:41   0:00 /sbin/getty -
root      4555  0.0  0.2  13944  3052 ?        Sl   13:01   0:00 /usr/sbin/sys
root      4558  0.0  1.0  12896 10920 ?        S    13:01   0:00 /usr/bin/perl
What Ports are Open on the System?

The command to determine what ports are open on the system is called netstat. When using netstat though, you should remember that Linux and Unix systems have two kinds of ports- network ports and Unix sockets. Unix sockets are used for communication by different processes on the same system, so in general we are uninterested in those.

The tool has a number of useful flags, including

  • -v Be verbose
  • -n Use numeric values for ports, rather than names
  • -A inet (or –inet) Show only IP (v4) connections
  • -A inet6 (or –inet6) Show only IPv6 connections
  • -x Show only Unix sockets
  • -t Show only TCP (v4/v6)
  • -u Show only UDP (v4/v6)
  • -p Show the PID for that connection
  • -l Show listening sockets (not shown by default)
  • -a Show listening and open sockets
  • -r Show routing table

If you want to find out what is listening on your system, a good set of flags is

seldon@nexon ~ $ sudo netstat -nlpv --inet
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address  Foreign Address  State    PID/Program name
tcp        0      0 0.0.0.0:8000   0.0.0.0:*        LISTEN   2039/python     
tcp        0      0 0.0.0.0:139    0.0.0.0:*        LISTEN   770/smbd        
tcp        0      0 0.0.0.0:9997   0.0.0.0:*        LISTEN   1778/splunkd    
tcp        0      0 0.0.0.0:22     0.0.0.0:*        LISTEN   737/sshd        
tcp        0      0 127.0.0.1:631  0.0.0.0:*        LISTEN   870/cupsd       
tcp        0      0 0.0.0.0:8089   0.0.0.0:*        LISTEN   1778/splunkd    
tcp        0      0 0.0.0.0:445    0.0.0.0:*        LISTEN   770/smbd        
udp        0      0 10.0.2.255:137 0.0.0.0:*                 1307/nmbd       
udp        0      0 10.0.2.5:137   0.0.0.0:*                 1307/nmbd       
udp        0      0 0.0.0.0:137    0.0.0.0:*                 1307/nmbd       
udp        0      0 10.0.2.255:138 0.0.0.0:*                 1307/nmbd       
udp        0      0 10.0.2.5:138   0.0.0.0:*                 1307/nmbd       
udp        0      0 0.0.0.0:138    0.0.0.0:*                 1307/nmbd       
udp        0      0 0.0.0.0:5353   0.0.0.0:*                 833/avahi-daemon: r
udp        0      0 0.0.0.0:39288  0.0.0.0:*                 833/avahi-daemon: r

Here we were able both TCP and UDP ports in numeric form, and we also know the name " PID of the process that opened the port.

What Resources are Opened by What Process?

You can ask a broader question, namely what resources are being used by the system. These resources can be network sockets, but they can also be devices (like your USB drive) or files. The tool here is called lsof, and it can do a lot.

Suppose we wanted to get the same kind of information we saw with netstat; we can then try

seldon@nexon ~ $ sudo lsof -i 4
COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd       737   root    3r  IPv4   8131      0t0  TCP *:ssh (LISTEN)
smbd       770   root   24u  IPv4   8326      0t0  TCP *:microsoft-ds (LISTEN)
smbd       770   root   25u  IPv4   8328      0t0  TCP *:netbios-ssn (LISTEN)
avahi-dae  833  avahi   13u  IPv4   8081      0t0  UDP *:mdns 
avahi-dae  833  avahi   15u  IPv4   8083      0t0  UDP *:39288 
cupsd      870   root    6u  IPv4   8366      0t0  TCP localhost:ipp (LISTEN)
nmbd      1307   root    9u  IPv4   8703      0t0  UDP *:netbios-ns 
nmbd      1307   root   10u  IPv4   8704      0t0  UDP *:netbios-dgm 
nmbd      1307   root   11u  IPv4   8706      0t0  UDP nexon.cosc.tu:netbios-ns 
nmbd      1307   root   12u  IPv4   8707      0t0  UDP 10.0.2.255:netbios-ns 
nmbd      1307   root   13u  IPv4   8708      0t0  UDP nexon.cosc.tu:netbios-dgm 
nmbd      1307   root   14u  IPv4   8709      0t0  UDP 10.0.2.255:netbios-dgm 
splunkd   1778   root    4u  IPv4  10558      0t0  TCP *:8089 (LISTEN)
splunkd   1778   root   21u  IPv4  13798      0t0  TCP *:9997 (LISTEN)
python    2039   root    5r  IPv4  17456      0t0  TCP *:8000 (LISTEN)
sshd      4155   root    3r  IPv4  22763      0t0  TCP nexon.cosc.tu:ssh-
>proclas.cosc.tu:49210 (ESTABLISHED)
sshd      4242 seldon    3u  IPv4  22763      0t0  TCP nexon.cosc.tu:ssh-
>proclas.cosc.tu:49210 (ESTABLISHED)
sshd      5422   root    3r  IPv4  26928      0t0  TCP nexon.cosc.tu:ssh-
>capella.cosc.tu:43317 (ESTABLISHED)
sshd      5485 mallow    3u  IPv4  26928      0t0  TCP nexon.cosc.tu:ssh-
>capella.cosc.tu:43317 (ESTABLISHED)
sshd      5888   root    3r  IPv4  28225      0t0  TCP nexon.cosc.tu:ssh-
>aurora.cosc.tu:56083 (ESTABLISHED)
sshd      5951 hardin    3u  IPv4  28225      0t0  TCP nexon.cosc.tu:ssh-
>aurora.cosc.tu:56083 (ESTABLISHED)

Here the flag -i 4 asks for information about all IPv4 connections; we can ask for IPv6 with -i 6

seldon@nexon ~ $ sudo lsof -i 6
COMMAND   PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd      737  root    4u  IPv6   8133      0t0  TCP *:ssh (LISTEN)
avahi-dae 833 avahi   14u  IPv6   8082      0t0  UDP *:mdns 
avahi-dae 833 avahi   16u  IPv6   8084      0t0  UDP *:53135 
cupsd     870  root    5u  IPv6   8365      0t0  TCP ip6-localhost:ipp (LISTEN)

If you want to know about TCP connections, you can use -i TCP, and if you want just port 22, then the flag would be -i TCP:22

seldon@nexon ~ $ sudo lsof -i TCP:22
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd     737   root    3r  IPv4   8131      0t0  TCP *:ssh (LISTEN)
sshd     737   root    4u  IPv6   8133      0t0  TCP *:ssh (LISTEN)
sshd    4155   root    3r  IPv4  22763      0t0  TCP nexon.cosc.tu:ssh->proclas.cosc.tu:49210 (ESTABLISHED)
sshd    4242 seldon    3u  IPv4  22763      0t0  TCP nexon.cosc.tu:ssh->proclas.cosc.tu:49210 (ESTABLISHED)
sshd    5422   root    3r  IPv4  26928      0t0  TCP nexon.cosc.tu:ssh->capella.cosc.tu:43317 (ESTABLISHED)
sshd    5485 mallow    3u  IPv4  26928      0t0  TCP nexon.cosc.tu:ssh->capella.cosc.tu:43317 (ESTABLISHED)
sshd    5888   root    3r  IPv4  28225      0t0  TCP nexon.cosc.tu:ssh->aurora.cosc.tu:56083 (ESTABLISHED)
sshd    5951 hardin    3u  IPv4  28225      0t0  TCP nexon.cosc.tu:ssh->aurora.cosc.tu:56083 (ESTABLISHED)

Suppose you are concerned that a connection from 10.0.2.2 might be hostile. Can you find out what other connections come from that host? Sure!

seldon@nexon ~ $ sudo lsof -i @10.0.2.2
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    5888   root    3r  IPv4  28225      0t0  TCP nexon.cosc.tu:ssh->aurora.cosc.tu:56083 (ESTABLISHED)
sshd    5951 hardin    3u  IPv4  28225      0t0  TCP nexon.cosc.tu:ssh->aurora.cosc.tu:56083 (ESTABLISHED)

If you prefer numeric port numbers, add -P

seldon@nexon ~ $ sudo lsof -i @10.0.2.2 -P
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    5888   root    3r  IPv4  28225      0t0  TCP nexon.cosc.tu:22->aurora.cosc.tu:56083 (ESTABLISHED)
sshd    5951 hardin    3u  IPv4  28225      0t0  TCP nexon.cosc.tu:22->aurora.cosc.tu:56083 (ESTABLISHED)

We can do more than just examine network connections with lsof. Suppose that you want to see what resources have been opened by all commands that contain the text &quotssh"; then we use the -c flag; just be prepared for a lot of data.

seldon@nexon ~ $ sudo lsof -c sshd | more
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system /home/seldon/.gvfs
      Output information may be incomplete.
COMMAND  PID   USER   FD   TYPE     DEVICE SIZE/OFF   NODE NAME
sshd     737   root  cwd    DIR        8,1     4096      2 /
sshd     737   root  rtd    DIR        8,1     4096      2 /
sshd     737   root  txt    REG        8,1   470240 809479 /usr/sbin/sshd
sshd     737   root  mem    REG        8,1    79476 263219 /lib/i386-linux-gnu/libz.so.1.2.3.4
sshd     737   root  mem    REG        8,1     9736 263146 /lib/i386-linux-gnu/libdl-2.13.so
sshd     737   root  mem    REG        8,1    71432 263203 /lib/i386-linux-gnu/libresolv-2.13.so
sshd     737   root  mem    REG        8,1    26400 263178 /lib/i386-linux-gnu/libnss_compat-2.13.so
sshd     737   root  mem    REG        8,1     9744 263214 /lib/i386-linux-gnu/libutil-2.13.so
sshd     737   root  mem    REG        8,1    34264 263142 /lib/i386-linux-gnu/libcrypt-2.13.so
sshd     737   root  mem    REG        8,1   191240 795294 /usr/lib/i386-linux-gnu/libgssapi_krb5.so.2.2
sshd     737   root  mem    REG        8,1    26112 795309 /usr/lib/i386-linux-gnu/libkrb5support.so.0.1
sshd     737   root  mem    REG        8,1   117960 263123 /lib/i386-linux-gnu/ld-2.13.so
sshd     737   root  mem    REG        8,1    30980 262408 /lib/libwrap.so.0.7.6
sshd     737   root  mem    REG        8,1    38500 263186 /lib/i386-linux-gnu/libnss_nis-2.13.so
sshd     737   root  mem    REG        8,1     9608 263141 /lib/i386-linux-gnu/libcom_err.so.2.1
sshd     737   root  mem    REG        8,1    46644 263191 /lib/i386-linux-gnu/libpam.so.0.82.3
sshd     737   root  mem    REG        8,1  1434180 263136 /lib/i386-linux-gnu/libc-2.13.so
sshd     737   root  mem    REG        8,1   711308 795307 /usr/lib/i386-linux-gnu/libkrb5.so.3.3
--More--

If you want to know all of the resources opened by a particular user, just specify the UID of the user

seldon@nexon ~ $ sudo lsof -u mallow | more
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system /home/seldon/.gvfs
      Output information may be incomplete.
COMMAND  PID   USER   FD   TYPE     DEVICE SIZE/OFF   NODE NAME
sshd    5485 mallow  cwd    DIR        8,1     4096      2 /
sshd    5485 mallow  rtd    DIR        8,1     4096      2 /
sshd    5485 mallow  txt    REG        8,1   470240 809479 /usr/sbin/sshd
sshd    5485 mallow  mem    REG        8,1    30980 262408 /lib/libwrap.so.0.7.6
sshd    5485 mallow  mem    REG        8,1  1341364 262340 /lib/libcrypto.so.0.9.8
sshd    5485 mallow  mem    REG        8,1    79476 263219 /lib/i386-linux-gnu/libz.so.1.2.3.4
sshd    5485 mallow  mem    REG        8,1    34264 263142 /lib/i386-linux-gnu/libcrypt-2.13.so
sshd    5485 mallow  mem    REG        8,1    26400 263178 /lib/i386-linux-gnu/libnss_compat-2.13.so
sshd    5485 mallow  mem    REG        8,1    38500 263186 /lib/i386-linux-gnu/libnss_nis-2.13.so
sshd    5485 mallow  mem    REG        8,1     5344 263242 /lib/i386-linux-gnu/security/pam_permit.so
sshd    5485 mallow  mem    REG        8,1    13672 266628 /lib/security/pam_ecryptfs.so
sshd    5485 mallow  mem    REG        8,1   199676 791892 /usr/lib/libssl3.so
sshd    5485 mallow  mem    REG        8,1     9568 791762 /usr/lib/libplds4.so
sshd    5485 mallow  mem    REG        8,1   711308 795307 /usr/lib/i386-linux-gnu/libkrb5.so.3.3
sshd    5485 mallow  mem    REG        8,1    95996 791692 /usr/lib/libnssutil3.so
sshd    5485 mallow  mem    REG        8,1     5376 263241 /lib/i386-linux-gnu/security/pam_nologin.so
sshd    5485 mallow  mem    REG        8,1     9524 266627 /lib/security/pam_ck_connector.so
sshd    5485 mallow  mem    REG        8,1    30684 263205 /lib/i386-linux-gnu/librt-2.13.so
--More--

You can do the same by PID, just with the -p flag.

What do we Know about a Process?

Suppose you see a suspicious process. What can you quickly determine about it? A great deal of information about that process is stored in the /proc directory, in the subdirectory named after that PID.

For example, suppose we are interested in the process with PID 15895. We first change to the proper directory

seldon@nexon ~ $ cd /proc/15895
seldon@nexon /proc/15895 $ ls
attr        coredump_filter  io         mounts         personality  stat
autogroup   cpuset           latency    mountstats     root         statm
auxv        cwd              limits     net            sched        status
cgroup      environ          loginuid   oom_adj        schedstat    syscall
clear_refs  exe              maps       oom_score      sessionid    task
cmdline     fd               mem        oom_score_adj  smaps        wchan
comm        fdinfo           mountinfo  pagemap        stack

We can find the command line used to start the process:

seldon@nexon /proc/15895 $ less cmdline 

vi^@test^@
cmdline (END) 

The current working directory is stored in cwd and the environment variables in environ

seldon@nexon /proc/15895 $ sudo ls -l cwd
lrwxrwxrwx 1 hardin hardin 0 2013-02-10 14:41 cwd -> /home/hardin

seldon@nexon /proc/15895 $ sudo less environ 
"environ" may be a binary file.  See it anyway? 

TERM=xterm^@SHELL=/bin/bash^@XDG_SESSION_COOKIE=8f714e2132c47e5cb774374b00000004-
1360519456.496207-766548711^@SSH_CLIENT=10.0.2.2 56083 
22^@SSH_TTY=/dev/pts/3^@USER=hardin^@MAIL=/var/mail/hardin^@PATH=/usr/local/sbin
:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games^@PWD=/home/hardin^@LANG
=en_US.UTF-8^@SHLVL=1^@HOME=/home/hardin^@LOGNAME=hardin^@SSH_CONNECTION=10.0.2.2 
56083 10.0.2.5 22^@_=/usr/bin/vi^@

To find the precise executable, we check exe, and if we want to see what files are opened, we can see which file descriptor points where by checking fd

seldon@nexon /proc/15895 $ sudo ls -l exe
lrwxrwxrwx 1 hardin hardin 0 2013-02-10 14:41 exe -> /usr/bin/vim.tiny
seldon@nexon /proc/15895 $ sudo ls -l fd
total 0
lrwx------ 1 hardin hardin 64 2013-02-10 14:41 0 -> /dev/pts/3
lrwx------ 1 hardin hardin 64 2013-02-10 14:41 1 -> /dev/pts/3
lrwx------ 1 hardin hardin 64 2013-02-10 14:41 2 -> /dev/pts/3
lrwx------ 1 hardin hardin 64 2013-02-10 14:41 3 -> /home/hardin/.test.swp

Information about the memory map for the process is held in maps, while
/proc/PID/net/ is a directory of network information

With all of this information, can you determine what PID 15895 was doing?

Responding to a Suspicious Process

Students first learning how to defend a network tend to have a consistent response towards suspicious processes- nuke’em from orbit as quickly as possible.

This is often bad- even if the process actually is malicious. Killing a process deprives you of the chance to find out more about what is going on. How did the process get started? What processes are related? Has it opened a network connection? To where? Is that address connected to other processes?

Take some time and look carefully at the process using these tools, and find out all you can about it, if it is malicious, and how they may have gotten in. Then you can decide what to do, and if appropriate, nuke it from orbit.

Advertisements
  1. February 18, 2013 at 12:16 am

    In the section “Sending Custom Log Messages” should be: “Modify your /etc/rsyslog.conf file…” not /etc/resolv.conf

  2. Donte
    February 19, 2013 at 8:31 am

    The original standard for syslog only allowed for sending and receiving remote logs via UDP on port 514. To configure our syslog server to do so, instead of providing a file name as the destination for our logs, we provide the ip address, preceded by “@”; so if we add the line

    *.* 10.0.2.3

    This screen shot should show *.* @10.0.2.3 the following line states that previous screen shot will send message but it will not with out the @ as stated above

  3. David Bitner
    February 21, 2013 at 5:37 pm

    In the open ssh section, when looking for the file in labshare, use the file as it is shown in the screenshot, not the one listed above it.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: