2015-05 SSH, File Servers, Operational Awareness

SSH

SSH Servers

On CentOS, the SSH server is already running and already allowed through the firewall; it is running version 5.3p1.

On Ubuntu systems (Desktop or Server), the server is not present. For a Desktop system, grab the package openssh-server_5.9p1-1ubuntu7_i386.deb from the labshare or online. Install it in the usual way:

zathras@achilles:~$ sudo dpkg -i /home/zathras/Desktop/openssh-
server_5.9p1-5ubuntu1_i386.deb 
[sudo] password for zathras: 
Selecting previously unselected package openssh-server.
(Reading database ... 141338 files and directories currently installed.)
Unpacking openssh-server (from .../openssh-server_5.9p1-5ubuntu1_i386.deb) ...
Setting up openssh-server (1:5.9p1-5ubuntu1) ...
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
ssh start/running, process 2855
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Processing triggers for ufw ...
Processing triggers for man-db ...

As noted earlier, the Ubuntu firewall is off by default; if you have enabled it, then you must open the necessary port.

If you want to know what services are running on your Ubuntu system, you might try a command like chkconfig, however it is not installed by default. The Ubuntu alternative is to use the command

zathras@achilles:~$ service --status-all

Getting the SSH server onto the Ubuntu Server system is complicated by the fact that it has no GUI, so the usual trick of dragging a file from the host and dropping it on the desktop of the guest will not allow us to copy the files. We could set up a web server or an ftp server, and use client software like wget on the server to get the files; we have already seen how to do that with Python. We could also take the server out out of the lab to an Internet connection and use ftp and/or wget.

There is another approach. From VMWare, select VM, then Settings. Under the Options tab, select Shared Folders. Select Add; then you can choose a folder on the host that will be mapped to a folder on the guest.

hgfs

In this example, I mapped the host directory d:\Mike\Desktop\New Software to the directory "Software". That directory will now appear in the guest, in the location /mnt/hgfs/Software

[Did he just say "hgfs"? Yep. Host-Guest-File-System. hgfs. It does make sense. Sort of.]

To actually install the SSH server, we need two packages- the server itself (openssh-server_5.9p1-5ubuntu1_amd64.deb) and the package libwrap0_7.6.q-18_amd64.deb. Install them both via dpkg.

Ubuntu 12.04 Server (x64) (Agamemnon)-2014-02-14-13-18-33

Mint 13 Cinnamon also does not come with an SSH server by default; however since it is a variant of Ubuntu, it can be installed using the same method and package as on Ubuntu.
Mint 13 (x64) (Alexander)-2014-02-14-13-25-54

The Mint 13 KDE system does have an SSH server already started.

An SSH server can also be installed on a Windows system. My preferred method for doing so is to use Cygwin, which provides Windows ports for a large number of Linux applications. The catch though, is that though it is possible to install and use Cygwin on systems not connected to the Internet, it is difficult, especially for beginners to Cygwin.

Rather than go through that trouble, let me offer as an alternative FreeSSHD. It is older, but it is free and (mostly) works on Windows systems. Installation is standard, though I should note that I have had nothing but bad luck when I installed the tool as a service.

The icon on the Desktop starts a tool to control the server; this controller lives in the System tray. Another place I have run into trouble is starting another instance of the controller when one is already running; because the system tray hides the FreeSSHd icon by default, this has happened embarrassingly often.

Another catch is that if you connect to the system using a Linux SSH client, it assumes that you are at the top of the screen; it also does not clear the screen. Thus, you may be staring at the bottom of the screen waiting for a response, when it has already authenticated you and moved your cursor to the top of the screen:
FreeSSHD

Note also that you need to manually specify which users are allowed to use the SSH server. They can be domain users if you wish, but you can also create new users that are not connected to either a domain account or a local Windows account.
FreeSSHDUsers

SSH Clients

On Linux systems, simply use the built-in command line client to connect to a remote SSH server.

The best client for SSH from a Windows System in PuTTY. It is available on the lab share and online.

Its general use is straightforward- enter the required connection information in the box and select Open. You can save the profile settings by simply giving it a name and choosing Save.
Putty1

Putty2

SSH PKI

SSH can use public keys to authenticate users; this is often more convenient than passwords. To set up the PKI in CentOS 6.2, modify /etc/ssh/sshd_config to uncomment out the following lines

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

The situation in Mint is similar, though it writes the location of the Authorized Keys File as

#AuthorizedKeysFile	%h/.ssh/authorized_keys

Restart sshd; on CentOS 6.2 you can use

[root@acheron ~]# service sshd restart
Stopping sshd:                                             [  OK  ]
Starting sshd:                                             [  OK  ]

while on Mint 11 & Ubuntu 11.04 you can use

zathras@alexander /etc/ssh $ sudo service ssh restart
[sudo] password for zathras: 
ssh stop/waiting
ssh start/running, process 3992

Each user that wants to log in to this server using SSH PKI needs to set the proper permissions on the ~/.ssh directory, create the file ~/.ssh/authorized_keys, and ensure that file also has the proper permissions. Indeed, we require

  • ~/.ssh is chmod 700 (rwx — —)
  • ~/.ssh/authorized_keys is chmod 644 (rw- r– r–)

We can do this manually now, but there is a method to do this at the same time the keys are copied to the server.

A user that wants to connect to a client via PKI must first generate the keys that will be used for the connection via ssh-keygen.

zathras@alexander ~ $ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/zathras/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/zathras/.ssh/id_rsa.
Your public key has been saved in /home/zathras/.ssh/id_rsa.pub.
The key fingerprint is:
0f:3c:53:1a:95:17:f1:1c:0e:28:0f:14:d2:d0:9e:cc zathras@alexander.cosc.tu
The key's randomart image is:
+--[ RSA 2048]----+
|      o=o..o+o.  |
|       .=.o .= . |
|       +.=..  +  |
|       .E+.      |
|        S        |
|         =       |
|          .      |
|                 |
|                 |
+-----------------+

Note that zathras did not use a passphrase to secure the keys.

What results are a pair of keys, a private key and a public key, stored in the directory /home/seldon/.ssh:

+-----------------+
zathras@alexander ~ $ ls -l .ssh/
total 8
-rw------- 1 zathras zathras 1675 Feb 14 14:30 id_rsa
-rw-r--r-- 1 zathras zathras  407 Feb 14 14:30 id_rsa.pub

To use SSH PKI, we need to copy the public key (ONLY) to the server; the content of that key needs to then be appended to the file ~/.ssh/authorized_keys on the server for that user. Note that the username on the client and the username on the server can be different. We can do this manually a number of ways, including using SSH with a password. However, there is a tool that will do this automatically for us, and will even ensure that the server’s directories and files have the correct permissions.

zathras@alexander ~ $ ssh-copy-id valen@acheron.cosc.tu
The authenticity of host 'acheron.cosc.tu (10.0.2.100)' can't be established.
RSA key fingerprint is 5a:b1:0d:b1:a0:ad:05:6c:28:51:ff:15:df:53:9e:1e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'acheron.cosc.tu,10.0.2.100' (RSA) to the list of
known hosts.
valen@acheron.cosc.tu's password: 
Now try logging into the machine, with "ssh 'valen@acheron.cosc.tu'", and check 
in:
  ~/.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

zathras@alexander ~ $ ssh valen@acheron.cosc.tu
[valen@acheron ~]$ 

Once this is complete, zathras on alexander can SSH into acheron as valen without using a password.

You can do a lot of surprising thing with SSH. Suppose you want to just execute a single command on a remote system- say cat /etc/hostname. Well, run it as follows:

[seldon@proclas ~]$ ssh nexon cat /etc/hostname
nexon.cosc.tu

Since PKI has been setup already, no password was required, and the result of the command is simply dropped to our screen. If we had not set up PKI, then we would need to enter our password each time we wanted to execute a command.

Suppose you wanted to copy the file /home/seldon/text to the remote host. Then you can use scp, which used ssh as its transport mechanism:

zathras@alexander ~/Documents $ scp text valen@acheron.cosc.tu:/home/valen/
copied_text
text                                          100%   27     0.0KB/s   00:00   

Let’s run ls on the remote host to see if the file arrived:

zathras@alexander ~/Documents $ ssh valen@acheron.cosc.tu "ls -al /home/valen"
total 36
drwx------. 5 valen valen 4096 Feb 14 11:49 .
drwxr-xr-x. 4 root  root  4096 Feb 14 11:40 ..
-rw-r--r--. 1 valen valen   18 Dec  2  2011 .bash_logout
-rw-r--r--. 1 valen valen  176 Dec  2  2011 .bash_profile
-rw-r--r--. 1 valen valen  124 Dec  2  2011 .bashrc
-rw-r--r--. 1 valen valen   27 Feb 14 11:49 copied_text
drwxr-xr-x. 2 valen valen 4096 Nov 11  2010 .gnome2
drwxr-xr-x. 4 valen valen 4096 Jan  4 01:56 .mozilla
drwx------. 2 valen valen 4096 Feb 14 11:40 .ssh

And now let’s bring the file back

zathras@alexander ~/Documents $ scp valen@acheron.cosc.tu:/home/valen/
copied_text ./copied_text

copied_text                                   100%   27     0.0KB/s   00:00    
zathras@alexander ~/Documents $ ls -l
total 8
-rw-r--r-- 1 zathras zathras 27 Feb 14 14:53 copied_text
-rw-r--r-- 1 zathras zathras 27 Feb 14 14:49 text

Suppose that there is a graphical program like firefox on your remote host. You can run it remotely, provided you pass the "-Y" flag. Here is a screenshot of the user zathrad on the Mint 13 system alexander logging in to the CentOS system acheron as valen, and then calling Firefox from the command prompt.
Mint 13 (x64) (Alexander)-2014-02-14-14-58-14

We can also use PKI with PuTTY. Start by running the PUTTYGEN key generator program; be sure you select SSH2-RSA for the key type, as there are weaknesses with the other approaches.
PuttyKey

Save both the private key and the public key. We don’t have the simple technique to add the public key to the remote server that we used before, however we can use SSH with a password, then copy and paste the public key on to the server. Unfortunately, the format that PuTTY uses is not one well suited to copying.

For example, here is the public key file generated in this example

---- BEGIN SSH2 PUBLIC KEY ----
Comment: "rsa-key-20140214"
AAAAB3NzaC1yc2EAAAABJQAAAIB+u/75GJgycyGe0tcwHzS6cckE+1skZbG1Me5y
FfJbtaXYDbY+JURdZ5DsDwAGxacROnUF1f/hwJxWkyglK7lo7xWoM2KJRE1WW+4D
nXofLY6I3HvkVN/N+Hvx2QFIsg+0cS2rW0tx8PSpsonnqCnpFC/kvT2/YVNAJq/O
U078CQ==
---- END SSH2 PUBLIC KEY ----

If you copy this to the file ~/.ssh/authorized_keys file for a user, it will fail. Why? Suppose that we have an existing authorized_keys file generated by OpenSSH; it has permissions 600, and has contents similar to the following

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3b9Hp1k4lqRUNIbx4jg1nrXJw1CkchKIgdtKrTSQi52
72tC84mg74XNlxxlNkzVs8cxxJTxfimXOhZUHWbaYHoMPius9o9yQk2U+2/CHMx6JS90pTaHkMvEgyM
aZNnSVZhIg2/v0n/aLOM2ite2iaTQKN3LNZ39rTdcejLSo/DGWJACR6JJS5Af3dD1AWiIyNhDELXCsJ
FMuQqhrIzIfNT52qJ3Yd8gmNbWctATSQKNxBKK1zoBCTVlYjZ/IkRhoANjpCsnFV20VOU9xFrQ2aQ6S
kXzp7NzsIk+DlGV1FefbnhNKiMGglzgPNLXWZ24Bjq/XsSeikFsCLLHOSLQeFw== valen@alexande
r.cosc.tu
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6y+v/2XXCk/694UzQ3egHOE0jWzAM/gwkLTbud371A6
TbcRS5+chfLwmDl1aO5+T/dz9ydkHKOOln4QbFs/JtCtJecpE9DlLC9jFQf9H1+4Vbv5eYYmPKlbkJa
GLoSPs4Vk1EVg75kx4pBGRdfR839t9axiDwIrku7HdGapoGRmFOzvcNmNbagx8Rz7/74GRmpdYCHbSL
dQ7BtcRHX3XV2NtdQIifCiUje5KSp+RT08JT82q7qZ++Hsbk6OnTJDBJ5TeJXXU4WgNikKlKn+gAYSW
DcjmYTfZLvnzkte6OFPNQXktvivI27y3lPRPWQXNQ7RdzL3hgDD/ro2bDlXJ+w== root@alexander
.cosc.tu

Here we see two RSA keys, one for the user valen on the host alexander.cosc.tu, and a second for the user root on the same host. We need to modify our PuTTY generated key so that it has the same format; is needs to start with the text ssh-rsa and should end with the user and host that has the corresponding private key; the commented lines need to be removed. Further, the result needs to be on a single line with no newlines.

Even better, if you open the public key with a simple text editor like Notepad, you won’t even realize that the key is split over multiple lines, so and copying it becomes even more trouble.

Here is one way to proceed. Open the public key in Wordpad. Under View, make sure that lines are not being wrapped. Delete the first two lines and the last line; then make sure that all that remains is on a single line. Be sure that the line starts with the magic text "ssh-rsa" and ends with the user name and host name where the private key is stored; you should end with something like this

PuttyKeyEdit

Add that single line into the file ~/.ssh/authorized_keys on the remote server; I did it by using SSH and a password to login, then used vi to open the file, and just pasted the result to the end of the file by right-clicking. You should obtain something like the following:
PuttyKey

Save the result. To log in using this key, specify the user name in the Data dialog box (beneath Connection), while the location of the private key is specified in the Auth dialog box (beneath Connection, beneath SSH).
Putty3

Given the settings in different places, I recommend that you save your resulting configuration (in Session).

Opening the connection will then give you access to the remote server without a password.

SSH can also be used to create tunnels, but we don’t have time to discuss SSH tunneling in the detail it deserves.

Securing SSH

check authorized keys

There are a number of things that we can do to try to secure our SSH server. First, we can prevent root from logging in directly. If someone needs root privileges from a remote site, they can log in as a regular user and then use either sudo or su. To make this change, add the line

PermitRootLogin no

to the sshd_config file. Note that the default state is to allow root logins; in the CentOS 6.2 default configuration file a commented out line allowing root logins is present, while on the Mint 13 / Ubuntu 12.04 default configuration files the line is present and explicitly allows root logins.

Speaking of users, you may want to allow only certain users to be allowed to use SSH to log in to the box. The line

AllowUsers zathras valen

will allow only the listed users to be able to SSH into the machine; other users (e.g.mcole) would not be able to log in. Though using a white list of allowed users is generally preferred, you could create a black list of users not allowed to log in via the DenyUsers directive.

Be sure to use only SSH protocol 2, as the older SSH-1 has a number of vulnerabilities and should be considered obsolete. Though SSH-2 is the default, your sshd_config file should contain the line

Protocol 2

to ensure that the system does not fall back to SSH-1.

You can also use the files /etc/hosts.allow and /etc/hosts.deny to restrict access. This is a three step process-

  • If a service / host combination is in the file hosts.allow, then access to the service is granted
  • If the combination is not in hosts.allow, then hosts.deny is checked; if the service / host combination matches then access is denied
  • If neither has occurred, then access is granted

Each line has the form

 service : host(s)

For our purpose the service can be either sshd or ALL. The host(s) can be specified by name, by IP address, or by IP Address Range. Multiple hosts can be separated by commas. Suppose we want to allow SSH from only the hosts 10.0.2.3 and 10.0.2.13; then we can set up the hosts.allow file as

# /etc/hosts.allow
sshd : 10.0.2.3, 10.0.2.13

while the hosts.deny file becomes

# /etc/hosts.deny
sshd: ALL

Then any ssh connection attempt from other than 10.0.2.3 or 10.0.2.13 will not even receive a response.

You are encouraged to read the manual page for hosts_access to learn more about the syntax for these files, and to learn other options.

Windows File Sharing

Microsoft Windows operating systems are designed to allow users to share files and folders.

Creating a (SMB) file share (Windows 8)

Creating a file share for a Windows 8 system connected to a domain is actually quite simple. Suppose that the domain user Marcus Cole has a folder called "Shared Folder" in the Documents Library. To share that folder, simply navigate to that folder and select Share, then Specific People. You can then grant permissions to user(s) and group(s) defined on the domain controller. So, for example we can give particular domain members permission to read the directory, and all members of the sales admin group (which we created in our discussion of group policy) both read and write permissions.
Windows 8 (Drakh)-2014-02-14-16-02-17

Creating a (SMB) file share (Windows 7)

Create a folder; navigate Folder (right click) → Properties → Sharing → Sharing. Specify the Share name, and the permissions of the users allowed to access the share. One apparent difference is that an administrator password was required for Windows 7 but not for Windows 8 to enable the share.

Setting up a file server (Windows 2012)

In most large organizations however, file shares are provided centrally as part of the overall network infrastructure and run from a file server.

In this example, we will set up a Windows 2012 system as a file server. This system will be a domain member, but will not be a domain controller. In this example, the system will be named minbari.corp.class.tu; it will be a member of the domain corp.class.tu, which has brakiri.corp.class.tu as the domain controller; the system will live at the IP address 10.0.6.50.

With the basics covered, go to initial configuration tasks, select Add Roles → File Services. In this example, we install both the File Server and the File Server Resource Manager.
Addfileserver

Installation may take a bit of time.

Once the installation completes, we need to create the shared directory that the approved domain users can use. From the Server Manager, select File and Storage Services, then select Shares.
Windows 2012 (File Server) Minbar-2014-02-14-17-21-37

From Tasks, select New Share. Two kinds of shares are available, SMB shares, which are a Windows standard, and NFS shares, which are an older *nix standard. For simplicity, let’s select a quick SMB share.

Once selected, you first select the volume that will be used for the shares; the default c: is appropriate here. Moving forward, we name the share "CommonShare" and accept the default local path and remote paths.
SharePath

Leave the other settings in their default settings; we will now enable access-based enumeration and we will not encrypt data access.

Next, and more important we need to specify the permissions that govern access. This process is greatly simplified from the settings in Windows 2008. Simply select Customize permissions from the menu. You can then customize permissions as needed from the Permissions tab. If we want to set up a common share for all domain users, we select Add, then select domain users, and grant Modify permissions, which includes read, write, and execute.
Windows 2012 (File Server) Minbar-2014-02-14-21-25-21

After the settings are confirmed, the share will be created.

The shares can be accessed from a Windows system by navigating to minbari\Commonshare
FileShare7

They can be mapped to a drive letter via the net use command

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\Users\zack>net use z: \\minbari\CommonShare
The command completed successfully.


C:\Users\zack>z:

Z:\>dir
 Volume in drive Z has no label.
 Volume Serial Number is E627-301D

 Directory of Z:\

02/14/2014  09:31 PM              .
02/14/2014  09:31 PM              ..
02/14/2014  09:31 PM                18 Test.txt.txt
               1 File(s)             18 bytes
               2 Dir(s)  52,669,808,640 bytes free

You can also access the file share from Linux systems. Suppose that you are on a CentOS system joined to the domain; then from Places, select Connect to Server. Add the name of the server and the name of the share; then select Connect.Screenshot-Connect to Server

A network shared folder will appear on the Desktop, and you can browse the folder quite simply.

The same process works on a Mint Cinnamon system. More generally, most Linux systems accept an address of smb://minbari/CommonShare in their file browser. In Ubuntu, you can startby browsing folders, then press ctrl+L to enter the name of the place.
Ubuntu 12.04 (x86) Hyach-2014-02-14-22-15-47
This will bring the user to the shared folder.

Automatic Drive Mappings via Group Policy

We would like each user to have a drive letter mapped to the common file share just created; this is done through Group Policy.

On a domain controller, as a domain administrator go to group policy management and create a new group policy object- say “Common Drive Mapping”. Edit that policy: Common Drive Mapping → User Configuration → Preferences → Windows Settings → Drive Maps.
Windows 2012 (AD) Brakiri-2014-02-14-22-26-51

In the Drive Maps window, right click and select New → Mapped Drive. For Action, select “Create”; for Location, select the file share created previously. Be sure that you include both the host name and the directory for your share. Select a label for the drive share; select a drive letter- say P:\. Apply the result.
NewDrive

Exit the group policy editor (with our policy now saved).

Return to Group Policy Management; select Group Policy Management → Forest → Domains → “YourDomainNameWhateverItIs” (Right click) → Link an existing GPO → Select your new GPO, which this example calls “Common Drive Mapping”. Apply this policy.

LinkedPOlicies

You do not need to apply this policy to the entire domain; you could simply apply it to one or more OUs.

Log onto a machine in your domain; verify that the drive P:\ is now mapped to the file share.

Individual file shares

We can also set up a directory to provide an individual file share for each domain user. We start by creating the file share itself, so we head back to the file server, and from Server Manager make our way back to Shares. From Tasks, select New Share, and once again we will use the profile SMB Share - Quick. We’ll keep the same share location on the C: drive, but now the share name will be UserData; then the local path to the shared data is c:\Shares\UserData while the remote path will be \\minbari\UserData. Again, we leave the other settings in their default values, so we don’t enable access-based enumeration or encrypt data access.

Once we get to the permissions section, we will diverge from our experience with the CommonShare. We Customize Permissions, and select Disable inheritance; convert all inherited permissions into explicit ones.
Windows 2012 (File Server) Minbar-2014-02-14-23-05-35

Next, we remove both sets of permissions for MINBARI\Users, but add permissions for Domain Admins, giving them Full Control. We end up with a permission set like the following:
Permissions

With these changes completed, create the share.

With the share complete, we now move to the individual users. From Server Manager on a domain controller, choose Tools, then navigate to Active Directory Users and Computers. Select a domain user, then select Properties. From the dialog box that results, choose the Profile tab. Under Home Folder, choose a drive letter and then provide the path to the file share we just created- with one additional twist. We will actually provide a link to a subdirectory based on the account user name; we can even simply specify %username in the directory name:
Windows 2012 (AD) Brakiri-2014-02-14-23-12-50
We repeat this process for each user.

When complete, we can simply login as a user, and verify that both drives are mapped to the correct places:
Shares

When you complete these tasks, you should also verify that you are unable to access the file shares of other users.

I would love to say that this process is painless, but it is not, as for some unknown reason it takes some time for information about the new file shares to propagate out. If you log in and the shared drives are not there, you may need to reboot the system.

Remote Desktop

If you want to gain graphical access to a Windows system, one approach is to use the Remote Desktop Protocol. Enabling an RDP server is quite simple. On a Windows 8 system as an example, from the Control Panel, navigate to System and Security, then to System. Select Remote Settings and pass through UAC. From the Remote tab, simply allow remote connections to the computer.
RDP
One nice feature is that this will automatically open the proper ports in the firewall, at least for users on the domain and on private networks. In case you forgot, RDP uses TCP/3389 and UDP/3389.

The situation on server systems is essentially the same.

To connect to a remote Windows system, you can use the Microsoft Terminal Services Client; and you can launch it by running mstsc.exe. Simply enter the name of the computer to which you want to connect, then provide the appropriate credentials when asked.
mstsc

A very simple client for Mint and Ubuntu systems is available, called rdesktop; I have made it available on the lab share. Install it via dpkg:

valen@hyach:~$ sudo dpkg -i /home/valen/Desktop/rdesktop_1.7.0-1ubuntu2_i386.deb

To use it, simply provide the hostname on the command line and then authenticate:
Ubuntu 12.04 (x86) Hyach-2014-02-16-14-53-29

A client is available for CentOS systems; you can install it via yum in the usual fashion

[root@gaim ~]# yum --disablerepo=\* --enablerepo=c6-media install rdesktop

where the CentOS system has presumably been configured as we did for BIND to use the CD as a repo. If you are online, a simple yum install rdesktop will work.

Operational Awareness in Linux

By running a service like SSH, you are now explicitly allowing other people to access your system. To do so safely, you need to know what those users are doing on your system; this is the concept of operational awareness.

Linux provides a number of tools to give you, the system administrator, information about what is occurring on your system.

psacct

There is a package that allows for much more detailed logging of what is being done on a Linux system called psacct or acct depending on the particular distribution

The tool is already installed on your CentOS virtual machine, though it is not enabled. Called psacct it is controlled in the usual fashion. The command

[root@gaim ~]# /etc/init.d/psacct start
Starting process accounting:                               [  OK  ]

will start and stop the service, and we can ensure it starts on boot either via chkconfig or by navigating System → Administration → Services.
Screenshot-Service Configuration

For Ubuntu or Mint systems the package is called acct, and for Ubuntu 12.04 / Mint 13 we use version 6.5.5-1ubuntu1. You can get the software from the labshare or directly from the Ubuntu software repository.
Install it in the usual fashion.

valen@markab ~ $ sudo dpkg -i /home/valen/Desktop/acct_6.5.5-1ubuntu1_amd64.deb [sudo] password for valen: 
Selecting previously unselected package acct.
(Reading database ... 141706 files and directories currently installed.)
Unpacking acct (from .../acct_6.5.5-1ubuntu1_amd64.deb) ...
Setting up acct (6.5.5-1ubuntu1) ...
Turning on process accounting, file set to '/var/log/account/pacct'.
 * Done.
Processing triggers for doc-base ...
Processing 30 changed doc-base files, 1 added doc-base file...
Registering documents with scrollkeeper...
Processing triggers for install-info ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot

Ubuntu and Mint set up the package so that it will start and will start on the system’s next boot.

Using psacct

The psacct package keeps track of a number of pieces of system information. For example, the command ac can be used to determine how may hours of connect time have been used.

CORP\mcole@markab ~ $ ac
	total       10.30

This shows that users have been connected to the system for 10.30 hours. If we want to see that number broken down by user, we can run

CORP\mcole@markab ~ $ ac -p
	zathras                              0.39
	valen                                5.56
	CORP\zack                            0.04
	CORP\mcole                           0.44
	CORP\ivanova                         3.87
	total       10.31

to see that essentially all of the time has been from the domain users valen amd ivanova, with the other users contributing essentially nothing to the total. We can use ac -d

CORP\mcole@markab ~ $ ac -d
Feb  1	total        6.17
Feb  7	total        2.88
Feb 14	total        0.04
Today	total        1.31

to get a breakdown by date; we can even combine them as ac -dp

CORP\mcole@markab ~ $ ac -dp
	valen                                1.91
	CORP\ivanova                         3.87
	zathras                              0.39
Feb  1	total        6.17
	valen                                2.88
Feb  7	total        2.88
	CORP\zack                            0.04
Feb 14	total        0.04
	valen                                0.77
	CORP\mcole                           0.55
Today	total        1.32

Another useful feature of this package is the ability to determine the commands that have been run on the system. For example, to find out each time the command su has been run, you can execute

CORP\mcole@markab ~ $ sudo lastcomm su
[sudo] password for CORP\mcole: 
su               S     CORP\zac __         0.00 secs Sat Feb 15 09:37
su               S     valen    __         0.00 secs Sat Feb 15 09:34
su                     CORP\mco pts/1      0.00 secs Sat Feb 15 09:37
su                     CORP\mco pts/0      0.00 secs Sat Feb 15 09:34

Note that this command requires root privileges or su. Running

CORP\mcole@markab ~ $ sudo lastcomm CORP\\mcole
bash              F    CORP\mco pts/0      0.00 secs Sat Feb 15 09:51
python                 CORP\mco pts/0      0.05 secs Sat Feb 15 09:51
sudo             S     CORP\mco pts/0      0.00 secs Sat Feb 15 09:51
sudo             S     CORP\mco pts/0      0.00 secs Sat Feb 15 09:51
bash              F    CORP\mco pts/0      0.00 secs Sat Feb 15 09:51
python                 CORP\mco pts/0      0.07 secs Sat Feb 15 09:51
sudo             S     CORP\mco pts/0      0.00 secs Sat Feb 15 09:49
lastcomm               CORP\mco pts/0      0.00 secs Sat Feb 15 09:49

we find the command recently executed by the domain user mcole; note the use of a pair of backslashes (which are normally used as escapes in bash). You can read the man page for lastcomm to determine the significance of the flags (e.g. the S that occasionally appears.)

If you want to find out information about previously run commands, you can use the sa tool; it is able to give you information on the cpu usage of all of the commands run on the system. See the man page for more details.

Who is Logged on the System?

The command w can be used to determine who is logged into the system.

CORP\mcole@markab ~ $ w
 09:59:10 up 52 min,  5 users,  load average: 0.31, 0.25, 0.18
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
CORP\mco tty9     :0               09:39   52:08   2.55s  0.27s gnome-session -
CORP\mco pts/0    :0.0             09:39    0.00s  0.14s  0.00s w
CORP\zac pts/1    hyach.corp.class 09:56    3:08   0.09s  0.09s -bash
CORP\mco pts/2    :0.0             09:57    1:13   0.09s  0.09s bash
valen    pts/3    10.0.6.101       09:59    4.00s  0.08s  0.08s -bash

Here we see three users logged in (CORP\zack, CORP\mcole) and valen, with two SSH logins from two different hosts.

Similar information is provided by the command who:

CORP\mcole@markab ~ $ who
CORP\mcole tty9         2014-02-15 09:39 (:0)
CORP\mcole pts/0        2014-02-15 09:39 (:0.0)
CORP\zack pts/1        2014-02-15 09:56 (hyach.corp.class.tu)
CORP\mcole pts/2        2014-02-15 09:57 (:0.0)
valen    pts/3        2014-02-15 09:59 (10.0.6.101)

without the truncation of the user names.

The data these commands rely on is stored in the file /var/run/utmp.

You may also want to know not who is currently logged in, but rather the users that have logged in recently. This information is available through the command last

CORP\mcole@markab ~ $ last
valen    pts/3        10.0.6.101       Sat Feb 15 09:59   still logged in   
CORP\mco pts/2        :0.0             Sat Feb 15 09:57   still logged in   
CORP\zac pts/1        hyach.corp.class Sat Feb 15 09:56   still logged in   
CORP\mco pts/0        :0.0             Sat Feb 15 09:39   still logged in   
CORP\mco tty9         :0               Sat Feb 15 09:39   still logged in   
CORP\mco pts/1        :0.0             Sat Feb 15 09:37 - 09:38  (00:01)    
CORP\mco pts/1        :0.0             Sat Feb 15 09:36 - 09:37  (00:00)    
CORP\mco pts/0        :0.0             Sat Feb 15 09:34 - 09:38  (00:04)    
CORP\mco tty9         :0               Sat Feb 15 09:34 - 09:38  (00:04)    
valen    pts/0        :0.0             Sat Feb 15 09:11 - 09:34  (00:22) 
....

This data is stored in the file /var/log/wtmp, which is why attackers often clobber that file before departing a system.

Remember that if commands are executed via a bash shell, then the list of past commands for a user is stored in the file ~/.bash_history. Thus, if you want to see the last 15 bash commands executed by the user , you can simply check the list

CORP\mcole@markab ~ $ sudo tail -n4 /home/local/CORP/zack/.bash_history 
sudo ls
pwd
vi TextFile
ifconfig

Remember though, that this file is written when the user exits their shell, so it won’t have the history of commands in a currently running shell.

What Processes are Running on the System?

If you want a real-time list of processes running on your system you can use the command top; here is the output of that program at a particular moment.

top - 10:07:34 up  1:00,  4 users,  load average: 0.09, 0.10, 0.13
Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1012348k total,   785476k used,   226872k free,    52424k buffers
Swap:  1046524k total,     4832k used,  1041692k free,   410880k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 4469 root      20   0  176m  41m  10m S  1.3  4.2   0:03.66 Xorg               
 4792 CORP\mco  20   0  556m  19m  12m S  1.3  1.9   0:02.36 gnome-terminal     
 4491 CORP\mco  20   0  381m  10m 7840 S  0.3  1.1   0:00.40 gnome-session      
 4617 CORP\mco  20   0  590m  22m  12m S  0.3  2.3   0:01.17 gnome-panel        
 4628 CORP\mco  20   0  586m  16m  11m S  0.3  1.7   0:00.35 nm-applet          
 4632 CORP\mco  20   0  465m  12m 8852 S  0.3  1.3   0:00.30 bluetooth-apple    
 4635 CORP\mco  20   0  978m  27m  17m S  0.3  2.7   0:00.81 nautilus           
 4648 CORP\mco  20   0  479m  15m  10m S  0.3  1.6   0:00.51 indicator-apple    
 4992 root      20   0     0    0    0 S  0.3  0.0   0:00.28 kworker/0:0        
    1 root      20   0 36132 2856 1972 S  0.0  0.3   0:00.78 init               
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd           
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.57 ksoftirqd/0        
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0        
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.01 watchdog/0      

The processes are listed in order, with the processes using most CPU listed at the top. If your system is slow or sluggish due to heavy load, this is the place to start diagnosing the problem.

If you want to know more about what your processes are up to, then you want to use the ps command. This is full of command switches that can give you some very different behaviors. If you want to know all of the processes running on a system, run

CORP\mcole@markab ~ $ ps aux 
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.2  36132  2856 ?        Ss   09:06   0:00 /sbin/init
root         2  0.0  0.0      0     0 ?        S    09:06   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    09:06   0:00 [ksoftirqd/0]
root         6  0.0  0.0      0     0 ?        S    09:06   0:00 [migration/0]
root         7  0.0  0.0      0     0 ?        S    09:06   0:00 [watchdog/0]
root         8  0.0  0.0      0     0 ?        S<   09:06   0:00 [cpuset]
root         9  0.0  0.0      0     0 ?        S<   09:06   0:00 [khelper]
root        10  0.0  0.0      0     0 ?        S    09:06   0:00 [kdevtmpfs]
root        11  0.0  0.0      0     0 ?        S<   09:06   0:00 [netns]
root        12  0.0  0.0      0     0 ?        S    09:06   0:00 [sync_supers]
root        13  0.0  0.0      0     0 ?        S    09:06   0:00 [bdi-default]
...

This gives so much information, it is often hides what you want to know. Suppose you want to find out if there are any SSH processes running; then you can pipe the output of the ps command directly to a grep to to some filtering:

CORP\mcole@markab ~ $ ps aux | grep sshd
root      5272  0.0  0.2  49948  2868 ?        Ss   09:55   0:00 /usr/sbin/sshd 
-D
root      5556  0.0  0.4  99760  4972 ?        Ss   09:59   0:00 sshd: valen 
[priv]  
valen     5575  0.0  0.2 100808  2764 ?        S    09:59   0:00 sshd: 
valen@pts/3   
372770881 5907  0.0  0.0  13576   932 pts/0    S+   10:12   0:00 grep --
colour=auto sshd

Now we see SSH related processes from root, seldon, mallow, and hardin.

If you want to see the structure, showing which process spawned another, you can use either the -H or --forest switches:

CORP\mcole@markab ~ $ ps aux --forest
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
....
372770881 4792  0.1  1.9 569576 19664 ?        Sl   09:39   0:02 gnome-terminal
372770881 4797  0.0  0.1  30704  1992 ?        S    09:39   0:00  \_ gnome-pty-h
372770881 4798  0.0  0.5  41352  5700 pts/0    Ss   09:39   0:00  \_ bash
372770881 5921  0.0  0.2  38524  2456 pts/0    R+   10:13   0:00  |   \_ ps aux
372770881 5501  0.0  0.5  41256  5496 pts/2    Ss+  09:57   0:00  \_ bash
372770881 4863  0.0  0.6 304640  6340 ?        Sl   09:39   0:00 /usr/bin/zeitge
372770881 4872  0.0  1.0 259544 10292 ?        Sl   09:39   0:00 /usr/lib/zeitge
372770881 4879  0.0  0.0  11372   612 ?        S    09:39   0:00  \_ /bin/cat
372770881 5112  0.0  0.2  46140  2156 ?        S    09:55   0:00 /usr/lib/gvfs/g
root      5272  0.0  0.2  49948  2868 ?        Ss   09:55   0:00 /usr/sbin/sshd
root      5556  0.0  0.4  99760  4972 ?        Ss   09:59   0:00  \_ sshd: valen
valen     5575  0.0  0.2 100808  2764 ?        S    09:59   0:00      \_ sshd: v
valen     5576  0.0  0.4  27368  4352 pts/3    Ss+  09:59   0:00          \_ -ba

Notice that many users are replaced by their UID; this is typical for users on a Linux system connected to a Windows domain. To find the username that corresponds to a uid, you can try

CORP\mcole@markab ~ $ getent passwd 372770881
CORP\mcole:x:372770881:372769281:Marcus Cole:/home/local/CORP
/mcole:/bin/bash

while if you want to go the other way, use id

CORP\mcole@markab ~ $ id CORP\\mcole
uid=372770881(CORP\mcole) gid=372769281(CORP\domain^users) groups=372769281
(CORP\domain^users),372769280(CORP\domain^admins),372769340(CORP\denied^rod
c^password^replication^group)
What Ports are Open on the System?

The command to determine what ports are open on the system is called netstat. When using netstat though, you should remember that Linux and Unix systems have two kinds of ports- network ports and Unix sockets. Unix sockets are used for communication by different processes on the same system, so in general we are uninterested in those.

The tool has a number of useful flags, including

  • -v Be verbose
  • -n Use numeric values for ports, rather than names
  • -A inet (or –inet) Show only IP (v4) connections
  • -A inet6 (or –inet6) Show only IPv6 connections
  • -x Show only Unix sockets
  • -t Show only TCP (v4/v6)
  • -u Show only UDP (v4/v6)
  • -p Show the PID for that connection
  • -l Show listening sockets (not shown by default)
  • -a Show listening and open sockets
  • -r Show routing table

If you want to find out what is listening on your system, a good set of flags is

CORP\mcole@markab ~ $ sudo netstat -nlpv --inet
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address   Foreign Address State   PID/Program name
tcp        0      0 0.0.0.0:139      0.0.0.0:*      LISTEN  564/smbd        
tcp        0      0 127.0.0.1:53     0.0.0.0:*      LISTEN  842/dnsmasq     
tcp        0      0 0.0.0.0:22       0.0.0.0:*      LISTEN  5272/sshd       
tcp        0      0 127.0.0.1:631    0.0.0.0:*      LISTEN  652/cupsd       
tcp        0      0 0.0.0.0:445      0.0.0.0:*      LISTEN  564/smbd        
udp        0      0 127.0.0.1:53     0.0.0.0:*              842/dnsmasq     
udp        0      0 0.0.0.0:44614    0.0.0.0:*              632/avahi-daemon: r
udp        0      0 10.0.255.255:137 0.0.0.0:*              764/nmbd        
udp        0      0 10.0.6.103:137   0.0.0.0:*              764/nmbd        
udp        0      0 0.0.0.0:137      0.0.0.0:*              764/nmbd        
udp        0      0 10.0.255.255:138 0.0.0.0:*              764/nmbd        
udp        0      0 10.0.6.103:138   0.0.0.0:*              764/nmbd        
udp        0      0 0.0.0.0:138      0.0.0.0:*              764/nmbd        
udp        0      0 0.0.0.0:5353     0.0.0.0:*              632/avahi-daemon: r

Here we were able both TCP and UDP ports in numeric form, and we also know the name & PID of the process that opened the port.

What Resources are Opened by What Process?

You can ask a broader question, namely what resources are being used by the system. These resources can be network sockets, but they can also be devices (like your USB drive) or files. The tool here is called lsof, and it can do a lot.

Suppose we wanted to get the same kind of information we saw with netstat; we can then try

CORP\mcole@markab ~ $ sudo lsof -i 4
COMMAND    PID       USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
smbd       564       root   29u  IPv4   8429      0t0  TCP *:microsoft-ds (LISTEN)
smbd       564       root   30u  IPv4   8431      0t0  TCP *:netbios-ssn (LISTEN)
avahi-dae  632      avahi   12u  IPv4   8627      0t0  UDP *:mdns 
avahi-dae  632      avahi   14u  IPv4   8629      0t0  UDP *:44614 
cupsd      652       root   10u  IPv4  17213      0t0  TCP markab.corp.class.tu:ipp (LISTEN)
nmbd       764       root    9u  IPv4   9015      0t0  UDP *:netbios-ns 
nmbd       764       root   10u  IPv4   9016      0t0  UDP *:netbios-dgm 
nmbd       764       root   11u  IPv4   9018      0t0  UDP markab.corp.class.tu:netbios-ns 
nmbd       764       root   12u  IPv4   9019      0t0  UDP 10.0.255.255:netbios-ns 
nmbd       764       root   13u  IPv4   9020      0t0  UDP markab.corp.class.tu:netbios-dgm 
nmbd       764       root   14u  IPv4   9021      0t0  UDP 10.0.255.255:netbios-dgm 
dnsmasq    842     nobody    4w  IPv4   9088      0t0  UDP markab.corp.class.tu:domain 
dnsmasq    842     nobody    5u  IPv4   9089      0t0  TCP markab.corp.class.tu:domain (LISTEN)
lwsmd     1428       root   64u  IPv4  32658      0t0  TCP markab.corp.class.tu:38540->brakiri.corp.class.tu:ldap (ESTABLISHED)
ubuntu-ge 4703 CORP\mcole    9u  IPv4  25078      0t0  TCP markab.corp.class.tu:37864->mistletoe.canonical.com:http (CLOSE_WAIT)
sshd      5272       root    3r  IPv4  28471      0t0  TCP *:ssh (LISTEN)
sshd      5556       root    3r  IPv4  29166      0t0  TCP markab.corp.class.tu:ssh->drazi.corp.class.tu:34104 (ESTABLISHED)
sshd      5575      valen    3u  IPv4  29166      0t0  TCP markab.corp.class.tu:ssh->drazi.corp.class.tu:34104 (ESTABLISHED)

Here the flag -i 4 asks for information about all IPv4 connections; we can ask for IPv6 with -i 6

CORP\mcole@markab ~ $ sudo lsof -i 6
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
smbd       564  root   27u  IPv6   8424      0t0  TCP *:microsoft-ds (LISTEN)
smbd       564  root   28u  IPv6   8427      0t0  TCP *:netbios-ssn (LISTEN)
avahi-dae  632 avahi   13u  IPv6   8628      0t0  UDP *:mdns 
avahi-dae  632 avahi   15u  IPv6   8630      0t0  UDP *:55798 
cupsd      652  root    9u  IPv6  17212      0t0  TCP ip6-localhost:ipp (LISTEN)
sshd      5272  root    4u  IPv6  28475      0t0  TCP *:ssh (LISTEN)

If you want to know about TCP connections, you can use -i TCP, and if you want just port 22, then the flag would be -i TCP:22

CORP\mcole@markab ~ $ sudo lsof -i TCP:22
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    5272  root    3r  IPv4  28471      0t0  TCP *:ssh (LISTEN)
sshd    5272  root    4u  IPv6  28475      0t0  TCP *:ssh (LISTEN)
sshd    5556  root    3r  IPv4  29166      0t0  TCP markab.corp.class.tu:ssh
->drazi.corp.class.tu:34104 (ESTABLISHED)
sshd    5575 valen    3u  IPv4  29166      0t0  TCP markab.corp.class.tu:ssh
->drazi.corp.class.tu:34104 (ESTABLISHED)

Suppose you are concerned that a connection from 10.0.6.104 might be hostile. Can you find out what other connections come from that host? Sure!

CORP\mcole@markab ~ $ sudo lsof -i @10.0.6.104
COMMAND  PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    6278      root    3r  IPv4  35409      0t0  TCP 
markab.corp.class.tu:ssh->hyach.corp.class.tu:50795 (ESTABLISHED)
sshd    6280 CORP\zack    3u  IPv4  35409      0t0  TCP 
markab.corp.class.tu:ssh->hyach.corp.class.tu:50795 (ESTABLISHED)

If you prefer numeric port numbers, add -P

CORP\mcole@markab ~ $ sudo lsof -i @10.0.6.104 -P
COMMAND  PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    6278      root    3r  IPv4  35409      0t0  TCP 
markab.corp.class.tu:22->hyach.corp.class.tu:50795 (ESTABLISHED)
sshd    6280 CORP\zack    3u  IPv4  35409      0t0  TCP 
markab.corp.class.tu:22->hyach.corp.class.tu:50795 (ESTABLISHED)

We can do more than just examine network connections with lsof. Suppose that you want to see what resources have been opened by all commands that contain the text "ssh"; then we use the -c flag; just be prepared for a lot of data.

CORP\mcole@markab ~ $ sudo lsof -c sshd | more
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system 
/home/local/CORP/mcole/.gvfs
      Output information may be incomplete.
COMMAND  PID      USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
sshd    5272      root  cwd    DIR                8,1     4096       2 /
sshd    5272      root  rtd    DIR                8,1     4096       2 /
sshd    5272      root  txt    REG                8,1   517112 1049446 /usr
/sbin/sshd
sshd    5272      root  mem    REG                8,1    52120  921880 /lib
/x86_64-linux-gnu/libnss_files-2.15.so
sshd    5272      root  mem    REG                8,1    47680  921884 /lib
/x86_64-linux-gnu/libnss_nis-2.15.so
sshd    5272      root  mem    REG                8,1    35680  921876 /lib
/x86_64-linux-gnu/libnss_compat-2.15.so
sshd    5272      root  mem    REG                8,1   105288  921913 /lib
/x86_64-linux-gnu/libresolv-2.15.so
sshd    5272      root  mem    REG                8,1    14360  921860 /lib
/x86_64-linux-gnu/libkeyutils.so.1.4
sshd    5272      root  mem    REG                8,1    31200 1058091 /usr
/lib/x86_64-linux-gnu/libkrb5support.so.0.1
sshd    5272      root  mem    REG                8,1   158168 1058085 /usr
/lib/x86_64-linux-gnu/libk5crypto.so.3.1
sshd    5272      root  mem    REG                8,1    14768  921842 /lib
/x86_64-linux-gnu/libdl-2.15.so
sshd    5272      root  mem    REG                8,1    97248  921874 /lib
/x86_64-linux-gnu/libnsl-2.15.so
--More--

If you want to know all of the resources opened by a particular user, just specify the UID of the user

CORP\mcole@markab ~ $ sudo lsof -u CORP\\zack | more
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system 
/home/local/CORP/mcole/.gvfs
      Output information may be incomplete.
COMMAND  PID      USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
sshd    6280 CORP\zack  cwd    DIR                8,1     4096       2 /
sshd    6280 CORP\zack  rtd    DIR                8,1     4096       2 /
sshd    6280 CORP\zack  txt    REG                8,1   517112 1049446 /usr
/sbin/sshd
sshd    6280 CORP\zack  DEL    REG                0,4            35464 /dev
/zero
sshd    6280 CORP\zack  mem    REG                8,1    43480  921623 /lib
/security/pam_gnome_keyring.so
sshd    6280 CORP\zack  mem    REG                8,1    18704 1049844 /lib
/x86_64-linux-gnu/security/pam_limits.so
sshd    6280 CORP\zack  mem    REG                8,1    10240 1049848 /lib
/x86_64-linux-gnu/security/pam_mail.so
sshd    6280 CORP\zack  mem    REG                8,1    10272 1049850 /lib
/x86_64-linux-gnu/security/pam_motd.so
sshd    6280 CORP\zack  mem    REG                8,1    14472 1053171 /usr
/lib/libck-connector.so.0.0.0
sshd    6280 CORP\zack  mem    REG                8,1   276392  921841 /lib
/x86_64-linux-gnu/libdbus-1.so.3.5.8
sshd    6280 CORP\zack  mem    REG                8,1    10296  921621 /lib
/security/pam_ck_connector.so
sshd    6280 CORP\zack  mem    REG                8,1    10304 1049867 /lib
/x86_64-linux-gnu/security/pam_umask.so
--More--

You can do the same by PID, just with the -p flag.

What do we Know about a Process?

Suppose you see a suspicious process. What can you quickly determine about it? A great deal of information about that process is stored in the /proc directory, in the subdirectory named after that PID.

For example, suppose we are interested in the process with PID 6460. We first change to the proper directory

CORP\mcole@markab ~ $ cd /proc/6460
CORP\mcole@markab /proc/6460 $ ls
attr             cpuset   limits      ns             sched      syscall
autogroup        cwd      loginuid    numa_maps      schedstat  task
auxv             environ  maps        oom_adj        sessionid  wchan
cgroup           exe      mem         oom_score      smaps
clear_refs       fd       mountinfo   oom_score_adj  stack
cmdline          fdinfo   mounts      pagemap        stat
comm             io       mountstats  personality    statm
coredump_filter  latency  net         root           status

We can find the command line used to start the process:

CORP\mcole@markab /proc/6460 $ cat cmdline 
vi./TextFile

The current working directory is stored in cwd and the environment variables in environ

CORP\mcole@markab /proc/6460 $ sudo ls -l cwd
lrwxrwxrwx 1 CORP\zack CORP\domain^users 0 Feb 15 10:52 cwd -> 
/home/local/CORP/zack CORP\mcole@markab /proc/6460 
$ sudo cat  environ 
TERM=xtermSHELL=/bin/bashSSH_CLIENT=10.0.6.104 50795 22SSH_TTY=/dev/pts/1
USER=CORP\zackMAIL=/var/mail/CORP\zackPATH=/usr/local/sbin:/usr/local/bin
:/usr/sbin:/usr/bin:/sbin:/bin:/usr/gamesPWD=/home/local/CORP/zackLANG=en
_US.UTF-8SHLVL=1HOME=/home/local/CORP/zackLOGNAME=CORP\zackSSH_CONNECTION
=10.0.6.104 50795 10.0.6.103 22_=/usr/bin/vi

To find the precise executable, we check exe, and if we want to see what files are opened, we can see which file descriptor points where by checking fd

CORP\mcole@markab /proc/6460 $ sudo ls -l exe
lrwxrwxrwx 1 CORP\zack CORP\domain^users 0 Feb 15 10:52 exe -> /usr/bin/vim.tiny
CORP\mcole@markab /proc/6460 $ sudo ls -l fd
total 0
lrwx------ 1 CORP\zack CORP\domain^users 64 Feb 15 10:56 0 -> /dev/pts/1
lrwx------ 1 CORP\zack CORP\domain^users 64 Feb 15 10:56 1 -> /dev/pts/1
lrwx------ 1 CORP\zack CORP\domain^users 64 Feb 15 10:56 2 -> /dev/pts/1
lrwx------ 1 CORP\zack CORP\domain^users 64 Feb 15 10:56 3 -> socket:[36731]
lrwx------ 1 CORP\zack CORP\domain^users 64 Feb 15 10:56 5 -> /home/local/CO
RP/zack/.TextFile.swp

Information about the memory map for the process is held in maps, while
/proc/PID/net/ is a directory of network information

With all of this information, can you determine what PID 15895 was doing?

Operational Awareness in Windows

As in the case of our Linux system, it is important to know what is occurring on our system.

Netstat

Like Linux systems, Windows systems also come with a tool called netstat. The tools are similar, though the options are quite different. To have netstat show all of the listenting ports, use the -a switch; to have the ports displayed in numeric form use -n and to include the PID of the process that opened the port, use -o. On our Windows File server, this results in a set of output like the following

C:\Users\abester>netstat -ano

Active Connections

  Proto  Local Address          Foreign Address        State           PID
  TCP    0.0.0.0:135            0.0.0.0:0              LISTENING       884
  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING       4
  TCP    0.0.0.0:8000           0.0.0.0:0              LISTENING       1960
  TCP    0.0.0.0:8089           0.0.0.0:0              LISTENING       1752
  TCP    0.0.0.0:49152          0.0.0.0:0              LISTENING       676
  TCP    0.0.0.0:49153          0.0.0.0:0              LISTENING       956
  TCP    0.0.0.0:49154          0.0.0.0:0              LISTENING       988
  TCP    0.0.0.0:49155          0.0.0.0:0              LISTENING       768
  TCP    0.0.0.0:49179          0.0.0.0:0              LISTENING       768
  TCP    0.0.0.0:49181          0.0.0.0:0              LISTENING       760
  TCP    10.0.6.100:139         0.0.0.0:0              LISTENING       4
  TCP    10.0.6.100:49187       10.0.6.101:9997        ESTABLISHED     1752
  TCP    10.0.6.100:49199       10.0.6.50:445          ESTABLISHED     4
  TCP    10.0.6.100:49224       184.29.105.33:80       CLOSE_WAIT      3320
  TCP    10.0.6.100:49274       72.247.8.72:80         CLOSE_WAIT      2920
  TCP    10.0.6.100:49336       23.3.109.48:80         ESTABLISHED     3984
  TCP    10.0.6.100:49339       23.3.109.18:80         ESTABLISHED     3984
  TCP    10.0.6.100:49340       23.3.109.48:80         ESTABLISHED     3984
  TCP    10.0.6.100:49341       23.3.109.18:80         ESTABLISHED     3984
  TCP    10.0.6.100:49342       23.3.109.48:80         TIME_WAIT       0
  TCP    10.0.6.100:49343       23.3.109.18:80         TIME_WAIT       0
  TCP    10.0.6.100:49348       168.63.124.173:80      ESTABLISHED     3984
  TCP    10.0.6.100:49357       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49359       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49361       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49363       10.0.6.101:9997        TIME_WAIT       0
  TCP    [::]:135               [::]:0                 LISTENING       884
  TCP    [::]:445               [::]:0                 LISTENING       4
  TCP    [::]:49152             [::]:0                 LISTENING       676
  TCP    [::]:49153             [::]:0                 LISTENING       956
  TCP    [::]:49154             [::]:0                 LISTENING       988
  TCP    [::]:49155             [::]:0                 LISTENING       768
  TCP    [::]:49179             [::]:0                 LISTENING       768
  TCP    [::]:49181             [::]:0                 LISTENING       760
  UDP    0.0.0.0:123            *:*                                    1092
  UDP    0.0.0.0:500            *:*                                    988
  UDP    0.0.0.0:4500           *:*                                    988
  UDP    0.0.0.0:5355           *:*                                    1276
  UDP    10.0.6.100:137         *:*                                    4
  UDP    10.0.6.100:138         *:*                                    4
  UDP    10.0.6.100:1900        *:*                                    2832
  UDP    10.0.6.100:52840       *:*                                    2832
  UDP    127.0.0.1:1900         *:*                                    2832
  UDP    127.0.0.1:52841        *:*                                    2832
  UDP    127.0.0.1:55319        *:*                                    3984
  UDP    127.0.0.1:60426        *:*                                    768
  UDP    127.0.0.1:61607        *:*                                    1276
  UDP    127.0.0.1:63587        *:*                                    988
  UDP    127.0.0.1:64122        *:*                                    3320
  UDP    [::]:123               *:*                                    1092
  UDP    [::]:500               *:*                                    988
  UDP    [::]:4500              *:*                                    988
  UDP    [::]:5355              *:*                                    1276
  UDP    [::1]:1900             *:*                                    2832
  UDP    [::1]:52839            *:*                                    2832
  UDP    [fe80::7045:880e:7eca:360b%12]:1900  *:*
     2832
  UDP    [fe80::7045:880e:7eca:360b%12]:52838  *:*
      2832

We see here that there are both IPv4 and IPv6 connections, with both TCP and UDP represented. If you just wanted, say TCP on IPv4, you could add the additional switch

C:\Users\abester>netstat -anop TCP

Active Connections

  Proto  Local Address          Foreign Address        State           PID
  TCP    0.0.0.0:135            0.0.0.0:0              LISTENING       884
  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING       4
  TCP    0.0.0.0:8000           0.0.0.0:0              LISTENING       1960
  TCP    0.0.0.0:8089           0.0.0.0:0              LISTENING       1752
  TCP    0.0.0.0:49152          0.0.0.0:0              LISTENING       676
  TCP    0.0.0.0:49153          0.0.0.0:0              LISTENING       956
  TCP    0.0.0.0:49154          0.0.0.0:0              LISTENING       988
  TCP    0.0.0.0:49155          0.0.0.0:0              LISTENING       768
  TCP    0.0.0.0:49179          0.0.0.0:0              LISTENING       768
  TCP    0.0.0.0:49181          0.0.0.0:0              LISTENING       760
  TCP    10.0.6.100:139         0.0.0.0:0              LISTENING       4
  TCP    10.0.6.100:49187       10.0.6.101:9997        ESTABLISHED     1752
  TCP    10.0.6.100:49199       10.0.6.50:445          ESTABLISHED     4
  TCP    10.0.6.100:49224       184.29.105.33:80       CLOSE_WAIT      3320
  TCP    10.0.6.100:49274       72.247.8.72:80         CLOSE_WAIT      2920
  TCP    10.0.6.100:49336       23.3.109.48:80         ESTABLISHED     3984
  TCP    10.0.6.100:49339       23.3.109.18:80         ESTABLISHED     3984
  TCP    10.0.6.100:49340       23.3.109.48:80         ESTABLISHED     3984
  TCP    10.0.6.100:49341       23.3.109.18:80         ESTABLISHED     3984
  TCP    10.0.6.100:49348       168.63.124.173:80      ESTABLISHED     3984
  TCP    10.0.6.100:49359       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49361       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49363       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49365       10.0.6.101:9997        TIME_WAIT       0

If you want to know the name of the process that opened the connection, you can add the -bswitch, however this requires administrator privileges.

C:\Windows\system32>netstat -anobp TCP

Active Connections

  Proto  Local Address          Foreign Address        State           PID
  TCP    0.0.0.0:135            0.0.0.0:0              LISTENING       884
  RpcSs
 [svchost.exe]
  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING       4
 Can not obtain ownership information
  TCP    0.0.0.0:8000           0.0.0.0:0              LISTENING       1960
 [splunkweb.exe]
  TCP    0.0.0.0:8089           0.0.0.0:0              LISTENING       1752
 [splunkd.exe]
  TCP    0.0.0.0:49152          0.0.0.0:0              LISTENING       676
 [wininit.exe]
  TCP    0.0.0.0:49153          0.0.0.0:0              LISTENING       956
  EventLog
 [svchost.exe]
  TCP    0.0.0.0:49154          0.0.0.0:0              LISTENING       988
  Schedule
 [svchost.exe]
  TCP    0.0.0.0:49155          0.0.0.0:0              LISTENING       768
 [lsass.exe]
  TCP    0.0.0.0:49179          0.0.0.0:0              LISTENING       768
 [lsass.exe]
  TCP    0.0.0.0:49181          0.0.0.0:0              LISTENING       760
 [services.exe]
  TCP    10.0.6.100:139         0.0.0.0:0              LISTENING       4
 Can not obtain ownership information
  TCP    10.0.6.100:49187       10.0.6.101:9997        ESTABLISHED     1752
 [splunkd.exe]
  TCP    10.0.6.100:49199       10.0.6.50:445          ESTABLISHED     4
 Can not obtain ownership information
  TCP    10.0.6.100:49224       184.29.105.33:80       CLOSE_WAIT      3320
 [wwahost.exe]
  TCP    10.0.6.100:49274       72.247.8.72:80         CLOSE_WAIT      2920
 [jusched.exe]
  TCP    10.0.6.100:49336       23.3.109.48:80         CLOSE_WAIT      3984
 [Explorer.EXE]
  TCP    10.0.6.100:49339       23.3.109.18:80         CLOSE_WAIT      3984
 [Explorer.EXE]
  TCP    10.0.6.100:49340       23.3.109.48:80         CLOSE_WAIT      3984
 [Explorer.EXE]
  TCP    10.0.6.100:49341       23.3.109.18:80         CLOSE_WAIT      3984
 [Explorer.EXE]
  TCP    10.0.6.100:49348       168.63.124.173:80      ESTABLISHED     3984
 [Explorer.EXE]
  TCP    10.0.6.100:49375       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49377       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49379       10.0.6.101:9997        TIME_WAIT       0
  TCP    10.0.6.100:49381       10.0.6.200:135         TIME_WAIT       0
  TCP    10.0.6.100:49382       10.0.6.200:49155       TIME_WAIT       0
  TCP    10.0.6.100:49383       23.3.109.48:80         ESTABLISHED     988
  ProfSvc
 [svchost.exe]
  TCP    10.0.6.100:49384       23.3.109.18:80         ESTABLISHED     988
 [svchost.exe]
  TCP    10.0.6.100:49385       10.0.6.101:9997        TIME_WAIT       0

Full details about some of the other netstat options can be found with the /? switch:

C:\Users\abester>netstat /?

Displays protocol statistics and current TCP/IP network connections.

NETSTAT [-a] [-b] [-e] [-f] [-n] [-o] [-p proto] [-r] [-s] [-t] [interval]

  -a            Displays all connections and listening ports.
  -b            Displays the executable involved in creating each connection or
                listening port. In some cases well-known executables host
                multiple independent components, and in these cases the
                sequence of components involved in creating the connection
                or listening port is displayed. In this case the executable
                name is in [] at the bottom, on top is the component it called,
                and so forth until TCP/IP was reached. Note that this option
                can be time-consuming and will fail unless you have sufficient
                permissions.
  -e            Displays Ethernet statistics. This may be combined with the -s
                option.
  -f            Displays Fully Qualified Domain Names (FQDN) for foreign
                addresses.
  -n            Displays addresses and port numbers in numerical form.
  -o            Displays the owning process ID associated with each connection.
  -p proto      Shows connections for the protocol specified by proto; proto
                may be any of: TCP, UDP, TCPv6, or UDPv6.  If used with the -s
                option to display per-protocol statistics, proto may be any of:
                IP, IPv6, ICMP, ICMPv6, TCP, TCPv6, UDP, or UDPv6.
  -r            Displays the routing table.
  -s            Displays per-protocol statistics.  By default, statistics are
                shown for IP, IPv6, ICMP, ICMPv6, TCP, TCPv6, UDP, and UDPv6;
                the -p option may be used to specify a subset of the default.
  -t            Displays the current connection offload state.
  interval      Redisplays selected statistics, pausing interval seconds
                between each display.  Press CTRL+C to stop redisplaying
                statistics.  If omitted, netstat will print the current
                configuration information once.
Process Explorer

Another useful tool is Process Explorer. This is not part of a standard Windows installation, but instead is available as a separate download as part of the Sysinternals suite. This tool provides a graphical view of the processes running on the system, with the ability to simply click to find more information. Here is what the tool looks like on our Windows Server with the lower pane enabled (CTRL+L)
ProcessExploirer

If you double click on a process, you can find out even more detail; if you double click on the jucheck process for example, you will be given a dialog box that lets you determine, for example, what TCP/IP connections that process has made. Here we see the process has connected to an Akamai server, almost certainly to tell me that my version of Java is old and out-of-date.
Windows 2012 (AD) Brakiri-2014-02-15-11-38-36

TCPView

If your interests are solely on the network connections being made to and from your system, you may want to try TCPView. This is another Sysinternals tool that provides just this information; it even highlights new connections for your attention (in green). Here is what the tool looks like while running on our File Server. You can see the NetBIOS session and directory services connections.
TCPView

Process Monitor

Process Monitor is another Sysinternals tool; it captures the different actions of the processes running on your system.
ProcMon

ProcDot

If you want to analyze a process in detail, a great tool is ProcDot. This tool does not gather information like Process Monitor does, but rather it displays the results in a useful format.

Installation of the tool is trivial, as it simply comes in the form of a .zipped executable; just be sure to run the 32-bit or 64-bit version as appropriate.

The tool does rely on a couple of other tools though:

  • GraphViz, which is a graphical visualization tool, and
  • WinDump, which is a Windows version of the venerable tcpdump.

When the tool is run for the first time, it will ask for paths to these tools; simply provide them.

To record process data, start Process Monitor, but be sure to modify the configuration as required for ProcDot; in particular be sure you

  • Disable "Show Resolved Network Addresses" from the Options menu
  • From the Options menu, select Column selection, and
    • Show Thread ID
    • Do’t show the Sequence number

You also need to record the packets; I prefer doing so with Wireshark.

Once the activity you wish to monitor has finished, save the packet caputure as a .pcap file, and the Process Monitor data as a .csv file.

Load both files into ProcDot, and select the Launcher button. It will take some time to ingest the data, but when it is complete, you will be able to select a process. Here is an example of the result; you’ll need to zoom in to get the details.
ProcDot
The key to note here, is that this is an Internet Explorer process; looking at the graph you can see that it visited a web site at 10.0.2.250 on port 8080, then again came back out to the same host but now on port 4444. Some work was done in the registry, then a new notepad.exe process was spawned and execution moved to that thread.

This is a classic example of a Metasploit attack on a browser; in this case the attack was MS12-063 Microsoft Internet Explorer execCommand Use-After-Free Vulnerability.

Though ProcDot is fantastic for evaluating a process, remember that Process Monitor collects a lot of data, far more that could reasonably be stored over a long period of time. It is well suited for analyzing a particular piece of malware, but it does not double as an intrusion detection system.

Logon Sessions

If you want to know the details of all of the users logged in to your system, you can use the Sysinternals tool LogonSessions. This is a command line tool; when run on the file server it gives information like the following:

c:\Users\mcole\Desktop\logonSessions-1.21>logonsessions.exe

Logonsesions v1.21
Copyright (C) 2004-2010 Bryce Cogswell and Mark Russinovich
Sysinternals - wwww.sysinternals.com


[0] Logon session 00000000:000003e7:
    User name:    CORP\MINBARI$
    Auth package: Negotiate
    Logon type:   (none)
    Session:      0
    Sid:          S-1-5-18
    Logon time:   2/15/2014 8:59:39 AM
    Logon server:
    DNS Domain:   corp.class.tu
    UPN:          MINBARI$@corp.class.tu

[1] Logon session 00000000:000003e5:
    User name:    NT AUTHORITY\LOCAL SERVICE
    Auth package: Negotiate
    Logon type:   Service
    Session:      0
    Sid:          S-1-5-19
    Logon time:   2/15/2014 8:59:55 AM
    Logon server:
    DNS Domain:
    UPN:

[2] Logon session 00000000:00010baf:
    User name:    Window Manager\DWM-1
    Auth package: Negotiate
    Logon type:   Interactive
    Session:      1
    Sid:          S-1-5-90-1
    Logon time:   2/15/2014 8:59:55 AM
    Logon server:
    DNS Domain:
    UPN:

[3] Logon session 00000000:00010bbf:
    User name:    Window Manager\DWM-1
    Auth package: Negotiate
    Logon type:   Interactive
    Session:      1
    Sid:          S-1-5-90-1
    Logon time:   2/15/2014 8:59:55 AM
    Logon server:
    DNS Domain:
    UPN:
...
PS Utilities

The next set of tools are the PS utilities. These are all command line tools that let you perform various tasks on different systems, provided you have the proper permissions. Here is a brief summary of the tools, taken from the Microsoft web page that describes them.

  • PsExec – execute processes remotely
  • PsFile – shows files opened remotely
  • PsGetSid – display the SID of a computer or a user
  • PsInfo – list information about a system
  • PsPing – measure network performance
  • PsKill – kill processes by name or process ID
  • PsList – list detailed information about processes
  • PsLoggedOn – see who’s logged on locally and via resource sharing
  • PsLogList – dump event log records
  • PsPasswd – changes account passwords
  • PsService – view and control services
  • PsShutdown – shuts down and optionally reboots a computer
  • PsSuspend – suspends processes
  • PsUptime – shows you how long a system has been running since its last reboot

The PSLoggedOn tool when run on the file server is able to clearly distinguish between who was logged in locally and who was accessing shares; this is an improvement over what we say with logonsessions which showed too much information.

c:\Users\mcole\Desktop\PSTools>PsLoggedon.exe

PsLoggedon v1.34 - See who's logged on
Copyright (C) 2000-2010 Mark Russinovich
Sysinternals - www.sysinternals.com

Users logged on locally:
     2/15/2014 11:41:48 AM      CORP\mcole

Users logged on via resource shares:
     2/15/2014 2:51:15 PM       CORP\mcole
     2/15/2014 2:45:36 PM       (null)\mcole
     2/15/2014 2:51:21 PM       CORP\abester
     2/15/2014 2:51:35 PM       CORP\zack
BGInfo

This is another Sysinternals tool. Unlike these others, it does not monitor your system. It merely sets the desktop background for your system on startup based on the values of various variables. If you have many systems, it is convenient when switching systems to see right on the desktop the hosts’s IP address, name, and other critical system information. BGInfo can provide that for you.

Responding to a Suspicious Process

Students first learning how to defend a network tend to have a consistent response towards suspicious processes- nuke’em from orbit as quickly as possible.

This is often bad- even if the process actually is malicious. Killing a process deprives you of the chance to find out more about what is going on. How did the process get started? What processes are related? Has it opened a network connection? To where? Is that address connected to other processes?

Take some time and look carefully at the process using these tools, and find out all you can about it, if it is malicious, and how they may have gotten in. Then you can decide what to do, and if appropriate, then nuke it from orbit.

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: