03- Linux: Logging & SSH

As an added bonus! More Linux Distributions!

In week 1, we set up networking in CentOS. Now let us do the same for a few other common distributions.

The udev subsytem
Some linux distributions, including Open SuSE use the udev subsystem to allow for hardware components to be added and removed from the system. In Open SuSE, this system records the data for each network adapter used, and assigns each a separate name. This can be a problem, because when copying an Open SuSE VM you want to change the MAC address of the network card. The udev system thinks that
you have used two different network cards and keeps the configuration information for both. We do now want this behavior.

The solution is to delete the information from the older network cards, and to update the values for the existing card. Open the file /etc/udev/rules.d/70-persistent-net.rules in a text editor (as root). You may see multiple lines for different PCI devices, with different MAC addresses and different names. Delete all of the entries save for the last. In the last entry, modify the name value to be “eth0”. Reboot the box.

At this point, the system should show only one network card.

Open SuSE
We can set up a static IP address in Open SuSE in the same fashion as we did for CentOS; i.e. through /etc/sysconfig/network. If we want to change the hostname, then we can modify the value in /etc/HOSTNAME. However, we can also make theses change through the YaST setup tool for OpenSuSE.

Run YaST (Start Iguana / Computer / YaST). Navigate Network Devices -> Network Settings. Modify the IP settings through the Overview tab, by editing the (one) device that appears. [If you have more than one, take a look at your udev as described above.]

The hostname/DNS tab is used to specify oddly enough the name of the host, as well as the locations of the name servers. Be sure to set the hostname, the domain name, and the DNS Server addresses.

The routing tab is used to specify the default gateway for the card; be sure it too is set. Once these values are all set, simply apply them. There is no
need to reboot the box.

Ubuntu Desktop

Note that Ubuntu does not set a password for the root user. Administrative tasks need to be done by a regular user through the use of sudo. What are the advantages and disadvantages of this security model?

In Ubuntu 9.10, the udev files must be edited in essentially the same fashion as for Open SuSE; the file that needs to be modified is /etc/udev/rules.d/70-persistent-net-rules

In Ubuntu 10.10, the udev files do not need to be modified; the system
works essentially as CentOS does.

To get networking set up in Ubunto 10.10, navigate: System / Administration / Network Tools – Devices -> Select Network Device eth0; select Configure
– Wired -> Auto eth0 -> Edit. Then from the IPv4 Setting tab, choose Method -> Manual, then Addresses -> Add. Set the Address, Netmask, and Gateway. Set DNS Server (IP Addresses). Set the Search Domains (This is the namespace for your network). Apply

Be sure to test your results.

Logging in Linux

CentOS

Read the man page for syslog.conf, and examine the file /etc/syslog.conf. What log entries go to /var/log/messages? What log entries do not go to /var/log/messages?

Examine the file /var/log/messages What is the format of the entries to /var/log/messages?

Modify your syslog configuration so that all log entries of level info or higher go to a file you specify. Does that change the set of log entries sent to /var/log/messages?

Read the man page for logger. Use logger to send yourself log messages of various types. Check to see that your modified syslog configuration works.

Use logger to write a message to the log that says that the ssh server needs to be restarted. What are the security implications of this ability?

In what follows, it is convenient (though not absolutely necessary) to have wireshark installed on the CentOS machine. To install it in the lab, in VMWare, from VM -> Settings, set the CDROM in the VM to point to the .iso image for CentOS. (Available on labshare2). Be sure that the CD is connected!

As root, from the directory /media/CentOS_5.5_Final/CentOS/ run:

# rpm -ivh libsmi-0.4.5-2.el5.i386.rpm wireshark-1.0.8-1.el5_3.1.i386.rpm wireshark-gnome-1.0.8-1.el5_3.1.i386.rpm

Run wireshark (Applications -> Internet -> Wireshark Network Analyzer), and start a packet capture

Modify the file /etc/syslog.conf to send log files to another host. Should you specify the host by name or by IP? What are the security implications of each?
Don’t worry if the host exists yet or not- we will handle that later. Restart the syslog system (# /etc/init.d/syslog restart), and examine the packet capture.

• What protocol was used to send the syslog data?
• What port(s) were used?
• What are the security implications?
• What information about the log message was present in the packet? Was it readable?
• What are the security implications?

Set up a second CentOS system to recieve your log messages (We will call this host #2, while the original is host #1.) Restart the syslog system on host #1. Did host #2 receive any log entries?

Repeat after me- “Firewall, Firewall, Firewall.” Open the firewall on host #2. [What port and protocol? You did answer the questions above, didnt’t you?]

Restart the syslog system on host #1. Did host #2 receive any log entries? Read the startup script /etc/init.d/syslog. Notice that it looks for the file /etc/sysconfig/syslog (and only if it does not find that file does it set any default options.). Examine the file /etc/sysconfig/syslog on host #2. Modify that file in the appropriate fashion. Restart syslog on host #2.

From host #1, use logger and send a syslog message to the log system on host #1. Is it received by host #2? [Hint: Yes]. Check the logs and see that it arrived.

As an added bonus- what are mark messages?

As a second added bonus- go to another linux machine. Run the following

$ nc -w0 -u 192.168.1.202 514 <<< “Remote Log”

where the IP address is replaced by the IP address of host #2, your remote log server. What entries (if any) appear in your log?

nc is the netcat command- it is well worth learning what we can do
with this command. If you want to specify the facility and/or priority via netcat, take a look at
http://linux.byexamples.com/archives/412/syslog-sending-log-from-remote-servers-to-syslog-daemon

What are the benefits of sending all of your logs to one location?

OpenSuSE

Read the man pages for rsyslog.conf and rsyslogd. Notice that (a) This is a different system than the syslog system in CentOS,(b) This is more complex and (c) That documentation mentioned in the man page sure would be handy.

Copy the file rsyslog-5.4.0-2.7.i586.rpm from the lab share to a convenient directory and install:

# rpm -ivh rsyslog-5.4.0-2.7.i586.rpm

The documentation can be read at /usr/share/doc/packages/rsyslog/doc/manual.html. Briefly skim through the documentation.

Read the file /etc/rsyslog.conf. Where are (most) of the log entries saved? Modify the file so that all entries are saved in /var/log/allmessages. [Hint- though the logging system is called rsyslog, the init script (in /etc/init.d) is called “syslog”. Go figure.]. Verify that your changes worked.

To use rsyslog to send remote logs (UDP), add the entry

facility.priority @ip.address

to /etc/rsyslog.d/remote.conf Verify that this works with your existing (CentOS) log server.

To install wireshark on OpenSuSE, set up VMWare so that the CD for the VM points to the install CD for Open SuSE.

One problem- the image for the install CD (actually a DVD) is larger than 4 GB- which means it can’t fit on a FAT 32 partition. Why is that a problem? Well my portable drives are all FAT 32. I compressed the image into two files using 7-zip; you will need to uncompress them to get the original .iso

Mount the cdrom; OpenSuSE does not do this automatically. There is an icon that appears in the bottom right tray that can be used; you can also use the My Computer icon in the Desktop Folder- just right click on the CDROM and select mount.

Navigate to the directory /media/CDROM/suse/i586, and install all of the required packages at once:

# rpm -ivh wireshark-1.2.1-3.8.i586.rpm libGeoIP1-1.4.6-4.2.i586.rpm libcares2-1.6.0-2.2.i586.rpm portaudio-19-262.4.i586.rpm libjack0-0.116.2-5.1.i586.rpm

To run wireshark; navigate Start Iguana -> Applications -> Internet
-> Network Analyzer.

rsyslog can also send logs via TCP; to do so add the entry

facility.priority @@ip.address:port

to /etc/rsyslog.conf

To enable rsyslog to receive remote logs (UDP), uncomment the following in /etc/rsyslog.d/remote.conf:

# UDP Syslog Server:
$ModLoad imudp.so # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514

Open the firewall. To do so, navigate: Start Iguana -> Computer -> YaST -> Security and Users -> Firewall. Select Allowed Services -> Advanced -> Port
and then Finish.

Restart rsyslog (etc/init.d/rsyslog restart)

To enable rsyslog to receive remote logs (TCP), uncomment the following in /etc/rsyslog.d/remote.conf:

$ModLoad imtcp.so # load module
– Add the line to that same file, (assuming you want to use port 514)
$InputTCPServerRun 514

Again, open the firewall, and restart rsyslog

For further details, see the documentation for the imtcp module (/usr/share/doc/packages/rsyslog/doc/imtcp.html).

What are the advantages of using TCP instead of UDP?

What does a packet capture of TCP syslog traffic show?

What happens if you try

$ nc 192.168.1.221 514 <<< “message”

where the ip address (and port) of the log server is 192.168.1.221 (514).

It is possible to encrypt your log messages (over TCP) see /usr/share/doc/packages/rsyslog/doc/rsyslog_tls.html

Ubuntu
Ubuntu 10.10 also uses rsyslog, so nearly everything above still applies; however Ubuntu uses version 4.2, while Open SuSE uses version 5.4, so their may be some differences.

What are the differences in the structure of the files to configure rsyslog between Ubuntu and Open SuSE?

Ubuntu uses a slightly different method to hande scripts; to view the status of rsyslog for example, you can use (in OpenSuSE)

# /etc/init.d/syslog status

while the corresponding method in Ubuntu is

$ sudo service rsyslog status

Modify the syslog configuration in Ubuntu to receive logs via TCP, port 514, and restart the service. Verify that your approach works.

Wait a moment- did you open the firewall? To see the default firewall policy in Ubuntu, run

$ sudo iptables -L

What are the security implications of what you see?

Log Rotation

The logs generated cannot be kept indefinitely; as they continue to expand in size they will eventually consume all system resources. The logrotate tool is used on linux systems to compress, archive, rotate and delete log files.

Read the man page for logrotate.

Example- CentOS

• Read the file logrotate.conf
• How often are the log files rotated?
• How long are log files kept as archives?
• Are the log files compressed?
• Are their any special features for the log rotation of the DNS server logs?
• What about the web server logs?

Example- Ubuntu
Are there significant differences between the logrotate.conf for Ubuntu and CentOS?

Example- Open SuSE
Are there significant differences between the logrotate.conf for Open SuSE and CentOS and Ubuntu?

Splunk

Installation: On CentOS or OpenSuSE run (as root)

# rpm -ivh splunk-4.1.7-95063.i386.rpm

while on Ubuntu, run

$ sudo dpkg -i /home/user/Desktop/splunk-4.1.7-95063-linux-2.6-intel.deb

Splunk has a free license that allows it to be used provided certain conditions are met; it also comes with a free trial license. You must accept the license agreement the first time you run the tool. To change the license (after the system is running) from the trial license to the free license, first start splunk, and visit the splunk web page. Navigate Manager -> License (near the bottom third of the page), and select Change License. The license key to use splunk as a free tool is available in the file /opt/splunk/etc/splunk-free.license. Copy the contents into the License key box (overwriting the old license), and then restart splunk.

To start splunk for the first time, run (as root, or via sudo)

# /opt/splunk/bin/splunk start

To use the tool, point a web browser to the host, on port 8000. If you want to use a browser from other than the host, you need to open the firewall.

Sorry- CentOS did not have a web browswer installed in the image presented to class (my bad!). To solve this, in VMWare, have the CD point to the installation CD .iso (available on labshare2). From the directory /media/CentOS_5.5_Final/CentOS/ run (as root)

# rpm -ivh ./firefox-3.0.18-1.el5.centos.i386.rpm

You also need flash installed; the rpm flash-plugin-10.2.152.27-release.i386.rpm is available on labshare2. Install it using the same rpm command as above.

The first time the splunk is run, the account and password are provided and need to be changed.

Proceed through the tutorial; be sure to add /var/log/messages as a data source from the local server.

Set up splunk to listen to a TCP port (e.g. 514) for syslog messages. Extra credit if you remembered to open the firewall before reading this!

Set up another linux machine to use rsyslog to send data to the splunk machine via TCP port 514.

Use the Search to look through both the local log messages as well as the remotely received log messages.

Create a sample search for data from the DNS server (named). [If you do this exercise on your DNS sevrer, you will have many log entries to analyze.]

You can also set up a schedule for a search- but not for the free version of splunk.

Create a report based on your search for named.

Update the credentials for the administrator account for splunk.

If you want splunk to automatically start on boot, run (as root)

# /opt/splunk/bin/splunk enable boot-start

The *nix App. Enable the *nix App by navigating Launcher -> Your Apps -> *nix -> Enable. This will require a splunk restart.

To configure the App, navigate Launcher -> Your Apps -> *NIX, then continue to app setup page.

Examine the different options available. How many of those would be useful in our classroom environment? Enable those. Examine the data that results.

Splunk can also directly forward the data it collects to another Splunk server, rather than e.g. relying on syslog. On the machine that will receive the data, you must set up a receiver. To do so, log into Splunk Web as admin, and navigate Manager -> Forwarding and receiving (bottom third of the page). Click Add new in the Receive data section. Specify which (TCP) port you want the receiver to listen on, say port 7999. Click Save. We need to restart splunk for this change to be applied; to do so, navigate: Splunk Web (admin) -> Manager -> Server Controls -> Restart Splunk. Don’t forget to open the firewall!

On the sender, we need to configure the forwarder. Log into Splunk Web as admin by navigating Manager -> Forwarding and receiving (bottom third of the page). Click Add new in the Forward data section. Enter the hostname or IP address, together with the port of the receiver. Save; then restart splunk

Test out the result, and notice that you can now see the logs from both machines from one location.

There is much more that can be done with splunk, but we are out of time.

SSH

Starting the SSH Server.

On CentOS, the SSH server is already running and already allowed through the firewall.

On Open SuSE, the server is installed but not enabled. To enable the server navigate
Start Iguana -> YaST -> System -> System Services (Runlevel). Select sshd, and select enable. To open the firewall, navigate Start Iguana -> YaST -> Security and Users -> Firewall. From allowed Services; select Secure Shell Server, then Add; or from Advanced, select TCP Port 22.

On Ubuntu, the server is not present. Grab the package openssh-server_5.5p1-4ubuntu4_i386.deb from labshare2. Install it by running

$ sudo dpkg -i openssh-server_5.5p1-4ubuntu4_i386.deb

For the firewall, remember what we said earlier about the Ubuntu firewall.

SSH PKI
SSH can use public keys to authenticate users; this is often more convenient than passwords. To set up the PKI, modify /etc/ssh/sshd_config to uncomment out the following lines:

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

Restart sshd (/etc/init.d/sshd restart)

On the SSH server, for each user,

• Ensure that ~/.ssh is chmod 700 (rwx — —)
• Create file ~./ssh/authorized_keys (touch ~/.ssh/authorized_keys)
• Ensure that it has the right permissions (chmod 644 ~/.ssh/authorized_keys)

On the clients, for each user, you must first generate the keys. Run (on client, as user) $ ssh-keygen. The public key is ~/.ssh/id_rsa.pub, while the private key is ~/.ssh/id_rsa.

Copy public key from client to the file ~/.ssh/authorized_keys (or to /etc/ssh/authorized_keys) on the server.

ssh user@server and you are done.

Things you can do with ssh….

1. ssh user@remote_host command
2. scp user@remote_host:/path/to/file /local/path/to/file
3. sftp user@remote_host
4. ssh -Y user@remote_host. What does this one do? Check the man page!

Useful References

Guide to Computer Security Log Management, Recommendations of the National Institute of Sandards and Technology, by Karen Kent and Murugiah Souppay. Online at http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf