2015-06 Apache 2.2.15; ModSecurity 2.7.7 on CentOS 6.2


This week we will configure Apache 2.2.15 on the CentOS 6.2 (x64) image provided in class. Set up your system with static networking and a host name; in my example I will use

  • Hostname: excalibur.cosc.tu
  • IP Address:

You will also need a functioning DNS server with these entries. That server can be either a BIND system or a Windows system as you see fit, but it needs to be set up correctly for the server and the clients.

Apache is installed on the virtual machines provided in class, but it is not started by default. To verify its status, you can run either

[root@excalibur ~]# service httpd status
httpd is stopped
[root@excalibur ~]# /etc/init.d/httpd status
httpd is stopped

It can be started, stopped, or restarted by using the appropriate verb. The first time it is started, you will receive a warning:

[root@excalibur ~]# service httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified 
domain name, using excalibur.cosc.tu for ServerName
                                                           [  OK  ]

This is caused by the fact that the service has not yet been properly configured. We will deal with this as we proceed.

To ensure the web server always starts, navigate System → Administration → Services and enable httpd. This can be done via the command line

[root@excalibur ~]# chkconfig httpd on

In either case, be sure to open the firewall: navigate System → Administration → Firewall.


To find the version of apache, run

[root@excalibur ~]# httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Dec  8 2011 18:10:49

This document describes the 2.2.x series for Apache, from CentOS 6.2. Other versions of Apache include the 1.x series, the 2.0 series, and the 2.4 series. which have slightly different internal structures. Version 1.3 reached end-of-life in 2010, while version 2.0 has been in maintenance mode since 2010. Version 2.2 is still under development. Version 2.4 was released early in 2012, and will also not be covered.

Apache Structure

Apache is structured around a series of modules, which can either be compiled into the program or added dynamically. To see the compiled modules, run

[root@excalibur ~]# httpd -l
Compiled in modules:

To see the dynamically loaded modules, we will need to look at the configuration file.

Directories and files

The configuration scheme for an Apache setup is somewhat complex, and varies depending on the distribution and version. These are the CentOS defaults; other systems use variations on this theme.

        httpd.conf      Main configuration file
    conf.d/             Subdirectories for specific tasks
    logs ->              link to /var/log/httpd/
    modules ->           link to /usr/lib64/httpd/modules/
    run ->               link to var/run/httpd/
    access_log          Requests served via http
    error_log           Errors & problems.
    cgi-bin/            directory for CGI script
    error/              HTTP error messages in multiple languages
    html/               Root directory for web
    icons/              Collection of public domain icons

Because of the size and complexity of this system, and because we have only a limited time to devote to this topic, we will only cover some of the functionality available in the system. You are encouraged to explore.


Apache comes with its own documentation, but it is not installed by default. To install it, we use yum, and to account for the fact that we are in the clasroom laboratory, we disable all repos other than the install DVD.

[root@excalibur ~]# yum --disablerepo=\* --enablerepo=c6-media install 

Be sure that your DVD is connected and pointing at a valid .iso before you try the command.

When the installation is complete, go ahead and restart Apache

[root@excalibur ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd: httpd: Could not reliably determine the server's fully qualified 
domain name, using excalibur.cosc.tu for ServerName
                                                           [  OK  ]

Then, since the example web server is located at terminus.cosc.tu, the Apache manual is accessible at http://excalibur.cosc.tu/manual

Screenshot-Apache HTTP Server Version 2.2 Documentation - Apache HTTP Server - Mozilla Firefox

In the following notes, if a URL to a site like http://httpd.apache.org/docs/2.2/mod/core.html#serverroot is given, note that you now have access to this locally at http://excalibur.cosc.tu/manual/mod/core.html#serverroot.

Tour of the configuration files


Line 57 sets the root directory for the server- this is the directory root for Apache configuration files.

ServerRoot "/etc/httpd"

This is (very) different than Document Root, which is where the documents used in a web site would be stored. Read http://httpd.apache.org/docs/2.2/mod/core.html#serverroot for more detail.

Line 136 sets the IP addressed and port(s) on which Apache is listening.

Listen 80

The default is to listen to all IP adresses of the host, on port 80. Later we will learn how to set your Linux system up to use multiple IP addresses and multiple interfaces.

Read http://httpd.apache.org/docs/2.2/mod/mpm_common.html#listen for details on the directive.

Lines 150++ load various dynamic modules.

LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so

How many dynamic modules does Apache load? Does this explain why they were not listed earlier in our discussion of compiled modules?

To find out more about these modules, read http://httpd.apache.org/docs/2.2/mod.

Line 221 specifies additional configuration files to read

Include conf.d/*.conf

This ensures that the configuration files in this subdirectory are also loaded by Apache. See http://httpd.apache.org/docs/2.2/mod/core.html#include for the syntax. Remember that we already saw that Server Root was set to /etc/httpd, so this directive is relative to that directive, and hence the files in question are /etc/httpd/conf.d/*.conf

Lines 242-243 set the user and group for the server

User apache
Group apache

Note that we are running the server under a user account dedicated for this purpose. See http://httpd.apache.org/docs/2.2/mod/mpm_common.html#user for directive documentation.

Line 276 sets the name of the server; it should be configured to match your hostname. In this example, the FQDN of the server is terminus.cosc.tu, so we set

ServerName excalibur.cosc.tu
Making changes to Apache configuration files

Once changes are made to any of the configuration files, we need to tell Apache to reread the configuration files. This can be done with the service command and the restart verb

[2] 6067
[root@excalibur ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

This does have a downside though- this would disrupt any current connections to the server; thus the verb reload is preferred, using

[root@excalibur ~]# service httpd reload
Reloading httpd: 

This will reload the configuration files, without stopping the server.

Now I know that no one in my class could ever make a syntax error in a configuration file, but it is possible that you may see it happen to someone else. If there is a syntax error in the configuration files, the reload will fail and the server will continue with the previous configuration file.

You can validate your configuration file before reloading or restarting. Suppose that we deliberately introduce a simple error into our configuration file, replacing the ServerName directive with

ServerName excalibur.co

Here we have "accidentally" split the hostname on to two lines. Clearly the second line is a syntax error. Attempting to reload the configuration throws an error:

[root@excalibur ~]# service httpd reload
Reloading httpd: not reloading due to configuration syntax error

but no information is provided as to the source of the problem. Here we run

[root@excalibur ~]# apachectl configtest
Syntax error on line 277 of /etc/httpd/conf/httpd.conf:
Invalid command 'sc.tu', perhaps misspelled or defined by a module not 
included in the server configuration

to locate the error. You should get in the habit of running configtest before you even consider reloading the Apache configs.

/etc/httpd/conf/httpd.conf, (continued)

Returning to our tour of /etc/httpd/conf/httpd.conf, line 292 sets the root directory for the web site (not the server)

DocumentRoot "/var/www/html"

In particular, files served will be located here. Keep in mind though, that this directory can contain links to other point in the file system, so this is not absolute protection for your file system. Note also that we can use aliases to store portions of the web site in completely different locations in the file system. See http://httpd.apache.org/docs/2.2/mod/core.html#documentroot for more information about the directive.

To determine if Apache will serve a file, it will check its policies, and these policies are set on a directory-by-directory basis. What makes this complex is the fact that these directory directives can overlap.

Lines 302-305 contain a directory directive that is applied to the root of the file system "/". What is that policy?

Start by taking a look at the documentation for the Directory directive at http://httpd.apache.org/docs/2.2/mod/core.html#directory to interpret what you see. Note that the syntax for the options for that directive are not described at there, instead, look at http://httpd.apache.org/docs/2.2/mod/core.html#options to see the meaning of the options.

Read the documentation at http://httpd.apache.org/docs/2.2/sections.html. What happens if two different <Directory> directives apply to a directory? Which takes precedence?

Lines 317-346 form another <Directory> directive. What is the directory to which they refer? Create a valid html file in that directory. Verify that it is served. Create a symlink to /etc/passwd and put it in the directory. What happens when that file is requested? Create a hard link to /etc/passwd and put it in the directory. What happens when that file is requested? Repeat the previous with /etc/shadow. Explain your observations.

What are the access control options for Document Root (lines 343-344)? See http://httpd.apache.org/docs/2.2/mod/mod_authz_host.html for details of the syntax.

If the Allow from all line is removed, what is the default access policy for the server? Verify this.

Lines 360-375 by default turn off the use of directories for individual users. See http://httpd.apache.org/docs/2.2/mod/mod_userdir.html and see also http://httpd.apache.org/docs/2.2/howto/public_html.html

Enable user directories, but only for one user. To do so for the user valen, modify these lines to include:

UserDir disable
UserDir enabled valen

Why is this approach preferred over enabling user directories for all users?

Set the directory that is to be shared to be public_html by uncommenting the line

UserDir public_html

Be sure to test then reload your configuration file.

If a request is made of the server for the file http://server/~user/dir/page.html where in the file system should that file be located? Create this directory and this file. Test to see if the server serves your file. It didn’t? Huh. What are the permissions of the directory /home/user? What permissions are necessary so that another user (say the user apache from group apache) can traverse this directory? Remember- execute permission is needed on a directory to cd into that directory while read permission is needed to read the list of files in that directory. How about we make that change! What are the permissions for the directory and the file for your web page? Test again!

The joy of SELinux

It still doesn’t work even though the permissions are right? Did you see an error like

[Sat Mar 01 18:56:14 2014] [error] [client] (13)Permission denied: access to /~zathras/index.html denied

in /var/log/httpd/error_log? Welcome to SELinux. There are people who think that this is a great tool for securing a Linux system. Other people. It turns out that SELinux is preventing Apache from reading the file you have set up because it is in a different security context. This would not be a huge hurdle, provided SELinux would explain what was being blocked and why. There are tools to try and figure out what SELinux is doing and why certain actions are being blocked, and that program is installed by default. Oh silly me, no it isn’t- that would take the fun out it.

Let’s solve our SELinux issue with the file. Run the command

[root@terminus selinux]# echo 0 >/selinux/enforce 

This will persist until the next reboot. To make the change permanent, open /etc/selinux/config and set


Now you may feel the urge to set that all the way to disabled. If you do though, you will have difficulty enabling SELinux should you later change your mind.

The problem here is not so much that SELinux is a bad idea- it really isn’t. The problem is that it reports so little information back to the user that debugging SELinux caused problems is really quite difficult- especially for students who are just learning how to get the systems set up correctly in the first place.

/etc/httpd/conf/httpd.conf, (continued) (continued)

Returning to our original question. Suppose that you let individual users set up their own web pages on a server in this fashion. How can this information be used by an attacker to determine the users of a computer?

Lines 381-392 are the <Directory> directive for the use of directories by individual users. Note that, they are commented out in the default configuration, meaning they fall back to the default Directory policy (lines 302-305).

To understand the syntax for the <Limit> directive, see http://httpd.apache.org/docs/2.2/mod/core.html#limit, while to understand the syntax for the <LimitExcept> directive, see http://httpd.apache.org/docs/2.2/mod/core.html#limitexcept

What HTTP methods are allowed for the public user directories? What HTTP methods are not allowed for the public user directories?

Line 402 determines what file will be served if the user asks for a directory.

Line 409 sets the default name (.htaccess) for directory-by-directory additional configuration files. For details on how this file is used, see http://httpd.apache.org/docs/2.2/mod/core.html#accessfilename

Will .htaccess files be read for directories within DocumentRoot (/var/www/html)? Hint: What happens on line 338? What about the default? (Hint: Line 304.)

Lines 415-419 prevent any files that start with the characters .ht from being served. See http://httpd.apache.org/docs/2.2/mod/core.html#files for documentation on the Files directive. If you want to, know more about regular expressions, go online- just remember that (1) Apache uses PCRE and (2) "Some people, when confronted with a problem, think ‘I know, I’ll use regular expressions.’ Now they have two problems."

Line 484 sets the location of the error log.

ErrorLog logs/error_log

Note that this is relative to the root directory for Apache configuration (/etc/httpd) and should now explain why that directory has a symlink from logs to /var/log/httpd.

See http://httpd.apache.org/docs/2.2/mod/core.html#errorlog for details; notice also that the log system can be set to use syslog.

Line 491 tells the server to log only at the level of warn or higher

LogLevel warn

See http://httpd.apache.org/docs/2.2/mod/core.html#loglevel for the details.

Lines 497-526 set up the access logs for the system; these will be discussed in some detail below.

Line 536 tells Apache to include information about the current server version when returning an error message. See http://httpd.apache.org/docs/2.2/mod/core.html#serversignature

Lines 551-558 set up an alias for the icons directory. View http://localhost/icons/.
(The trailing / is necessary.) Note the list of icons- even though there is no icons directory in document root- the alias command is mapping a different file location that that portion of the web site.

If an alias points outside of documentroot, permissions need to manually set on the directory. Aliases are quite powerful- see http://httpd.apache.org/docs/2.2/mod/mod_alias.html

Line 576-587 sets up an alias for CG scripts; this alias is similar to the alias described above. To practice with a CGI script, create the file test.pl with the following content:

print "Content-type: text/html\n\n";
print "This is a cgi script\n";

Place the file in the CGI directory, and make it world-executable.

The first line tells the system how to execute the file. The second sets up the headers (and it needs BOTH newlines). The rest- well you can have it do whatever you want at that point. Note that this is code that is being executed by the server, on the server. Then http://localhost/cgi-bin/test.pl will execute the script and return the result.

It is possible to set up per-user CGI script directories if individual users have their own web directories. See http://httpd.apache.org/docs/2.2/howto/cgi.html for more on CGI scripting. See http://httpd.apache.org/docs/2.2/mod/mod_alias.html#scriptalias for more on the syntax for ScriptAlias

There is more to this configuration file, but it is of lesser importance than the above, and we do nto have time for the discussions.


This is used to configure SSL on the server. We will discuss this in detail later when we cover SSL.

Note that this file is automatically included in the Apache configuration at line 221 of /etc/httpd/httpd.conf, where the directive

Include conf.d/*.conf

exists. Note that the structure of this include statement only includes files that end in “.conf”. Because many text editors automatically create backups of files, if you are not careful with this directive you can end up including both the original and the backup file, duplicating some statements and causing no end of hassle.


Apache should have a separate user:group to run the server. (It does- apache:apache by default in CentOS). It does need to start as root so that it can bind to port 80. The choice is made in httpd.conf, lines 242-243. What would happen if this file is writable by other than root?

Permissions on the binaries need to be set correctly. What are the default permissions on the executable for the apache web server? [Indeed- what is the name of the executable?]

Permissions on the logs should be set so that only root can read them. What are the permissions on the directory /var/log/httpd? Why? These will store user supplied data- if they can get a process to read the data, it might be “trusted” as it comes from the local file system.

There is a separate file clobbering issuse described in http://httpd.apache.org/docs/2.2/misc/security_tips.html#serverroot


NOTE- This is for non-SSL logging. Logs for SSL connections will be described later.

There are two kinds of logging- Request logging and Error logging.

Request logging

One must first create a log format using a LogFormat directive. See either http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#logformat or http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats for more details.

Four common log formats types are defined:

  • combined
  • common
  • referer
  • agent

By default, Apache uses the combined format. [See line 526 of /etc/httpd/conf/httpd.conf]

Logging itself can be accomplished with one of two commands:


(http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#customlog) or


(http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#transferlog). This second choice is rarely used.

One nice feature of these logs is that they are easily scriptable. Suppose for example that you want to print out all of the IP addresses that have been used to access your site. This is a simple job for Python and some regular expressions. For example, following the approach of Jochen Voss you can build code like


import re

parts = [
    r'(?P<host>\S+)',                   # host %h
    r'\S+',                             # indent %l (unused)
    r'(?P<user>\S+)',                   # user %u
    r'\[(?P<time>.+)\]',                # time %t
    r'"(?P<request>.+)"',               # request "%r"
    r'(?P<status>[0-9]+)',              # status %>s
    r'(?P<size>\S+)',                   # size %b (careful, can be '-')
    r'"(?P<referer>.*)"',               # referer "%{Referer}i"
    r'"(?P<agent>.*)"',                 # user agent "%{User-agent}i"
pattern = re.compile(r'\s+'.join(parts)+r'\s*\Z')

request_ip = []

logfile = open('/var/log/httpd/access_log','r')
for line in logfile:
   m = pattern.match(line)
   res = m.groupdict()
   if not res['host'] in request_ip:

for ip in request_ip:
   print ip

Then running this code will simply print out a list of all of the IP addresses that connected to the web server. You can just as easily print out the user agent or the request or any of the other recorded data.

This approach proceeds via Python regular expressions, making use of named symbolic groups to simplify the result.

Error Logging

Error logs are specified with the ErrorLog directive. [See line 484 of /etc/httpd/conf/httpd.conf]. Read http://httpd.apache.org/docs/2.2/mod/core.html#errorlog for details.

Error logs are written according to a priority specified by the LogLevel directive. Error levels are:

  • emerg Emergencies – system is unusable.
  • alert Action must be taken immediately.
  • crit Critical Conditions.
  • error Error conditions.
  • warn Warning conditions.
  • notice Normal but significant condition.
  • info Informational.
  • debug Debug level messages.

Error logs can be sent to

  • A file- e.g.
    ErrorLog /var/log/httpd/error_log
  • An application that will handle the data- e.g.
    ErrorLog "|/usr/local/bin/httpd_errors"

    Note that the programs so started are started by the web server, and may have root privileges.

  • To syslog, by default to local7. You can specify the facility though (the priority is set by Apache); for example to use the user facility user rather than the default local7, use the line
    ErrorLog syslog:user

Try some of these different options.

Virtual Hosts

Virtual hosts are a way that a single web server can run multiple web sites.

  • You can have one host with multiple DNS names serve different websites depending on the hostname in the request
  • You can have one host with multiple IP addresses serve different websites depending on the IP address in the request.
  • You can have one host serve different websites depending on the port in the request.

These different web sites can have different

  • Server admin
  • Document root
  • Server Name

Apache uses virtual hosts as a key mechanism for SSL protected web sites. See http://httpd.apache.org/docs/2.2/mod/core.html#virtualhost, http://httpd.apache.org/docs/2.2/vhosts/, and http://httpd.apache.org/docs/2.2/vhosts/examples.html for additional documentation.

What happens is the same directory is accessible to two or more virtual hosts? They both can access it! This can cause security issues, as we will shortly see.


The default installation of Apache does not install the components needed to Apache to provide SSL services. We can add this feature to our system simply enough with a yum command; we will again only use the install media for the installation to avoid the problems that our private network might generate.

[root@excalibur Desktop]# yum --disablerepo=\* --enablerepo=c6-media install 
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package mod_ssl.x86_64 1:2.2.15-15.el6.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

 Package       Arch         Version                        Repository      Size
 mod_ssl       x86_64       1:2.2.15-15.el6.centos         c6-media        87 k

Transaction Summary
Install       1 Package(s)

Total download size: 87 k
Installed size: 183 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : 1:mod_ssl-2.2.15-15.el6.centos.x86_64                        1/1 

  mod_ssl.x86_64 1:2.2.15-15.el6.centos                                         


The configuration for the SSL secured component of a website is contained in the file /etc/httpd/conf.d/ssl.conf. Because we are including all of the files from this directory (/etc/httpd/conf/httpd.conf line 221) this configuration file will now be also loaded automatically into our Apache system once we reload

[root@excalibur ~]# service httpd reload
Reloading httpd: 

Line 12 loads the module needed for SSL

By default, SSL listens on port 443; this is enabled on line 18.

Lines 20-68 set options that are global to all SSL protected web sites on that server, including properties of the random number generators and the cache.

To configure a portion of the website to use SSL, Apache uses a <VirtualHost> directive, in lines 74-221. Note first that the directive

<VirtualHost _default_:443>

matches any IP address not listed in another <VirtualHost> directive, on port 443. [c.f. http://httpd.apache.org/docs/2.2/mod/core.html#virtualhost ].

Virtual hosts can specify their own document root separately from the main host; thus we can specify a different document root for the SSL portion of the web site. In the default configuration file, line 77, the DocumentRoot directive is commented out, and hence anything in the DocumentRoot for the main web site can also be served via SSL.

Recall, if a directory is accessible via two <VirtualHost> directives, then both directories will serve that document. Verify this behavior. Earlier you created a file for your web server to serve. Now access it via SSL (i.e. https://yoursite/yourfile). Ignore any SSL warnings for the moment, and verify that you get the same page.

Line 78 sets the server name for this virtual host, or it would it it were not commented out. Remember that one Apache server can use virtual hosts with different host names. See http://httpd.apache.org/docs/2.2/mod/core.html#servername for more details.

Line 82 sets the location of the error log, line 83 sets the location of the transfer log and line 84 sets the log level for this virtual host.

Line 88 enables SSL for access to this virtual host.

Remember- the virtual host directive here was <VirtualHost _default_:443>, so anything that comes in via port 443 to any IP adress not bound to another virtual host will use SSL. However- this does not mean that any access to a directory inside this virtual host will automatically require SSL. If a file is in the document root for more than one virtual host, then either host can serve that file! We have seen this already!

Don’t confuse the fact that the virtual host uses SSL with the notion that the files necessarily require SSL- that depends on the totality of your Apache configuration.

See http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslengine for the syntax

The directive SSLProtocol on line 93 specifies the versions of SSL that will be accepted. See http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslprotocol for the syntax. Note that SSLv2 is known to be flawed; it is disabled by default in modern browsers (e.g. IE7+, FF2+)

The precise Cipher is set by the SSLCipherSuite directive in line 98.

The private and public key locations are set in lines 105 & 112.

Read the remainder of the configuration file; note the additional log statement in lines 218-219.

To illustrate, let us set up an SSL protected directory on our web server that is only accessible via SSL.

Create the directory /var/www/html/ssl. Be sure that permissions are set so that it can be read by user apache of group apache.

Create the file /var/www/html/ssl/index.html.

Verify that right now this file can be accessed via both ssl (https) and non-ssl (http).

Create the following Directory directives OUTSIDE THE (SSL) VIRTUAL HOST

<Directory "/var/www/html/ssl">
SSLOptions +StrictRequire

Why do these need to be outside the SSL virtual host? What is the best file to contain these? Why?

Restart Apache. Verify that your file can be accessed via SSL but not without SSL.

You should still be getting certificate warnings. If you have accepted the warnings from an earlier visit to your website, the warnings may no longer appear; simply close your browser and start again.

Certificate Authorities

Now we are going to create our own certificates and set up our own CA to sign them; doing so will let us import the certificate of the CA into browsers so that all of the certificates signed by that CA will no longer raise browser warnings.

We will be using this host to act as the equivalent of a trusted signer like VeriSign or Thawte. This host is not the web server. It should not be providing network services and should be maximally protected, as if an attacked compromises this machine, then all of your SSL certificates will be of no value.

In this example, the CA system will be called furies.cosc.tu. The common name (CN) of the CA needs to be different from the common name of any machine for which you generate a certificate; remember that our example web server is at excalibur.cosc.tu.

Generating the CA key

Start by generating the private key for our CA:

[root@furies ~]# openssl genrsa -des3 -out /etc/pki/CA/private/ca.key 4096
Generating RSA private key, 4096 bit long modulus
e is 65537 (0x10001)
Enter pass phrase for /etc/pki/CA/private/ca.key:
Verifying - Enter pass phrase for /etc/pki/CA/private/ca.key:

Since this is the key for the CA, it should have a good passphrase. This passphrase will be needed whenever we want to use this key for signing.

Examine your CA key

[root@furies ~]# cat /etc/pki/CA/private/ca.key 
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,846E37F973C86E1D

<-- deleted material -->

Because this is the master key for the CA, it needs to be maximally protected. Notice that the default permissions on the directory /etc/pki/CA are 700

[root@furies ~]# ls -l /etc/pki/CA
total 16
drwxr-xr-x. 2 root root 4096 Dec  7  2011 certs
drwxr-xr-x. 2 root root 4096 Dec  7  2011 crl
drwxr-xr-x. 2 root root 4096 Dec  7  2011 newcerts
drwx------. 2 root root 4096 Mar  2 07:28 private

so that it is not readable by non-root users. If an outsider gains read access to this directory and your passphrase, they will be able to sign certificates just as you can; this would be bad.

Generating the CA Certificate

Now we generate a certificate for our CA with our key. You will need the passphrase for your server key. You will need to enter some data for the CA certificate. Remember that the CN we will use for the CA is different than the name we will use for any web server certificates.

[root@furies ~]# openssl req -new -x509 -days 365 -key
/etc/pki/CA/private/ca.key -out /etc/pki/CA/certs/ca.crt

Enter pass phrase for /etc/pki/CA/private/ca.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:Maryland
Locality Name (eg, city) [Default City]:Towson
Organization Name (eg, company) [Default Company Ltd]:Towson University
Organizational Unit Name (eg, section) []:Security Laboratory
Common Name (eg, your name or your server's hostname) []:furies.cosc.tu
Email Address []:   

This completes the setup of our CA.

Creating the web server certificate

Now we return to the web server (excalibur.cosc.tu). The Apache install has already created a key and certificate for the web server as localhost, stored in /etc/pki/tls/. In general though, I do not trust keys installed as part of the installation and prefer to generate new ones.

We will create a new web server key by running

[root@excalibur ~]# openssl genrsa -out /etc/pki/tls/private/excalibur.key 4096
Generating RSA private key, 4096 bit long modulus
e is 65537 (0x10001)

Note that we have stored the key in /etc/pki/tls/private and named it after the hostname. This is not the same directory we used in the CA; not only are we on a different host, but we are in /etc/pki/tls/ rather than /etc/pki/CA; this makes sense as in the first case we were setting up a CA, while now we are setting up TLS/SSL connections.

Verify that you have a valid key on the web server

[root@excalibur ~]# cat /etc/pki/tls/private/excalibur.key 
<-- deleted material -->

If you want to see your public key, you can run

[root@excalibur ~]# openssl rsa -in /etc/pki/tls/private/excalibur.key -pubout
writing RSA key
-----END PUBLIC KEY-----

We want the CA to vouch for this web server, so we start by creating a certificate signing request (csr) on the web server. Again you will need to answer some questions, but now about your web server. One catch here is that the common name must match the name of your web site; the certificate is matched to the name of the server. If they do not agree, you will see a domain mismatch error. Remember- this name must be different than the CN you selected for the CA.

[root@excalibur ~]# openssl req -new -key /etc/pki/tls/private/excalibur.key 
-out /etc/pki/tls/misc/excalibur.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:Maryland
Locality Name (eg, city) [Default City]:Towson
Organization Name (eg, company) [Default Company Ltd]:Towson University
Organizational Unit Name (eg, section) []:Security Laboratory
Common Name (eg, your name or your server's hostname) []:excalibur.cosc.tu
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Next, we send the certificate signing request over to our CA for its signature. In the lab this can be handled by simply copying the file on one virtual machine and pasting it in the other; I simply used SSH

[root@excalibur ~]# scp /etc/pki/tls/misc/excalibur.csr 
root@furies.cosc.tu's password: 
excalibur.csr                                 100% 1740     1.7KB/s   00:00    

With the CSR on the CA, we sign the result (on the CA!)

[root@furies ~]# openssl x509 -req -days 365 -in /etc/pki/CA/excalibur.csr 
-CA /etc/pki/CA/certs/ca.crt -CAkey /etc/pki/CA/private/ca.key 
-set_serial 01 -out /etc/pki/CA/newcerts/excalibur.crt
Signature ok
subject=/C=US/ST=Maryland/L=Towson/O=Towson University
/OU=Security Laboratory/CN=excalibur.cosc.tu
Getting CA Private Key
Enter pass phrase for /etc/pki/CA/private/ca.key:

Be sure to increment the serial number for each certificate.

Now we copy the signed certificate back to the web server (excalibur.cosc.tu) and store it in the file /etc/pki/tls/certs/excalibur.crt.

[root@furies ~]# scp /etc/pki/CA/newcerts/excalibur.crt 
root@excalibur.cosc.tu's password: 
excalibur.crt                                 100% 1968     1.9KB/s   00:00  

Be sure to pay attention to file permissions. Copying files to/from a Windows host can leave the result executable and world writable. I wonder if that would be bad?

Configuring Apache to use the certificate

Now we will tell the web server to use these keys. In the <VirtualHost> directive of the /etc/httpd/conf.d/ssl.conf file modify the following lines

SSLCertificateFile /etc/pki/tls/certs/terminus.crt
SSLCertificateKeyFile /etc/pki/tls/private/terminus.key

(line 105 and 112).

Restart Apache. Check your logs (/var/log/messages and /var/log/httpd/error_log) to be sure that Apache restarted cleanly.

View the SSL enabled page in a browser; you should still see an error saying that the issuer certificate is unknown. From the browser, view the certificates that are being used. You should see the certificate for the server, as well as the certificate of the CA.
Screenshot-Certificate Viewer^%&%excalibur.cosc.tu&%

Now we will import the certificate for the CA into our browser. Be careful here- we are not copying anything from the web server; in fact the idea is that the user doesn’t trust the web server, but the user does trust the CA.

Place the certificate /etc/pki/CA/certs/ca.crt from the CA into a convenient location. [In an exercise, you may want to host this certificate on a public web site with an explanation of its significance.]

In Firefox, navigate Edit → Preferences → Advanced → Encryption → View Certificates. Select Import, then select the file ca.crt; trust this CA to identify web sites.
CentOS (DNS) Acheron-2014-03-02-11-06-43

When this is complete, you will see the entry for the CA in the list of trusted authorities. You should be able to visit any web site whose crt has been signed by that CA and view it without any SSL errors.

Notice that if you visit the web page without using the precise name in the certificate- by e.g. visiting rather than https://excalibur.cosc.tu/ssl then you will see that the certificate is only valid for the selected hostname.

Access Control

There are three important types of authentication

  • Basic Authentication
  • Digest Authentication
  • Form-based Authentication

The first two of these are part of the HTTP protocol (RFC 2167). The last is most common, and requires the web page(s) themselves to handle the authentication (via e.g. PHP). We will not cover this here.

Digest authentication, though an improvement on basic authentication, is not commonly supported by browsers; we will not cover this either.

Documentation on how to set up authentication can be found at http://httpd.apache.org/docs/2.2/howto/auth.html

Here is a demonstration. For this purpose, we will put all of our configuration information in the main file (/etc/httpd/conf/httpd.conf) rather than in per-directory .htaccess files.

Create the directory /var/www/auth. We will store our authentication data here (outside of document root for the web server). Create the Apache password file, starting with a single user (LewisPirenne) as follows:

[root@excalibur ~]# htpasswd -m -c /var/www/auth/passwd Ivanova
New password: 
Re-type new password: 
Adding password for user Ivanova

Note that the password for this user is manually entered.

Let’s view our password file:

[root@excalibur ~]# cat /var/www/auth/passwd 

The -m option above told Apache to use the MD5 hash, which is exactly what we see in the file. Other options include DES (crypt()), SHA, and plaintext. More options for htpasswd can be found in the documentation at http://httpd.apache.org/docs/2.2/programs/htpasswd.html

Let’s create a directory that will require password protection, and put a page inside.

[root@excalibur ~]# mkdir /var/www/html/secret
[root@excalibur ~]# vi /var/www/html/secret/index.html
[root@excalibur ~]# cat /var/www/html/secret/index.html 
This is a secret page

At this point, we can still directly view the page at http://excalibur.cosc.tu/secret/. You should be able to explain why!

Now we update our configuration file (/etc/httpd/conf/httpd.conf) by adding the following:

<Directory "/var/www/html/secret">
AuthType Basic
AuthName "Restricted Files"
AuthUserFile /var/www/auth/passwd
Require valid-user

Reload the Apache configuration. Check your logs to verify that the restart was clean.

Now we are required to authenticate with a valid user/password combination from our file before we can view this page. Verify this!

Full documentation (including how to set up groups) can be found at http://httpd.apache.org/docs/2.2/howto/auth.html.

It is also possible to use other than text files for authentication, including LDAP and databases. Consult the documentation for further details.



ModSecurity is a web application firewall; it allows us to filter and analyze requests made of our Apache server at the application level. In this it differs from network firewalls, which focus on the packet level.

We are going to install the latest version (2.7.7) from source.

Before compiling ModSecurity, we need to install a few dependencies; the core requirements for installation include the packages

  • gcc
  • make
  • libxml2
  • libxml2-devel
  • httpd-devel
  • pcre-devel
  • curl-devel

We can install all of these (and their dependencies) with a simply yum command; again we respect the closed nature of the lab and only use the install DVD as our source.

[root@excalibur ~]# yum --disablerepo=\* --enablerepo=c6-media install 
libxml2 libxml2-devel httpd-devel pcre-devel curl-devel

This command will install a total of 15 different packages.

Grab the package modsecurity-apache_2.7.7.tar.gz either from the labshare or online. Uncompress the package in the usual fashion in the usual source directory (/usr/local/src)

[root@excalibur src]# tar -xzvf ./modsecurity-apache_2.7.7.tar.gz 

and descend into the directory modsecurity-apache_2.7.7 that is created. From within that directory, run the configure script

[root@excalibur modsecurity-apache_2.7.7]# ./configure 

This should complete without error if all of the required packages have been installed. To compile and install the program, run

[root@excalibur modsecurity-apache_2.7.7]# make
[root@excalibur modsecurity-apache_2.7.7]# make install

With the installation complete, add the lines

LoadFile /usr/lib64/libxml2.so
LoadFile /usr/lib64/liblua-5.1.so
LoadModule security2_module modules/mod_security2.so

to the file /etc/httpd/conf/httpd.conf after all of the other LoadModule lines (roughly line 215 depending on how you have modified your file).

ModSecurity also requires the module unique_id_module; this is one of the modules not loaded by default. Uncomment out the line

LoadModule unique_id_module modules/mod_unique_id.so

We will also need a configuration file for ModSecurity. Fortunately it comes with a template that we can later customize; we move it to /etc/httpd/conf.d:

[root@excalibur ~]# cp /usr/local/src/modsecurity-apache_2.7.7
/modsecurity.conf-recommended /etc/httpd/conf.d/modsecurity.conf

Because this file is located within the conf.d subdirectory, it will be loaded as part of the Apache startup.

We also need to copy over the file unicode.mapping to its proper location

[root@excalibur ~]# cp /usr/local/src/modsecurity-apache_2.7.7/unicode.mapping 

Validate your configuration and start the web server:

[root@excalibur ~]# apachectl configtest
Syntax OK
[root@excalibur ~]# service httpd start
Starting httpd:                                            [  OK  ]

The first time you reload with modsecurity, or each time you restart with modsecurity you should see lines similar to these in your /var/log/httpd/error_log file

[Sun Mar 02 09:06:23 2014] [notice] ModSecurity for Apache/2.7.7 
(http://www.modsecurity.org/) configured.
[Sun Mar 02 09:06:23 2014] [notice] ModSecurity: APR compiled version="1.3.9"; 
loaded version="1.3.9"
[Sun Mar 02 09:06:23 2014] [notice] ModSecurity: PCRE compiled version="7.8 "; 
loaded version="7.8 2008-09-05"
[Sun Mar 02 09:06:23 2014] [notice] ModSecurity: LIBXML compiled version="2.7.6"
Configuring ModSecurity

To configure ModSecurity, return to the file /etc/httpd/conf.d/modsecurity.conf. The first directive in that file

SecRuleEngine DetectionOnly

on line 7 tells ModSecurity that it should process all of its rules, but it should not take any action to modify or disrupt the connection. Documentation for this and for the other directives can be found online at https://github.com/SpiderLabs/ModSecurity/wiki/.

Since this is not (yet) a production server, go ahead and change that setting to

SecRuleEngine On

Next, move to the end of the modsecurity.conf file and add the line

SecRule ARGS, "zzz" phase:1,log,deny,status:503,id:1

We are going to use this rule solely for testing. It tells ModSecurity to look at the arguments of the request for the string "zzz"; if it appears then log the request, and return a service temporarily unavailable (503) error. We are using the rule id #1; we are permitted to use any number in the range 1-99,999 for local rules.

With these changes made, restart Apache. Then from a browser on another system visit the web page of your server and include the text "zzz" as one of the parameters. Since GET parameters are specified in the URL, we can simply add these to the URL; if ModSecurity is correctly running, it will stop us with a 503 error.
Screenshot-503 Service Temporarily Unavailable - Mozilla Firefox

Your error log (/var/log/httpd/error_log) will then tell you what happened:

[Sun Mar 02 09:09:36 2014] [error] [client] ModSecurity: Access 
denied with code 503 (phase 1). Pattern match "zzz" at ARGS:a. [file 
"/etc/httpd/conf.d/modsecurity.conf"] [line "214"] [id "1"] [hostname 
"excalibur.cosc.tu"] [uri "/"] [unique_id "UxNl0H8AAAEAAD4pDZgAAAAB"]

Now that we know our installation is working, lets return to the modsecurity.conf file.

The configuration directives in the Request body handling and the Response body handling sections (lines 10-95, 96-122) can be left in their default state.

If your web server is used solely as a web server, then you can keep the Filesystem configuration variables (lines 133-139) in their default state; i.e. using /tmp as the temporary directory for ModSecurity. If multiple users are going to be present on the system though, then you may wish to consider an alternate structure. You could, for example, create the directory /var/www/modsecurity, set its permissions and owner appropriately (e.g. owner apache:apache, permissions 700) and specify that location here.

The File uploads handling configuration (lines 141-160) determines what should happen to uploaded files after ModSecurity finished processing them. To keep things simple, let’s agree to drop them and so set

SecUploadKeepFiles Off

The Debug log is, oddly enough used to store debugging messages from ModSecurity. The commented out location in the configuration file (/opt/modsecurity/var/log/debug.log) does not even exist. Start by creating a file- say /var/log/modsec_debug.log, and modify the directive:

SecDebugLog /var/log/modsec_debug.log

The SecDebugLogLevel sets the level amount of detail to be contained in that file.

  • Level 0: No logging
  • Level 1: Fatal errors
  • Level 2: Warnings
  • Level 3: Notices
  • Level 4: Informational
  • Level 5: Details
  • Level 9: Everything

Though useful, be careful with this setting. These are debugging messages, and can be used to debug rules that you create. On the other hand, they generate a lot of data. At level 5 with just the one rule, the debug log ends up with 24 entries for a single request. The recommendation is to use this log sparingly; certainly not higher than level 3 without cause.

ModSecurity began as a project to provide full HTTP transactional logs; this is done through the Audit logs (lines 171-192). They are able to log all parts of a web request, both inbound to the server and outbound from the server.

The default settings have ModSecurity only log those requests that generate a server error (other than 404), though the settings can be modified to record every request.

Each request is broken into parts, including

  • The request header
  • The request body
  • The response header
  • The response body

Some, all, or none of these parts can be recorded; this is set via SecAuditLogParts; see the documentation for more detail.

A typical audit log entry looks like

[02/Mar/2014:09:09:36 --0800] UxNl0H8AAAEAAD4pDZgAAAAB 47941 80
GET /?a=zzz HTTP/1.1
Host: excalibur.cosc.tu
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Firefox/17.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive

HTTP/1.1 503 Service Temporarily Unavailable
Content-Length: 406
Connection: close
Content-Type: text/html; charset=iso-8859-1

<title>503 Service Temporarily Unavailable</title>
<h1>Service Temporarily Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
<address>Apache/2.2.15 (CentOS) Server at excalibur.cosc.tu Port 80</address>

Message: Access denied with code 503 (phase 1). Pattern match "zzz" at ARGS:a. 
[file "/etc/httpd/conf.d/modsecurity.conf"] [line "214"] [id "1"]
Action: Intercepted (phase 1)
Stopwatch: 1393780176100857 509 (- - -)
Stopwatch2: 1393780176100857 509; combined=39, p1=37, p2=0, p3=0, p4=0, p5=2, 
sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.7 (http://www.modsecurity.org/).
Server: Apache/2.2.15 (CentOS)
Engine-Mode: "ENABLED"


Here you can see the headers and the body for both the request and the response. They are coded by the particular request (2ab6ad6a) and the part (e.g. A, B, … H, Z)

By default, the audit log is stored in the file /var/log/modsec_audit.log, and there is no need to modify this from the default.

ModSecurity Core Rules

At this point, the installation of ModSecurity is complete. However, this does not mean that ModSecurity is doing much. In fact, so far all it is doing is throwing an alert whenever the text "zzz" appears in the arguments of a request- and we should probably remove that test rule too.

Though the ModSecurity module is now inspecting the traffic coming to and from the web server, we really have not given it much information on what sort of traffic should be blocked or what should be allowed. To that end, we turn to OWASP and the OWASP ModSecurity Core Rule Set. These are a set of rules for ModSecurity that will let us detect and block a number of common attacks on web applications.

Begin by grabbing the core rule set, either from the labshare or online. Unpack the package in a convenient location and move the result to /etc/httpd/crs.

[root@excalibur ~]# tar -xzvf ./SpiderLabs-owasp-modsecurity-crs-2.2.7-11-
[root@excalibur ~]# mv ./SpiderLabs-owasp-modsecurity-crs-52370fc/ 

The rule set comes with an example configuration file; we rename that file:

[root@excalibur crs]# mv modsecurity_crs_10_setup.conf.example 

If you look at the structure of the /etc/httpd/crs/ directory, you will see separate directories for activated rules, base rules, experimental rules, optional rules, and slr (Spider Labs) rules. To make our lives simplest, we can put symbolic links in the activated rules directory that point to the various rule sets that we want to use.

We can do this manually, or via a short one line bash script. Run the following (as root)

[root@excalibur crs]# for f in `ls /etc/httpd/crs/base_rules/` ; do 
ln -s /etc/httpd/crs/base_rules/$f /etc/httpd/crs/activated_rules/$f ; 

Note how it loops over all of the base rules, and creates a symbolic link (hence the -s) in the activated rules directory. Check the contents of that directory when you are finished to see that the links were created.

You can add links to the optional, experimental, and Spider Labs rules as you see fit.

Next, we need to modify the Apache configuration to correctly load these files. We do so by adding

# Add ModSecurity Rules Settings
<IfModule security2_module>
   Include crs/modsecurity_crs_10_config.conf
   Include crs/activated_rules/*.conf

With this complete, simply restart Apache.

Now suppose that an attacker is trying to exploit a vulnerable web application; in many such vulnerabilities, the attacker needs to null terminate a string. Suppose for simplicity that the string is being passed as a GET parameter; the usual way to null terminate a GET parameter is to encode the null using a URL in the form


Here we see the null byte to the end of the GET parameter a.

If a user makes a non-malicious request of our ModSecurity protected web server, they get the correct response:
Screenshot-Mozilla Firefox

On the other hand, if they pass a null at the end of the GET parameter, they end up at a default page:
Screenshot-Apache HTTP Server Test Page powered by CentOS - Mozilla Firefox
and they end up in the modsec_audit.log

[02/Mar/2014:12:11:27 --0800] UxOQb38AAAEAAEUvDEkAAAAD 47969 80
GET /?a=string%00 HTTP/1.1
Host: excalibur.cosc.tu
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Firefox/17.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive

HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 5039
Connection: close
Content-Type: text/html; charset=UTF-8

Message: Access denied with code 403 (phase 2). Found 1 byte(s) in ARGS:a 
outside range: 1-255. [file "/etc/httpd/crs/activated_rules/modsecurity
_crs_20_protocol_violations.conf"] [line "534"] [id "960901"] [rev "2"] 
[msg "Invalid character in request"] [severity "ERROR"] [ver "OWASP_CRS
/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATI
Action: Intercepted (phase 2)
Stopwatch: 1393791087640722 1381 (- - -)
Stopwatch2: 1393791087640722 1381; combined=304, p1=221, p2=54, p3=0, 
p4=0, p5=29, sr=24, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.7.7 (http://www.modsecurity.org/); 
Server: Apache/2.2.15 (CentOS)
Engine-Mode: "ENABLED"


Multiple IP Addresses

You may want to set up a host with multiple external IP addresses. These separate IP addresses can also have their own separate names. Since most services allow you to specify the IP address(es) on which the service listens, one physical host can appear to the network to be multiple servers with multiple addresses and multiple host names.

To set this up in, for example, CentOS, you can modify the network configuration scripts in /etc/sysconfig/network-scripts.

Create a new file, called ifcfg-eth0:0; copy it from your existing /etc/sysconfig/network-scripts/ifcfg-eth0. Modify the Device name in that file to read eth0:0. Modify the static IP address to whatever value you wish. You cannot use DHCP, but you can delete the gateway specification. Do not modify the hardware MAC address. The resulting file will look something like the following:

[root@terminus ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0:0

After a reboot, you will see both interfaces in ifconfig:

[[zathras@excalibur ~]$ ifconfig 
eth0      Link encap:Ethernet  HWaddr 00:0C:29:3E:1A:02  
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::20c:29ff:fe3e:1a02/64 Scope:Link
          RX packets:478 errors:0 dropped:0 overruns:0 frame:0
          TX packets:107 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:43260 (42.2 KiB)  TX bytes:12185 (11.8 KiB)

eth0:0    Link encap:Ethernet  HWaddr 00:0C:29:3E:1A:02  
          inet addr:  Bcast:  Mask:

lo        Link encap:Local Loopback  
          inet addr:  Mask:
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:480 (480.0 b)  TX bytes:480 (480.0 b)

This process is called setting an alias for an interface. A similar process can be used on other Linux distributions.

Because your host now has two IP addresses, it can also have two host names, provided the DNS server is appropriately configured.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: