To run a web service you need four things: a connection to the outside world (network), a machine to run the service from (hardware), a program to run it with (software) and people to maintain both the server and the data it serves (‘wetware’).

Your network needs to be a permanent connection and your server needs to have a constant IP address. You neeed to know what the support is for your network, who to contact in case of problems, when the vulnerable periods are, etc. (The CUDN has Monday to Saturday 0800–0930,1700–1900 and Sunday 0000 to Monday 0800 for its vulnerable periods.)

Your machine must have the power to support the number of hits the server will get. Note that it must be powerful to cope with the peak demand and not just the mean or modal demand. The most important element of your hardware for a web server is the network card; buy a good one. Next most important is the amount of RAM. The CPU comes last in the list. Unless you are planning to run very computationally expensive CGI programs you don't need the latest, greatest, fastest chip in the world.

Clearly you neeed a good web server program. The program described in this course is Apache. It has all the facilities you could want (and then some), is free. (and that's ‘free’ as in speech as well as in beer. It is also the most widely used web server on the Internet, being used by >65% of active sites. (Source: Netcraft Web Server Survey, February 2002.)

Finally, it is important to realise that to provide a web service rather than just a server you need people. Pages get dated, users' needs change, links pointing out of your site go stale, dump tapes need to be changed and error reports need to be addressed.

A web server serves out web pages. However, to populate the web site the pages need to be written and checked and log files may need to be analysed.

Firstly you will need to write the web pages, or possibly edit those submitted by others. The author still regards a plain text editor (emacs, vi or perhaps even pico -w) as the best tool for editing web pages. Contrary to popular belief, the dedicated web authoring tools are still not very good. Of the various authoring packages by far the worst is Microsoft Word's ‘save as HTML’ feature. The quality of HTML generated by this is appalling and it should be avoided like the plague.

The HTML in the page still needs to be checked for syntax, link integrity and accessibility. This is true whether or not a dedicated HTML authoring package was used; indeed, if one was used then a ‘second opinion’ is all the more important. The text itself should also be checked for spelling and grammar, but beware the rather over-simplistic grammar rules in some word processors.

In addition to the text there are all the other media formats, with static graphics the most common. The GNU Image Manipulation Program (the GIMP), gimp, is a massively powerful GUI image manipulator that starts at the level of Adobe Photoshop but takes things much further, including having a scripting language. For simple viewing, croping, rescaling and format conversion the Electric Eyes program, ee, is considerably simpler to learn and use. Images can be initially created interactively (e.g. with the GIMP), with a digital camera or by scanning in photographs.

The other support you may need is for CGI programs. First you need to make a decision: are you going to permit the running of programs on the server? While this course is not going to review the technologies in any depth it should give some idea of the spectrum available and the dangers asssociated with them.

The author is aware of only one vulnerability of a system through the web server itself. He is, however, aware of many break-ins via the CGI programs run by web servers. Static pages are vastly more secure.

The simplest of this style of program is the ‘server side include’ facility. This allows you to add certain tags to a web page which are not valid HTML but which are transformed by the server into valid HTML with dynamic content. A common example is the SSI tag that says when the page was last modified. However, consider whether you want the ‘last updated’ tag to be the last time you fixed your spelling (automatic version) or the last time you changed the content (manual version). Slightly beyond this is the ‘server side include executable’ where the tag runs an external program to generate the content. It is at this point, where an extra program is run by the web server, that you need to start being very careful about security. (It is possible to turn off the SSI executable feature while retaining the weaker SSI functionality.)

PHP takes this one stage further, offering a scripting language embedded in the HTML to provide powerful functionality and logic. The Perl and Python CGI modules take the page author away from HTML all together. The CGI modules are presented with a URL (and some input data for POST queries) and have to write their own HTTP as well as HTML, in the format described for the ‘as is’ pages in the section called “Writing HTTP rather than HTML”. The modules provide simple function calls for most of this though.

Finally, unless you plan to work exclusively at the console (you don't) you will need secure network access to your server. Don't use telnet or the ‘r-commands’ (rlogin, rsh, rcp and rsync) but their secure analogues provided by the ‘ssh’ suite of programs.

Red Hat Linux version 7.0 and above ship with an SSH system. Also, Unix Support provides a CD with ssh clients for most platforms including a Red Hat Linux packaging of the software suite for the Intel platform. The CD is free from the CS Reception.

The example server we are going to use for this course is a 700MHz Athlon with 256MB of RAM a 1GB disc and a 3Com 3c905B card. This is adequate for a production server. If it was very heavily used I would increase the disc size. The RAM and the CPU are perfectly adequate.

We will be running Red Hat Linux 7.3. Typically we would not be running X on the web server but we will for this example because we will be our own client too. We will run with Apache 1.3.23 which is the version shipped with Red Hat Linux 7.3.

# mount -o ro nfs-uxsup.csx.cam.ac.uk:/linux/redhat /mnt
# cd /mnt/updates/7.3/en/os/i386/
# ls -l apache-*
-rw-r--r--    ...    apache-1.3.23-14.i386.rpm
-rw-r--r--    ...    apache-devel-1.3.23-14.i386.rpm
-rw-r--r--    ...    apache-manual-1.3.23-14.i386.rpm

# rpm --query --info --package apache-1.3.23-14.i386.rpm
Name        : apache                     Relocations: (not relocateable)
Version     : 1.3.23                          Vendor: Red Hat, Inc.
Release     : 14                          Build Date: Wed 19 Jun 2002 16:55:48
Install date: (not installed)             Build Host: daffy.perf.redhat.com
Group       : System Environment/Daemons  Source RPM: apache-1.3.23-14.src.rpm
Size        : 1248999                          License: Apache Software License
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Summary     : The most widely used Web server on the Internet.
Description :
Apache is a powerful, full-featured, efficient, and freely-available
Web server. Apache is also the most popular Web server on the
Internet.

# rpm --query --list --package apache-1.3.23-14.i386.rpm
/etc/httpd/conf
/etc/httpd/conf/httpd.conf
 ...
/etc/rc.d/init.d/httpd.init
 ...
/var/www
/var/www/html
/var/www/html/index.html
/var/www/icons
/var/www/icons/a.gif
 ...
/usr/man/man8/httpd.8
 ...
/usr/sbin/httpd
 ...

# rpm --install apache-1.3.23-14.i386.rpm
# cd
# umount /mnt

We install Apache as root and then configure it so that root will not be needed subsequently for the configuration or administration of the server except to shut it down or restart it.

We use the network file system (NFS) to mount Unix Support's mirror of the Red Hat distribution. Within it (/mnt/updates/7.3/en/os/i386/) are all the software packages, including Apache (apache-1.3.12-2.i386.rpm).

We examine the Apache package for information and a listing of its contents and finally we install it. Once we've done the installation we unmount the file server.

This installation has not started the server but has arranged that it will be started on the next reboot. (Though we don't need to and won't reboot just to start it.)

               +--- conf/ ---+--- *.conf
               |                               +--- access.log
/etc/httpd/ ---+--- logs -> /var/log/httpd/ ---+
               |                               +--- error.log
               +--- modules -> /usr/lib/apache

            +--- cgi-bin/                  empty
            |
/var/www/---+--- icons/  --- *.gif
            |
            +--- html/   --- index.html    default

The files installed come in three classes: the configuration files (/etc/httpd/, /etc/logrotate.d/apache) that the server managers need access to, the data files (/var/www/) that the web page authors and editors need access to and the system files (everything else) that we aren't going to touch. We will define groups to keep these categories apart.

If you were prepared to do all the updates to the web pages as root and had no special requirements such as access controls then you could just run the program now. The website exists under /var/www/html/ and I wish you much happiness with rogether. However a small amount of work (typically about 15 minutes) will make everything a lot easier and safer.

# groupadd -r webadmins
# groupadd -r webeditor
# vi /etc/group

# ls -ld /var/www /etc/httpd /var/log/httpd
drwxr-xr-x 3 root root     1024 Jun 27 12:09 /etc/httpd
drwxr-xr-x 5 root root     1024 Jun 27 12:09 /var/www
drwxr-xr-x 2 root root     1024 Jun 27 16:36 /var/log/httpd

# ls -ld /var/www /etc/httpd /var/log/httpd /etc/logrotate.d/apache
drwxrwsr-x 3 root webadmins 1024 Jun 27 12:09 /etc/httpd
-rw-rw-r-- 1 root webadmins  172 Jun 27 12:09 /etc/logrotate.d/apache
drwxrwsr-x 5 root webeditor 1024 Jun 27 12:09 /var/www
drwxrwsr-x 2 root webadmins 1024 Jun 27 12:09 /var/log/httpd

We create system groups for the administration of the server and the management of the web pages. On a small server these can be combined. On large servers you may well want multiple groups to manage various subsets of the web pages. We specifically want to avoid requiring root access to reconfigure the web server. root will be used to start or stop the server and nothing else.

We set the permissions on the data and configuration directories so that members of the relevant group can make changes (g+w on files and directories) and any files or subdirectories created will have matching group ownership (g+s on directories).

The chmod (change the mode (permissions) of a file system object) and chgrp (change the group ownership of a file system object) commands (and the chown command which changes the user ownership of a file system object, though we aren't using that command here) have a -R option to make them behave recursively. Every file system object beneath the named directories will have their mode or group modified.

The find command is slightly trickier. We want to apply the g+s mode change to every directory beneath the named directories but we don't want to apply it to the files. The find command shown starts at each of the three directories listed and checks each file system element beneath them, testing to see if the element in question is a directory (‘-type d’). If it is then it executes a command (‘-exec ... \;’) and that command is ‘chmod g+s dir’. (‘{}’ is replaced by the name of the file system element being considered.)

# /etc/rc.d/init.d/httpd start
Starting httpd:                                            [  OK  ]

While we're here, we shall describe the manual stopping of the server, which we will hardly ever need, and the manual restarting of the server which we will use frequently in this course to bring in a new configuration file. Restarting is just stopping and starting wrapped into a single command.

# /etc/rc.d/init.d/httpd restart
Shutting down http:                                        [  OK  ]
Starting httpd:                                            [  OK  ]

# /etc/rc.d/init.d/httpd stop
Shutting down http:                                        [  OK  ]

Red Hat's packaging of Apache's configuration files echoes an obsolete format of having three distinct configuration files in the /etc/httpd/conf/ directory. In this course we will put all our configuration in the single file: /etc/httpd/conf/httpd.conf and we will write this file from scratch to better learn what it all means.

The only other configuration file we will need is the log rotation file in /etc/logrotate.d/apache. We need to change this only if we change either the log files being kept or the duration they are kept for. These two reasons take on extra significance given the lunacy of the 1998 Data Protection Act. The client machine names or addresses that appear in logs and your record of what they have fetched may constitute personal data. We will return to this file in the section called “Log rotation”

ServerType standalone
ServerRoot /etc/httpd
DocumentRoot /var/www/html
Port 80
User apache
Group apache
ServerAdmin rjd4@cam.ac.uk
ServerName www.inst.cam.ac.uk
ErrorLog /var/log/httpd/error_log
LogLevel info
Options None

PidFile /var/run/httpd.pid
LockFile /var/lock/httpd.lock
ScoreBoardFile /var/run/httpd.scoreboard
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 20
StartServers 8
MaxClients 150
MaxRequestsPerChild 100

We can run a web server with just the configuration lines we have met so far. It will be not very good, to say the least. Its principal failing is that it has no concept of the MIME content types of the objects it serves and dishes everything up as MIME content type ‘text/plain’. If we look at http://localhost/index.html we see the HTML source (because the browser has been told that the document is of type text/plain). We need to add some functionality to Apache: the ability to determine what MIME content type a document is.

Apache's functionality comes in a set of files called ‘modules’. We start by clearing any default modules built into the system by default. Without this line many modules would be available by default. Partly because this is a lesson and partly because all good system administrators are control freaks (regarding the systems, not the users!) the only modules used here will be the ones we explicitly add. The mod_so.c module is built in to the Apache binary. But because we have cleared the module list it is not turned on by default. This is the module that allows us to load extra modules that are not built into the binary.

# Start with an empty module list

ClearModuleList
AddModule mod_so.c

Options +FollowSymLinks

The server at the moment also doesn't respect symbolic links, refusing to follow them either for pages or directories. Following symbolic links is an option under Apache and, as you will recall, we turned off all options so that we would notice them. There are two options relevant to symbolic links: FollowSymLinks and SymLinksIfOwnerMatch.

The Options directive has a catch. If we give the line

Options FollowSymLinks

Options +FollowSymLinks

LoadModule mime_module        modules/mod_mime.so
AddModule mod_mime.c

TypesConfig /etc/mime.types
DefaultType text/plain

AddEncoding x-compress Z
AddEncoding x-gzip gz tgz

Now we see our first use of an external module. The syntax for the process is rather obscure. This is unfortunate but nothing we can't handle.

When a module is activated some commands are added to the set permitted in the configuration file. The three directives used here (‘TypesConfig’, ‘DefaultType’ and ‘AddEncoding’) are all provided by mime_module module and would be invalid without the preceding LoadModule and AddModule lines.

Next we will consider those extra commands that the mod_mime module adds. Unless the module is loaded and added before these commands are used they will result in a syntax error.

# MIME type                 Extension
application/activemessage
application/andrew-inset    ez
application/applefile
application/mac-binhex40    hqx
application/octet-stream    bin dms lha lzh exe class
application/postscript      ai eps ps
application/x-dvi           dvi
application/x-javascript    js
image/gif                   gif
image/jpeg                  jpeg jpg
image/x-xwindowdump         xwd
message/partial
message/rfc822
model/vrml                  wrl vrml
text/plain                  asc txt
text/html                   html htm

At the moment we are only logging errors. There is an independent mechanism to log transfers and it comes as a module. Furthermore, we have no means to deal with the log files generated. This section will address the first issue and the following will address dealing with log files once we've got them.

LoadModule config_log_module  modules/mod_log_config.so
AddModule mod_log_config.c

HostnameLookups On
IdentityCheck Off

CustomLog /var/log/httpd/access_log "%t %h \"%r\" %>s %B"

The CustomLog directive takes two arguments. The first is the file name to log to and the second is the format of the log itself. The format line consists of a series of ‘escape sequences’ (the things starting with percentage characters). Each of these is replaced by some piece of information about the request or the server's response to it. There is no reason why you should not have more than one log file; you just have multiple CustomLog lines each defining a different log file.

The simple escape sequence is ‘%X’ for some value of ‘X’. See the figure for the most useful examples. It is possible to log an arbitrary header from the query or response. For the server it is usually of more use to see the incoming headers. See the syntax description for some examples. Of most use in log files is the referring page. For example, you could strip out just those log lines with status code 404 (page not found) and check the refering page. If it's an internal page you can fix the link and if it's external you can contact the webmaster responsible.

The %h code requires the server to perform a lookup in the DNS to turn the IP address of the incoming request into a name. This is not an expensive operation, but if your web site is very heavily used you may want to avoid it. There are two ways to go about this. You can use %a instead. This just logs the IP address and attempts no lookup. Alternatively you can use %h but set the directive HostnameLookups Off. Under these circumstances %h behaves like %a. However, if you want to do access control based on client host name you must have HostnameLookups On, hence the provision of %a.

The %l escape also requires some explanation. The ident protocol provides a means for the server to ask of the client the name of the user on the client (or some tag uniquely identifying the user) who is making the connection. This is only possible if the client system is running the corresponding ident server. This server is quite common on multi-user systems and almost unknown on single-user systems. Again, the load is small for a lightly loaded web server but potentially severe for a heavily loaded one. (Far more so than for the hostname lookups.)

Finally we need to explain the ‘%>s’ construction. We will see in a later section that some modules run a page through quite intricate processing. ‘%s’ is the status code for the processing of the query and ‘%>s’ is the status code finally returned to the client. The latter is typically what we really want. The figure below lists the most commonly seen status codes. The full set can be found in RFC 2616.

The escape sequences can be more involved than this. Full details are in the Apache documentation.

The %i logging option records the value of an incoming, request header. The most commonly useful headers are given below.

[17/Apr/2000:10:10:25 +0100] hostname "GET /index.html HTTP/1.0" 200 1316
[17/Apr/2000:10:11:00 +0100] hostname "GET /bogus.html HTTP/1.0" 404 0
[17/Apr/2000:10:12:00 +0100] hostname \
 "GET http://elsewhere/index.html HTTP/1.0" 200 1316
[17/Apr/2000:10:30:23 +0100] hostname \
 "GET /cgi-bin/phf?Qalias=x%0a/bin/cat/%20/etc/passwd HTTP/1.0" 404 0

The figure has four example log lines in the format defined in our configuration file.

[17/Apr/2000:10:10:25 +0100] hostname "GET /index.html HTTP/1.0" 200 1316

The first line shows a succesful transfer of the URL http://machine/index.html. Note that the client need only request the local part of the URL having determined what machine to connect to itself.

[17/Apr/2000:10:11:00 +0100] hostname "GET /bogus.html HTTP/1.0" 404 0

The second line shows an unsuccessful transfer request. The file being looked for does not exist (status code 404). Note that the logged number of bytes sent back is 0.

[17/Apr/2000:10:12:00 +0100] hostname "GET http://elsewhere/index.html HTTP/1.0" 200 1316

The third line is an example of someone trying to use the server as a proxy server. If a request comes in for a fully qualified URL some servers (and Apache if you configure it appropriately) will act as a web client, fetch that URL and pass it back to you. By default Apache does not do this. Instead, it ignores the http://elsewhere component and treats it as a request for the local URL /index.html. Note that this request generates a status code 200 and returns 1316 bytes—exactly the same number as in line one.

[17/Apr/2000:10:30:23 +0100] hostname \
 "GET /cgi-bin/phf?Qalias=x%0a/bin/cat/%20/etc/passwd HTTP/1.0" 404 0

The fourth line is an example of an unsuccesful hacking attempt. The phf script has a hole permitting arbitrary shell commands to be run. Note that these would have run as the user apache which has no special privilege, but it is still a way in.

The Data Protection Act (1998). The Data Ptotection Comissioner's office has advised us that machine names and IP addresses that can be used to identify an individual (e.g. that of the computer in a student's room) may constitute personal data in the meaning of the DPA(98). Until there is an expensive test case and some ignorant, senile, senior judge pronounces precedent we won't know for certain.

In this section we consider what we can do with the logs and, in particular, how to stop them growing out of control.

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# send errors to root
errors root

# create new (empty) log files after rotating old ones
create

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

/var/log/httpd/access_log /var/log/httpd/error_log {
    missingok
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
}

Red Hat Linux provides a service called ‘log rotation’ which provides a uniform mechanism to stop log files growing out of control over time. At regular intervals (nightly, weekly and monthly are all common) the log file error_log, say, is renamed to error_log.1. If there was a previously existing error_log.1 it is renamed to error_log.2, error_log.2 to error_log.3 and so on up to some limit. The default frequency of this operation is defined in the file /etc/logrotate.conf to be weekly and the number of log files kept is set to default to 4. error_log.3 is discarded rather than renamed to error_log.4. A new error_log is created.

The directory /etc/logrotate.d/ contains the rotation instructions specific to the log files for a particular package. The log files for the apache package are kept in the file /etc/logrotate.d/apache. These are given as /var/log/httpd/error_log and /var/log/httpd/access_log. The empty brackets after the /var/log/httpd/error_log line means that there is no special action needed after the error log file has been rotated. The three lines in the brackets after the /var/log/httpd/access_log line identify a (single line) shell script that should be run after the access log file has been rotated. This sends the HUP signal to the web daemon which causes it to reopen all its log files so that it is now logging to the newly created log files rather than the .1 versions.

While this course does not consider the analog log analysis program, we will remark that the log rotation script is a good place to run it from. Each time the system rotates a log file, analog gets to process it. We might also want to address the DPA(98) issues here by insisting that the log files not be world-readable. The create line stipulates that when the logs files are rotated a new, empty one is to be created which is read/write to root, read-only to members of group webadmins and not readable at all by anyone else.

/var/log/httpd/access_log /var/log/httpd/error_log {
    missingok
    sharedscripts
    create 0640 root webadmins
    postrotate
        /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
}

Figure 56. Resolving a URL to a file via an alias

By default, the ‘local part’ of any URL is converted to a file name by simply resolving it as a file name relative to ServerRoot, which is /var/www/html/ on a Red Hat Linux installation. So, for example, the URL http://server/wombat/index.html would resolve to the file /var/www/html/wombat/index.html.

However, sometimes we want a URL to point out of the ServerRoot directory tree. For example we can see that the Red Hat Linux Apache installation puts a collection of GIF files in /var/www/icons/ which is not below /var/www/html/. We might want the URL http://server/icons/new.gif to resolve to the file /var/www/icons/new.gif which it won't by default.

We can accomplish this in two ways: either we create a symbolic link from /var/www/html/icons to /var/www/icons/ or we tell Apache to override the ServerRoot setting in certain regards. As this is an Apache course, we will do the latter.

# Aliases

LoadModule alias_module       modules/mod_alias.so
AddModule mod_alias.c

Alias /icons/  /var/www/icons/

As before, to add functionality to Apache we need a module. In this case it is the mod_alias module. This module adds a number of keywords to the configuration syntax but we need only one for now. In the slide the Alias directive maps a set of URLs with local parts starting /icons/ to the directory /var/www/icons/.

[27/Apr/2000:15:47:11 +0100] hostname "GET /index.html HTTP/1.0" 200 2537
[27/Apr/2000:15:48:09 +0100] hostname "GET / HTTP/1.0" 404 0

At the moment, while our web server can handle files, determine their MIME content and encoding types from their names' extensions and log their transfer, it still can't handle URLs that resolve to directories. Attempts to get such a URL (e.g. the top level URL for the site as a whole) give 404 errors. This is clearly unacceptable.

There are two ways to handle this and most sites implement both.

The first is to provide automatic indexing. Given a URL corresponding to a directory, the web server will create an HTML web page giving a list of all the entries in that directory. These can be annotated with icons (or their ALT text) to identify the corresponding MIME content types. They can be labelled with sizes, titles etc. or left completely plain. We will start with the basic functionality (and the relevant module) and slowly add in some flashier functions.

The other approach is to nominate one or more filenames so that if such a file exists within a directory then that file will be displayed instead. The name index.html is traditional for this, but is not compulsory.

# Automatic indexing of directory URLs

LoadModule autoindex_module   modules/mod_autoindex.so
AddModule mod_autoindex.c

Options +Indexes

                    Index of /
     * Parent Directory
     * index.html
     * poweredby.png

IndexOptions +FancyIndexing

                          Index of /

Name                Last modified       Size  Description
  __________________________________________________________________
 Parent Directory    25-Apr-2000 14:00      -
 index.html          25-Apr-2000 18:08     2k
 poweredby.png       01-Mar-2000 18:37     1k
     _____________________________________________________________

IndexOptions +SuppressLastModified +ScanHTMLTitles

                          Index of /

Name                Size  Description
  __________________________________________________________________

 Parent Directory        -
 index.html             2k  Test Page for the Apache Web Server on Re>
 poweredby.png          1k
     _____________________________________________________________

IndexOptions IconWidth IconHeight

AddIconByType (HTM,/icons/layout.gif)   text/html
AddIconByType (TXT,/icons/text.gif)     text/*
AddIconByType (IMG,/icons/image2.gif)   image/*
AddIconByType (MOD,/icons/world2.gif)   model/*
AddIconByType (SND,/icons/sound2.gif)   audio/*
AddIconByType (VID,/icons/movie.gif)    video/*

IndexOptions IconWidth=X IconHeight=Y

AddIconByType (_PS,/icons/a.gif)        application/postscript
AddIconByType (PDF,/icons/a.gif)        application/pdf
AddIconByType (HQX,/icons/binhex.gif)   application/mac-binhex40
AddIconByType (DVI,/icons/dvi.gif)      application/x-dvi
AddIconByType (TEX,/icons/tex.gif)      application/x-tex
AddIconByType (TAR,/icons/tar.gif)      application/x-tar
AddIconByType (BIN,/icons/binary.gif)   application/octet-stream
AddIconByType (XXX,/icons/unknown.gif)  application/*

AddIcon (_UP,/icons/back.gif)   ..
AddIcon (DIR,/icons/folder.gif) ^^DIRECTORY^^
AddIcon (---,/icons/blank.gif)  ^^BLANKICON^^

                          Index of /
        Name                    Size  Description
  __________________________________________________________________________
 [_UP]  Parent Directory            -
 [HTM]  index.html                 2k  Test Page for the Apache Web Server on R
e>
 [DIR]  manual/                     -
 [IMG]  poweredby.png              1k
     _________________________________________________________________

HeaderName HEADER.html
ReadmeName README.html

   This is some text to go at the top of the page above the listing.
        Name                    Size  Description
  __________________________________________________________________________
 [_UP]  Parent Directory            -
 [HTM]  HEADER.html                1k
 [HTM]  README.html                1k
 [HTM]  index.html                 2k  Test Page for the Apache Web Server on R
e>
 [DIR]  manual/                     -
 [IMG]  poweredby.png              1k
     _________________________________________________________________

IndexIgnore .??* *~ *# HEADER* README* SCCS RCS CVS

# Default files in directory URLs

LoadModule dir_module         modules/mod_dir.so
AddModule mod_dir.c

DirectoryIndex index.html index.htm

We saw in the logging section that HTTP (the transfer protocol, not the language of the web pages) has the concept of status codes, with 200 being the ‘OK’ response and 404 being the ‘file not found’ response. From time to time, we may want to force the generation of a particular error message or status code. There are two ways to go about doing this.

The core Apache system has a directive called ErrorDocument. This lets us specify exactly what page will be sent back to accompany a 404, say, status code.

ErrorDocument 404 /errors/404.html
ErrorDocument 500 "Oops, server goof."

This depends on the server generating a specific status code. You will recall that status code 403 corresponds to ‘forbidden’. We might want to indicate that trying to fetch a particular URL was expressly forbidden rather than just not present. For example, given a directory URL, we might want to display an index.html file if one exists but give a 403 status code if one does not. So we need a way to generate pages with status codes of other than 200. We could do this just by turning off or on the indexing option but the mechanisms described here provide more flexibility.

This functionality is provided by a module called mod_asis. This lets us provide web pages that aren't HTML or any other MIME type but which are the entire HTTP response to a query. This allows us to add status codes and other HTTP metadata beyond just the HTML content.

First let's see what a full HTTP session looks like.

$ telnet draig.csi.cam.ac.uk 80
Trying 131.111.10.224...
Connected to draig.csi.cam.ac.uk.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.1 200 OK
Date: Tue, 16 May 2000 08:54:29 GMT
Server: Apache/1.3.12 (Unix)  (Red Hat/Linux)
Last-Modified: Tue, 25 Apr 2000 17:08:10 GMT
ETag: "f242-9e9-3905d0fa"
Content-Length: 2537
Connection: close
Content-Type: text/html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
  <HEAD>
 ...
 </BODY>
</HTML>

So, if we are going to generate a status code 403, say, then we will need to create that first line and perhaps some others. The module will assist us with many of them, though.

The module works as follows: we create a new, fake MIME content type called httpd/send-as-is and associate it with files ending with one or more suffixes (.asis, traditionally). The module then causes the server to process these files as nearly raw HTTP rather than as HTML or some other MIME content type. Because httpd/send-as-is is not a true MIME type, we don't want to define it in the /etc/mime.types file, so we use the AddType directive of the mod_mime module to define it purely within the web server. This gives us a module dependency: the mod_asis module cannot be used without the mod_mime module already being added.

# Send .asis files "as is"

AddType httpd/send-as-is asis

LoadModule asis_module        modules/mod_asis.so
AddModule mod_asis.c

Now, if we wanted to provide for forbidding directory indexing in certain directories as opposed to providing an index.html file, we could provide the DirectoryIndex line

DirectoryIndex index.html index.asis

Let's now look at constructing a plausible index.asis file.

Status: 403 Directory searching is prohibited
Content-Type: text/html

<!DOCTYPE HTML PUBLIC
 "-//W3C//DTD HTML 4.0 Transitional//EN"
 "http://www.w3.org/TR/REC-html40/strict.dtd">

<HTML><HEAD>
<TITLE>Security policy violation</TITLE>
</HEAD><BODY>
<H1>Security policy violation</H1>
<P>This web site's security policy prohibits the autoindexing of this
directory.  Your request has been logged.</P>
</BODY></HTML>

A more useful page would give links to a search engine and the such like. More importantly, observe the headers at the start of the page, split from the body by the first blank line of the page. (The line is truly empty; there are no spaces or other whitespace characters in it.) The Status: header introduces the status code and the explanatory text message. We don't get to specify the HTTP version being spoken; the server will take care of that for us. Any following lines (before the blank line) that look like HTTP headers will be passed through untouched and must be valid HTTP header lines. The server will add the Server:, Date: and Connection: lines and we should not write these.

$ telnet draig.csi.cam.ac.uk 80
GET /two/ HTTP/1.0

Trying 131.111.10.224...
Connected to draig.csi.cam.ac.uk.
Escape character is '^]'.
Connection closed by foreign host.
HTTP/1.1 403 Directory searching is prohibited
Date: Tue, 16 May 2000 11:30:40 GMT
Server: Apache/1.3.12 (Unix)  (Red Hat/Linux)
Connection: close
Content-Type: text/html

<!DOCTYPE HTML PUBLIC
 "-//W3C//DTD HTML 4.0 Transitional//EN"
 "http://www.w3.org/TR/REC-html40/strict.dtd">

<HTML><HEAD>
<TITLE>Security policy violation</TITLE>
</HEAD><BODY>
<H1>Security policy violation</H1>
<P>This web site's security policy prohibits the autoindexing of this
directory.  Your request has been logged.</P>
</BODY></HTML>

If we inspect the access log file we will see the 403 lines there too.

[16/May/2000:12:06:30 +0100] hydra.csi.cam.ac.uk "GET /two/ HTTP/1.0" 403 345

This is where we get to see the difference between the logging code ‘%s’ and ‘%>s’. The former would log a status code of 200 because the .asis file was processed correctly. The latter shows 403 because that is the ultimate status code after all the internal reprocessing is complete.

# Users' web pages

LoadModule userdir_module     modules/mod_userdir.so
AddModule mod_userdir.c

UserDir public_html

Apache contains a mechanism (read module) that allows users to supply their own web pages. It is not uncommon for a web server to offer nothing but these and to have no ‘central’ web pages at all (except perhaps for a top level index.html file. The mod_userdir module provides a single directive, UserDir. It can be used in a number of ways, however.

So far, our editing of the file httpd.conf has set parameters for the entire server. On occasion it is appropriate to have one set of parameters for one set of web pages and another for other parts. We need some way to pass directives applicable just to a certain set of pages. There are a number of ways to describe subsets of pages and Apache supports them all. We will restrict ourselves to just the simplest in this course.

The simplest is by considering subtrees of the web pages. We can tag a directory with some special options and have those apply to every web page beneath it. For example we might want to restrict access to everything under /var/www/html/restricted/.

We might want to tag multiple directory trees by applying these overrides to every directory that matches a regular expression, rather than by specifying its explicit name. For example, anything under any directory called restricted might get special rules.

We might just specify a regular expressions that matches files and apply the rules to any files (as opposed to directories) that match. So any web page called special.html might get nonstandard rules.

Alternatively, rather than specify the restriction by file name (after the URL has been resolved) we might change the rules according to the URL quoted before this gets mapped onto a file (or directory) name. Again this could be a subtree of URLs or the set of URLs that match a regular expression.

Using the directory structure to control options also permits the placing of special files in the directory structure to control the trees beneath them (traditionally called .htaccess). These control files, in turn, might benefit from the filename matching rules to stop their being fetched by clients.

While these all make sense in isolation, the combination of rules governing directory trees, URLs, filename regular expressions and URL regular expressions is a recipe for trouble. We are going to approach this issue from the KISS (‘Keep It Simple, Stupid!’) standpoint and restrict ourselves to directory subtrees here.

Our configuration file will run with

DirectoryIndex index.html

Options +Indexes

Options -Indexes

# Default
Options +Indexes

# Subdirectory restriction
<Directory /var/www/html/fubar/>
Options -Indexes
</Directory>

The <Directory> tag limits the application of parts of the configuration to just those files and directories beneath /var/www/html/fubar/.

We start to see a problem here, though. Inevitably, the directory structure will get larger and larger. The set of overrides and rules will get longer and longer. More and more people will need access to the httpd.conf file. More and more lines will get added to it. This is bad. What is needed is a way to delegate the controls over a directory tree to the directory itself. This facility exists, using control files in some directories, traditionally called .htaccess. The file can, however, have any name we choose to give it. However, before we start delegating control we might want to restrict just what configurations in the httpd.conf file we are prepared to have overridden in the delegated control files.

AccessFileName .config

<Directory /var/www/html>
AllowOverride AuthConfig FileInfo Indexes
</Directory>

The delegated control file was originally used to control access to subtrees of web pages (and we'll see how to do this soon) and the name of the directive that sets it (AccessFileName) reflects that history. It is a more general overriding facility, though, so to reinforce that, we'll use the name AccessFileName directive to set the name of the delgated configuration file to be .config.

The second line specifies what facets of the Apache configuration can be overridden in the .config files. This aspect of the Apache control mechanim is not as refined as it might be, unfortunately. Any directive that appears in the httpd.conf file and which ‘makes sense’ applied to a directory tree (more precisely, any directive that could appear in a <Directory>...<Directory/> block) can be placed in this subconfiguration file.

Now let's return to the case study in the slide. We will drop the subdirectory directives entirely and instead specify what overrides we will permit and in what file. We then have a .config directory that changes the options again.

# Default
Options +Indexes
AccessFileName .config
<Directory /var/www/html>
AllowOverride Options
</Directory>

Options -Indexes

Now we start to see Apache creak at the seams. Note that to change the nature of indexes (using the IndexOptions directive) we would need to allow the override Indexes. However, because turning automatic indexing on or off (Options +Indexes or Options  -Indexes) is handled by the Options directive we have to permit the Options override. This is unfortunate because there are other suboptions to Options that we might not want to delegate. Mercifully, indexing is the exception rather than the rule in this regard. In most other cases the controls over delegation do make sense.

The next question to address is how these files nest. If I have a default state of Options +Indexes and a file /var/www/html/fubar/.config containing Options -Indexes, what happens if I have a file /var/www/html/fubar/snafu/.config with Options +Indexes? As you might expect the fubar/snafu/.config overrides the fubar/.config file for the contents of fubar/snafu/.

There are basically two ways to restrict access to web pages: by the client's IP address or by making the client quote a userid and password. For the time being we will control the entire web site. We can then use the previous section to control just subsets of the site. In this section we will restrict by IP address and in the following section we will describe a superior system.

So, first of all, a warning about IP access restrictions: web proxies can really spoil your day. A web proxy is a system that forwards web requests on to another server. So if www.inst.cam.ac.uk restricts access to clients inside cam.ac.uk it is vulnerable to proxies within cam.ac.uk. If randompc.example.com makes a direct request it will be rejected. However, if randompc.example.com makes a request of a web proxy, proxy.college.cam.ac.uk, then the proxy forwards the request to www.inst.cam.ac.uk. The latter sees a query coming from within cam.ac.uk and honours it. The proxy then forwards ther answer back to randompc.example.com. The moral of this tale is to use client address restriction only when you restrict to a set of machines you control (enough to restrict proxies on them). Don't use it blindly.

To give an example of how hard this is the Computing Service discovered it was running an unintended proxy which allowed the CS minutes (restricted to the CS internal network by IP address) to be read from any machine in the world if they knew about our proxy. The CS friendly probing suite now probes for web proxies so you won't be surprised by yours the way we were by ours.

And now, we will demonstrate how to add client address access restrictions in the Apache configuration file using the mod_access module.

# Access control by IP address

LoadModule access_module      modules/mod_access.so
AddModule mod_access.c

order deny,allow
allow from .csi.cam.ac.uk
deny from all
allow from .csx.cam.ac.uk

The order line is read first. The ‘deny,allow’ argument specifies that

regardless of the order the lines appear in.

In the example given, access is permitted to the site from clients in the two domains csi.cam.ac.uk and csx.cam.ac.uk but no others. Note the use of the leading dot to indicate that, for example, .csx.cam.ac.uk is a domain and not a hostname. Also note that for access control by domain name to work you need to have HostnameLookups set to On.

As said before, the author advises very strongly against restricting access to .cam.ac.uk. You may be able to get away with restrictions to inst.cam.ac.uk if you rule your institution with an iron fist. Often it is far more useful is to require authorised users to authenticate themselves against the server. The mechanisms for this are split over a variety of modules and core Apache functionality depending on how to want to run the authentication. We will take the simplest approach here and authenticate against a text password file. Equivalent modules exist for authenticating against more complex databases. This becomes necessary if the database gets too big and linear text file searching too slow.

LoadModule auth_module modules/mod_auth.so
AddModule mod_auth.c

<Directory /var/www/html/restricted>
AuthType Basic
AuthName wombat
AuthUserFile /etc/httpd/conf/passwd
require valid-user
</Directory>

$ touch /etc/httpd/conf/passwd
$ ls -l /etc/httpd/conf/passwd
-rw-rw-r--    1 root     webadmin        0 Jun  1 10:12 passwd
$ htpasswd /etc/httpd/conf/passwd demouser
New password: dem0user
Re-type new password: dem0user
Adding password for user demouser

First let's consider what we've done to the httpd.conf file. We have included a module mod_auth whose function is to permit checking IDs against a plain text password file. This module provides us with the AuthUserFile directive which specifies the location of that password file. The AuthName and AuthType directives belong to the core Apache functionality because they are independent of the supporting database. The AuthType directive specifies the mechanism that is going to be used to transmit the ID and password. If we are going to use the mod_auth module we must specify Basic as the authentication type because this is the only one widely understood. This sends passwords unencrypted over HTTP.

Basic authentication is best illustrated by using telnet as our web client again.

$ telnet hydra.csi.cam.ac.uk 80
Trying 131.111.11.148...
Connected to hydra.csi.cam.ac.uk.
Escape character is '^]'.
GET /restricted/ HTTP/1.0

HTTP/1.1 401 Authorization Required
Date: Thu, 01 Jun 2000 10:29:37 GMT
Server: Apache/1.3.12 (Unix)  (Red Hat/Linux)
WWW-Authenticate: Basic realm="wombat"
Connection: close
Content-Type: text/html; charset=iso-8859-1
 ...
Connection closed by foreign host.

So our attempt to get the /restricted/ URL fails with a status code 401 ‘Authorization Required’. Note the HTTP header line

WWW-Authenticate: Basic realm="wombat"

The browser will then send back the same request as before but this time quoting the ID and password given, Base64 encoded. (The Base64 encoding of ‘demouser:dem0user’ is ‘ZGVtb3VzZXI6ZGVtMHVzZXI=’.)

$ telnet hydra.csi.cam.ac.uk 80
Trying 131.111.11.148...
Connected to hydra.csi.cam.ac.uk.
Escape character is '^]'.
GET /restricted/ HTTP/1.0
Authorization: Basic ZGVtb3VzZXI6ZGVtMHVzZXI=

HTTP/1.1 200 OK
Date: Thu, 01 Jun 2000 11:09:15 GMT
Server: Apache/1.3.12 (Unix)  (Red Hat/Linux)
Last-Modified: Thu, 01 Jun 2000 10:28:10 GMT
ETag: "6b543-144-39363aba"
Accept-Ranges: bytes
Content-Length: 324
Connection: close
Content-Type: text/html
 ...

The browser will typically remember the userid and password for realm ‘wombat’ and if challenged for the same realm again won't reprompt the user.

To date we have just explained how the Basic authentication authenticates a web user. We still haven't really explained why the user is subsequently let in. There are two sides to the permissions: First, the client must authenticate themseves to the server as a particular ID. Second, the ID must, of itself, have permission to access the pages. This second stage is covered with the require directive. The line in our example file

require valid-user

demouser:RGMhGsfmvLQeE
bob:ylxjJ83Fx7p8E
tom:C6QeAIpNqz9IE
dick:yfPWrksACScys
harry:tXFkoaIYJqbrk

The password file maintained by the htpasswd program uses the same password hashing algorithm as the traditional Unix password file, but note that you cannot use the system password file for the Apache system. This file must be maintained separately. Also note that the IDs used in this file are not login names. There need be no relation at all between the IDs used for web authentication and the system's login names.

LoadModule auth_module modules/mod_auth.so
AddModule mod_auth.c

<Directory /var/www/html/restricted>
AuthType Basic
AuthName wombat
AuthUserFile /etc/httpd/conf/passwd
</Directory>

<Directory /var/www/html/restricted/alpha>
require valid-user
</Directory>

<Directory /var/www/html/restricted/beta>
require user tom dick harry
</Directory>

<Directory /var/www/html/restricted/gamma>
require user bob tom
</Directory>

In the slide we see an alternative use of requires. Here, we set up a single mechanism to authenticate the clients for the directory /var/www/html/restricted and three different schemes for determining who (once authenticated) is allowed in.

LoadModule auth_module modules/mod_auth.so
AddModule mod_auth.c

<Directory /var/www/html/restricted>
AuthType Basic
AuthName wombat
AuthUserFile /etc/httpd/conf/passwd
AuthGroupFile /etc/http/conf/group
</Directory>

<Directory /var/www/html/restricted/alpha>
require valid-user
</Directory>

<Directory /var/www/html/restricted/beta>
require group betagrp
</Directory>

<Directory /var/www/html/restricted/gamma>
require group gammagrp
</Directory>

betagrp: tom dick harry
gammagrp: bob tom

There is one level of sophistication above lists of users: lists of groups. In addition to the password file for web IDs to be authenticated there can be a group file assigning these web IDs to web groups. Again, these are completely independent of the Unix login groups and note that the web group file has a different format from the Unix group file.

It's worth recalling that anything that appears in a <Directory> block can also appear in the directory's corresponding .htaccess (or whatever you chose to call it with the AccessFileName directive) file.

GET / HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.14-6.1.1 i686)
Host: hydra.csi.cam.ac.uk
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: es, en
Accept-Charset: iso-8859-1,*,utf-8

A header that is part of the HTTP/1.1 spec. but has been a standard extension to HTTP/1.0 in all browsers is the Host: header. This identifies by name the host the brower was trying to connect to.

At first glance this is pointless. If the browser hadn't been trying to connect to the server it wouldn't have been connecting to this instance of Apache in the first place! However, it is possible to have multiple different names all pointing to the same IP address and hence the same instance of Apache.

There are several ways to do this in the DNS but the most common and easiest is to have a single real name for the IP address (its ‘canonical name’) specified in the DNS by an A record (so called because it looks up the address corresponding to a name) and one or more aliases. These aliases are other names defined to be equivalent ot the real name by CNAME records in the DNS (so called because they look up the canonical name of the alias).

www-uxsup.csx.cam.ac.uk.  1D IN CNAME  nymph.csi.cam.ac.uk.
nymph.csi.cam.ac.uk.      1D IN A      131.111.10.245

By explicit inclusion of the originally requested host name it is possible to have a multiplicity of websites each corresponding to different names for the same host. This is managed in the configuration file with the <VirtualHost> directive.

# Virtual host example
<VirtualHost cockatrice.csi.cam.ac.uk>
DocumentRoot /var/www/cock
</VirtualHost>

The slide shows the setting up of a virtual host with the system definitions but with a different document root. You might want to create separate Unix user groups for the control of the content of the virtual host data and the ‘canonical’ host data.

On systems that run multiple virtual hosts, it is very common for the canonical document root to have nothing but a home page saying ‘go to one of these virtual hosts’ and for all the data to be under the document trees for the various virtual hosts.

The title of this document is: Web Server Management: Running Apache on Red Hat Linux
URL: http://www-uxsup.csx.cam.ac.uk/courses/apache/student.html