sorenpoulsen.com header

Install Apache Tomcat 9 on Ubuntu 16.04 for production

As tempting as it might be to slap on the OpenJDK and Apache Tomcat APT packages from Ubuntu's repositories, production systems often have a different setup.

In this guide we will install Oracle's Java SE instead of OpenJDK and Apache Tomcat we be installed from the original binary distribution instead of the APT packages. The Java process will be locked down with strict permissions and Tomcat's hot deployment disabled. Apache Tomcat will be managed as a Systemd service.

Binaries instead of APT packages. There are two advantages in using the binary distribution from the original distributors rather than the packages from Ubuntu's APT repositories. First we are free to choose newer major versions that are unavailable through the official APT repositories for Ubuntu 16.04 LTS. We are going with Tomcat 9. Secondly the Tomcat APT packages are distributed through the community driven "Universe" repository with no guarantee from Canonical of timely security updates.

As for Java we are going with Oracle Java SE version 10. According to the documentation "Apache Tomcat 9 was designed to run on Java 8 or later", but the most battle tested setup with Tomcat is probably around Java SE 8, 9 and 10. Version 11 changes the game by not having a separate JRE only a JDK and it requires a commercial license for production use.

Tomcat's hotdeployment disabled. Tomcat's hotdeployment feature is not a security vulnerability by itself, but it is often exploited as part of attacks on vulnerable web applications to deploy backdoors into a running Tomcat. We are going to restrict the Tomcat service account from writing to it's hotdeployment folder and disable hotdeployment altogether.

A pattern for strict file permissions. On Linux systems we cannot meaningfully restrict file permissions from a process if the service account running the process owns the file. The owner can simply add permissions back. Group members however cannot change permissions.

If we are to restrict Tomcat from writing to the hotdeployment folder, then we have to set root as owner and only give Tomcat read permission through the folder's group.

A typical pattern for file and folder permissions, that we will use throughout most of the Java and Tomcat installation, is exemplified with the Tomcat hotdeployment folder:

drwxr-x---  3 root tomcat  4096 Jan 21 11:25 webapps/

Here the root account owns the folder - not Tomcat. The folder is made available to Tomcat through the "tomcat" group with read and execute permission but no write permission. All other users have no permissions.

As we shall see later, we will use a Systemd unit file to launch Tomcat with the "tomcat" service account and "tomcat" group.

Install Oracle Java SE 10

Download Oracle Java SE 10 from http://www.oracle.com/technetwork/java/javase/downloads/

Hit the terminal ctrl+alt+t. Unpack the tar and move it to /usr/local:

$ cd Downloads
$ tar xvzf jre-10_linux-x64_bin.tar.gz
$ sudo mv jre-10 /usr/local/

Make root the owner of the files and only provide access to Java through a group called "java". Later we will add the Tomcat service account to this group.

$ sudo addgroup java
$ cd /usr/local
$ sudo chown -R root:java jre-10/
$ sudo chmod -R o-rwx jre-10/

Create a symbolic link "jre" that links to the current installation folder jre-10. If at some point we want to install a new version of Java, then all we have to do is point the "jre" link to the new installation folder and restart Java. The JRE_HOME environment variable, that we will set up later, need not change if it only refers to the "jre" link and not directly to the physical installation folder.

$ sudo ln -s /usr/local/jre-10/ /usr/local/jre

Finally make sure the Oracle Java installation is used over any other Java installation on path, by giving it a higher priority through the Debian Alternatives system. Even if we don't explicitly install OpenJDK it might sneak in as a dependency of other packages.

$ sudo update-alternatives --install /usr/bin/java java /usr/local/jre/bin/java 2000

Install Apache Tomcat 9

Download the tar.gz distribution of Apache Tomcat 9 from https://tomcat.apache.org/

Set a umask that strips write permission for group and other users. This will take affect on all files that we unpack in a moment.

$ umask 022

Unpack Tomcat and move it to /usr/local

$ tar xvzf apache-tomcat-9.0.14.tar.gz
$ sudo mv apache-tomcat-9.0.14 /usr/local/

Create a symbolic link called "tomcat" that links to the physical installation folder apache-tomcat-9.0.14. Just like the Java installation folder we can install new versions of Tomcat alongside the old installation, relink the symbolic tomcat folder to the new installation folder and restart the Tomcat server. The CATALINA_HOME environment variable need not change because it refers to the symbolic link tomcat:

$ sudo ln -s /usr/local/apache-tomcat-9.0.14 /usr/local/tomcat

Create a service account named tomcat and add it to the java group:

$ sudo useradd -r -s /usr/sbin/nologin tomcat
$ sudo adduser tomcat java

System administrator expect all services to log to a folder under /var/log. Lets create a folder /var/log/tomcat and redirect Tomcat logs here by replacing Tomcat's own log folder with a symbolic link to the /var/log/tomcat folder.

We are going to set the group on the log folder to "adm". This is the standard group on Ubuntu for administrators that need to read the log files. In addition we set the SETGID permission on the log folder. This insures that new files "inherit" the group "adm" from the folder and not the group of the user that creates the file.

$ sudo mkdir /var/log/tomcat
$ sudo chown tomcat:adm /var/log/tomcat
$ sudo chmod 750 /var/log/tomcat
$ sudo chmod g+s /var/log/tomcat
$ cd /usr/local
$ rm -rf tomcat/logs
$ sudo ln -s /var/log/tomcat /usr/local/tomcat/logs

Now it's time to implement the pattern for secure file permissions described earlier. We restrict Tomcat from writing to its hotdeployment folder, but need to grant write permission to the temp and work folders. The Tomcat service account will not own the installation files and folders, but is granted some permissions through the tomcat group.

$ sudo chown -R root:tomcat apache-tomcat-9.0.14/
$ sudo chmod g+rx apache-tomcat-9.0.14/conf
$ sudo chmod g+r apache-tomcat-9.0.14/conf/*
$ sudo chmod g+w apache-tomcat-9.0.14/temp apache-tomcat-9.0.14/work

Tomcat as a Systemd service

Create a Systemd unit file for the Tomcat service:

$ sudo vim /etc/systemd/system/tomcat.service

Press i to enter insert mode, then paste this content to the file:

[Unit]
Description=Apache Tomcat
After=network.target
[Service]
Type=simple
Environment=CATALINA_HOME=/usr/local/tomcat
Environment=JRE_HOME=/usr/local/jre
ExecStart=/usr/local/tomcat/bin/catalina.sh run
ExecStop=/usr/local/tomcat/bin/catalina.sh stop
SuccessExitStatus=143
User=tomcat
Group=tomcat
[Install]
WantedBy=multi-user.target

The press ESC to exit insert  mode and type ":wq" to write and quit.

The unit file is set up to launch Tomcat using its standard script catalina.sh. This way all the normal launch features of Tomcat work the way they always do - such as its use of environments variables, the sourced setenv.sh file etc.

It's important to use "catalina.sh run" instead of "catalina.sh start". The latter launched the process in the background and this can cause Systemd loose track of the main process. If that happens then Systemd will erroneously conclude that the main process is done when the catalina.sh script finishes and consequently call the ExecStop script.

Since Tomcat is not launched as a background process by the catalina.sh script itself  we have to use "Type=simple" not "forking"!

Now activate the Tomcat Systemd service:

$ sudo systemctl daemon-reload
$ sudo systemctl enable tomcat

From this point on wards we can start and stop Tomcat using these commands:

$ sudo systemctl start tomcat
$ sudo systemctl stop tomcat

Check it's running on http://localhost:8080/

If everything is OK then it's time to remove the bundled web apps:

$ sudo systemctl stop tomcat
$ sudo rm -rf tomcat/webapps/*

Tuning Tomcat's internal logging

Tomcat's own internal logging needs a bit of tuning for production use. We are going to disable console logging and make sure its Async logger uses a buffer instead of writing log statements to disk one by one.

Open Tomcat's logging configuration for editing.

$ sudo vim tomcat/conf/logging.properties

The root handlers are configured in a line that begins with ".handlers".

.handlers = 1catalina.org.apache.juli.AsyncFileHandler, java.util.logging.ConsoleHandler

Press i to enter insert mode and remove the ", java.util.logging.ConsoleHandler" from the line.

Then add these lines:

1catalina.org.apache.juli.AsyncFileHandler.bufferSize = 8192
2localhost.org.apache.juli.AsyncFileHandler.bufferSize = 8192

Press ESC to exit insert mode and type ":wq" to write and quit.

The catalina.out file should not receive much output after we disabled Tomcat's Console logging. Catalina.out is special because its not handled by a logging framework, its just the stdout being appended to the file using the Linux operator ">>". It does not roll over, pack or delete.

First we set up logrotate to roll the file over daily:

$ sudo vim /etc/logrotate.d/tomcat

Press i to enter insert mode and add this content to the file:

/var/log/tomcat/catalina.out {
  copytruncate
  daily
  rotate 30
  compress
  missingok
  create 640 tomcat adm
}

Next set up a Cron job to compress log files every day and delete the files after 30 days.

$ sudo vim /etc/cron.daily/tomcat

Press i to enter insert mode and add this content to the file:

#!/bin/sh
MAX_AGE_DAYS=30
if [ -d /var/log/tomcat ]; then 
  find /var/log/tomcat -name 'catalina.*.log' -daystart -mtime +0 -print0 | xargs --no-run-if-empty -0 gzip -9
  find /var/log/tomcat -name 'localhost.*.log' -daystart -mtime +0 -print0 | xargs --no-run-if-empty -0 gzip -9
  find /var/log/tomcat -name 'catalina.*.log.gz' -mtime +$MAX_AGE_DAYS -print0 | xargs --no-run-if-empty -0 rm --
  find /var/log/tomcat -name 'localhost.*.log.gz' -mtime +$MAX_AGE_DAYS -print0 | xargs --no-run-if-empty -0 rm --
fi

Press ESC to exit insert mode and type ":wq" to write and quit.

Set the file permissions of the Cron job:

$ sudo chmod 755 /etc/cron.daily/tomcat

Deployment

Disable hotdeployment:

$ sudo vi tomcat/conf/server.xml

Press i to enter insert mode and search for the <Host> tag and change unpackWARS and autoDeploy to "false":

<Host name="localhost" appBase="webapps" unpackWARs="false" autoDeploy="false">

Search for the <Server> tag and change the shutdown command to anything random:

<Server port="8005" shutdown="83n22ordnyodn24sewrg">

With hotdeployment disabled we have to stop Tomcat for deployments, unzip the WAR into the webapps folder, set appropriate permissions and restart tomcat. Let's imagine we are deploying a WAR file called helloworld.war:

$ sudo su -
$ mv helloworld.war /usr/local/tomcat/webapps/ $ cd /usr/local/tomcat/webapps/ $ systemctl stop tomcat $ unzip helloworld.war -d helloworld $ chown -R root:tomcat helloworld/ $ chmod -R 0640 helloworld/ $ systemctl start tomcat

Security announcements

Sign up to the tomcat-announce mailinglist to get security announcements. When a new security update is released simply install it side by side with the old installation folder, then relink /usr/local/tomcat to point to the new installation folder and restart tomcat.

{{model.usr.name}}
{{cmt.user.name}}
{{cmt.user.name}}
{{childcmt.user.name}}
{{childcmt.user.name}}