TLDR;
I implemented a solution for fetching and renewing TLS certs without restart via Let’s Encrypt that works with standalone and embedded Tomcat as well as Spring Boot. It’s packaged into a Docker image, allowing for easy reuse. The certs are fetched and renewed using dehydrated, a bash script with few dependencies. This makes the TLS requesting agnostic to the application or framework in use, as long as the application is shipped as a Docker image. The downside is, that it’s not a plain Java solution, i.e. it requires an additional process (within the container) and a Linux environment.
Having used cloud-native products like Caddy, Traefik, CertManager or Micronaut that naturally obtain and renew Let’s Encrypt certificates for HTTPS communication, I wondered if there is an easy way to use a feature such as this in a web server as long-established as Apache Tomcat. You might say: Tomcat usually is used behind reverse proxies that do all the TLS offloading. But then there’s something like the Apache Portable Runtime (APR) based Native library for Tomcat that brings the heart of the Apache Web Server to Tomcat as a system library and with it it’s high TLS performance, that might even outperform NGINX. So obviously, this should be possible.
Surprisingly, I couldn’t find a solution with batteries included. So, let’s quickly implement our own. How hard can it be?
Dehydrated tomcat
Presenting: letsencrypt-tomcat – it packs everything needed for automatic TLS with tomcat into a single docker image:
- dehydrated to manage certs via Let’s Encrypt,
- tomcat-reloading-connector for hot reloading certs at runtime after renewal,
- an init system (dumb-init) for properly handling tomcat and dehydrated processes,
- an entrypoint script that starts up tomcat and dehydrated as well as
- a pre-compiled version of Apache Portable Runtime (APR) and JNI wrappers for APR used by Tomcat (libtcnative), so tomcat delivers the best TLS performance possible.
This sounds like a lot of complicated stuff, but as the examples bellow will show, it’s rather easy to use. If you care about the details, check the “Challenges” bellow.
From the image everything can be copied as needed in the actual application’s Dockerfile. It’s not intended to as a base image (“inheritance”), but as a mere file archive to copy everything from a single source.
Usage
The way to use depends on the tomcat “kind” in use: standalone, embedded Tomcat orSpring Boot (which also uses embedded tomcat under the hood).
There’s one thing the setup for all kinds have in common: you pass the FQDN as environment variable DOMAIN to the container at startup, e.g. with docker run -e DOMAIN=example.com <image>
If necessary more parameters (optional) are available.
The remaining setup depends on the kind of tomcat used.
Sprint Boot
pom.xml or build.gradle: Add tomcat-reloading-connector to your dependencies (example)Dockefile:
- Copy
/letsencrypt and /lib folder to the root (/) of your application’s docker image (example),
- Set
ENTRYPOINT or CMD to execute /meta-entrypoint.sh and pass the usual command for starting your spring boot app as parameter, for example:
ENTRYPOINT [ "/meta-entrypoint.sh", "java", "-jar", "/app/app.jar" ]
application.properties or .yaml (example):
- Configure Tomcat to serve Let’s Encrypt challenges:
spring.resources.staticLocations=file:/static
- Configure paths for certificate files:
server.ssl.certificateKeyFile, etc.
- Serve both HTTP and HTTPS ports
- In the code that creates your Spring Boot app: Setup
tomcat-reloading-connector: @Import(ReloadingTomcatServletWebServerFactory.class) (complete example)
Embedded tomcat
pom.xml or build.gradle: Add tomcat-reloading-connector to your dependencies (example)Dockefile:
- Copy
/letsencrypt and /lib folder to the root (/) of your application’s docker image (example),
- Set
ENTRYPOINT or CMD to execute /meta-entrypoint.sh and pass the usual command for starting your app as parameter, for example:
ENTRYPOINT [ "/meta-entrypoint.sh", "java", "-jar", "-Dcatalina.home=/tmp", "/app/app.jar" ]
- In the code that creates embedded tomcat:
- Configure Tomcat to serve Let’s Encrypt challenges (example)
- Use
tomcat-reloading-connector and configure paths for certificate files: ReloadingTomcatConnectorFactory.addHttpsConnector(tomcat, HTTPS_PORT, PK, CRT, CA); (complete example)
Standalone tomcat
Dockefile:
- copy the following to the root (
/) of your application’s docker image (example):
/letsencryptfolder,tomcat-reloading-connector and- (if necessary)
/lib folder (already included in bitnami-tomcat used in our example) ,
- Make standalone tomcat serve Let’s Encrypt challenges (example)
server.xml:
- Use
tomcat-reloading-connector (example)
- Configure paths for certificate files (example)
Real-world example
My use case for using this whole thing is my git-based wiki smeagol-galore, which relies on letsencrypt-tomcat. It already had a quite complex startup process, so I decided here to not use the meta-entrypoint but to copy the parts needed into the existing entrypoint.
Challenges
Now that we have seen working examples, it’s a good time to sum up technical challenges and decisions encountered within the process. Turns out the whole indeavor had more challenges than expected.
Choosing an ACME implementation
As of 09/2020 there doesn’t seem to be a drop-in web app for tomcat or a spring boot annotation that just handles the ACME protocol used by Let’s encrypt for us.
There are Java libraries that promise to help achieving this, most prominently acme4j. But looking at the docs or this tutorial the whole process seems rather complicated. There’s the spring-boot-starter-acme which seems be able to get a cert. To use the cert, we have to restart the server, though. It also does not renew, is still a SNAPSHOT and hasn’t been updates for some years.
A completely different alternative would be to use the certbot client. But then it brings it’s own server, which would require our application to stop and start again, once certbot has finished renewing the cert.
This is where dehydrated comes in handy: it implements the whole ACME protocol used by Let’s encrypt, in a single bash script with no other dependencies than bash, curl and OpenSSL. It’s also easy to use and does not bring it’s own webserver. That’s good, because we already have a webserver: Tomcat.
How to best integrate a bash script and a tomcat process into a single artifact? Why not use an OCI (a.k.a. Docker) image? Two processes in one container? But doesn’t this violate the “one process per container” mantra?
I’m convinced that if there are good reasons for it, why not use two coherent processes in one container? We might even start chaning the mantra to “one thing per container“. The fact that there a multiple popular init systems for Docker images makes me think that I’m not the only one holding this opinion.
So let’s combine tomcat with dehydrated within a Docker container.
Integrating Tomcat and Dehydrated
The basic idea is:
- Tomcat starts,
- Dehydrated starts the certificate signing process via Let’s Encrypt,
- serves the challenge via Tomcat, then
- downloads the certificate, so that finally
- Tomcat can server the certificate.
- Once a day, Dehydrates checks again if the certs must be renewed.
This results in two “interfaces” between Tomcat and Dehydrated:
- the folder where Dehydrated places the challenges must be served by Tomcat at
http://$DOMAIN/.well-known/acme-challenge/and
- the cert files, written by Dehydrated and read by Tomcat.
Depending on the kind of Tomcat, configuring the cert files and making Tomcat serve the the challenge folder on the HTTP port requires a different setup (see “Usage” above).
Self-signed cert
Once this is done, let’s start Tomcat. Only it does not start without certificate. But we need it started, in order to answer Let’s Encrypt’s challenge.
Who was first, the chicken or the egg?
One solution: create a self-signed cert at startup (see meta-entrypoint.sh). With a validity of 30 days or less, dehydrated will try to renew the cert right away.
Synchronize startup of Tomcat and Dehydrated
Tomcat might get its HTTP ports ready to serve traffic in a matter of seconds but, depending on the application, it could als take minutes. The latter resulting in Dehydrated failing to get the the certificate from Let’s Encrypt. Solution: Poll the HTTP port, waiting for a succesfull HTTP status code.
PEM vs KeyStore
With Tomcat’s default Java implementation of HTTPS (Http11NioProtocol) we would also have issues with the certificate file formats, because it requires a Java Keystore, whereas Let’s Encrypt provides private key, certificate and CA as PEM. However, we are using APR (Http11AprProtocol), which also requires PEM. So it just works™️.
Reloading the certificate
Believe it or now: At this point we have a valid certificate. Only that Tomcat still serves the old self-signed one.
Next stop, how to make Tomcat reload the cert? Tomcat 9.x (might even have been backported to 8.5) provides an AbstractHttp11Protocol.reloadSslHostConfigs() method. But how to call it?
Dehydrated offers a hook for calling “something” after success. But how to call this method inside Tomcat from a shell script? JMX? Or via an URL request on Tomcat Manager web app? These solution would all require configuration, increase complexity and might even impair security.
The other option is to call the method from within Tomcat, by implementing our own protocol class. It could regularly check and reload the certificate on change. But then polling is just a waste of resources and it will always take longer than something that is called on change.
Other idea: Set a file watch. For better reuse, maintainance and separation of concerns, I implemented this in a separate project, tomcat-reloading-connector (too late I figured out that “tomcat-reloading-protocol” would be a better fit 🙈 ) and published it on maven central. While implementing, I came to the realization that reloading “the certificate” should be an atomic operation, containing all certificate-related files. After all, what I refer to as “the certificate” above, actually consists of a number of files, like private key, cert and CA. Reloading only one file might result in an inconsistent state. In order to avoid this, I added a reload delay of a few seconds. A changing file within the folder of the certificate results in all SSLHostConfigs being reloaded a couple of seconds later. Still not perfect, as the delay is a compromise between a fast reload and an inconsistent SSLHostConfig. For starters, let’s stick with it. For Dehydrated 3s seemed fine from my empirical studies. If it does not work for you, just define what ever suits your needs via the TOMCAT_DELAY_RELOAD_CERTIFICATES_MILLIS environment variable.
Conclusion
Creating a dehydrated Tomcat wasn’t straightforward. The most laborious task being the reload of the certificate, resulting in a separate project: tomcat-reloading-connector.
With letsencrypt-tomcat most of the work has been done, allowing for rather easy reuse. However, for each project, there still is some bootstrapping to be done. A 3rd party solution such as this can never compete with built-in support for Let’s Encrypt as offered by other products. On the other hand there now is a reliable solution that can run on all kinds of Tomcats be it standalone or embedded, even with Spring Boot.
Future work
That’s all for now. As always, there some work is still to be done for the curios reader.
Getting letsencrypt-tomcat to work wit Let’s Encrypt’s TLS-ALPN-01 challenge would be useful to get rid of the HTTP port. It generally is supported by Dehydrated but might be challengin to realize using Tomcat.
One issue with relying on a custom Dockerfile for creating the artifact is that it does not work intuitively with Cloud Native Build Packs. Spring Boot 2.3 comes with built-in support for building OCI images without Dockerfile. How can we integrate this with letsencrypt-tomcat? It should be possible to just add another layer, and adapt the entrypoint.
IT affinity! 