Generating a hard-coded build number/version name in your Java app

TLDR;

It’s now possible to easily generate a version number for Java apps during the build as a static final field without any runtime dependencies using the annotation processor of the cloudogu/versionName library.


While developing and operating apps with Java for more than a decade, I found one of the recurring problems when working with (or debugging) an app is the matching of the app to the corresponding source code revision/commit/version. So it’s usually a good idea to add a version name, commit id, or build number to your app. I wrote about this topic multiple times and authored a library (now cloudogu/versionName) to automate this task. I thought I’d be through with this topic. Thouroughly. Honestly.

However, while getting started with GraalVM native images, I struck me that the cloudogu/versionName library has shortcomings: It reads the version name from a file (MANIFEST.MF or properties file) within the jar. GraalVM native images are not packaged as jar, though. While we could of course copy the file manually, it made me wonder if a reading a read-only value from a text files isn’t the proper solution. Why don’t we hard-code this into a Java constant? After all, this feels more read-only than a text file and is much easier to read. Also, there’s no more risk of IOExceptions, different behaviours in JDKs, or environments (jar/war). But obviously the writing of the version name field (i.e. generating code) has to be automated.

I sat and thought how this could be realized. The official way of generating code with Java are annotation processors.  From a user perspective an implementation that uses byte code manipulation could be easier to use. OTOH this would require lots of vodoo. If you’re interested in the amount of voodoo, I recommend reading this classic: Project Lombok – Trick Explained.

So, annotations processors it is! Now, I’m happy to announce that the new version 2.1.0 of cloudogu/versionName contains an annotation processor, that generates a Version.NAME constant. This has another neat side effect: at runtime, there is no longer a need to include the library itself. All is done at compile time, zero dependencies at runtime!

All we have to do is add the dependency during compilation (e.g. using Maven’s provided scope) and set the version we want to create as a compiler argument. When using a build tool, it’s easy to generate the version name or build number as described in one the previous articles, and then pass them on to the compiler. For example with Maven:

<dependency>
    <groupId>com.cloudogu.versionName</groupId>
    <artifactId>processor</artifactId>
       <version>2.1.0</version>
   </dependency>
    <!-- This dependency is only needed during compile time and should not be packaged into final JAR -->
    <scope>provided</scope>
</dependency>
 
<!--- .. -->
 
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.8.0</version>
    <configuration>
        <compilerArgs>
            <arg>-AversionName=${versionName}</arg>
        </compilerArgs>
    </configuration>
</plugin>

As it is an annotation processor, we finally have to add an annotation @VersionName to a package or class for the annotation processor to hook in the compilation process. This will cause the processor to create a class Version containing a static final field NAME, during compilation. In the application, this can be read like any other field without the risk of any failures at runtime. Why didn’t I think of this before!
Here’s an example for switiching from the regular versionName library to the annotation processor.

One last side note: The annotation processor’s features are verified using unit tests, which is more complex as it might seem with the compilation process and all. Fortunately, I could reuse the approach of the Shiro Static Permission library (by my colleage @cloudogu, Sebastian Sdorra) that uses Google’s compile-testing library. Unit tests for annotation processors, how awesome is that?!

GraalVM (a bit) beyond Hello World

TLDR;

GraalVM native images are a real innovation for Java apps, providing much smaller Images and faster startup, especially useful for apps that are horizontally autoscaled. However, this causes a higher development effort, especially for existing apps. If included in new apps from the start, this effort should be smaller, but on the other hand we’re pinned to JDK8 with native images.


After working as developer with Java and Docker for many years (and more recently also as trainer) it’s obvious to me that Docker and Java are not a perfect match. As for all interpreted languages, the interpreter (the JVM, in case of Java) imposes a burden in both size and startup time on our Docker image/container.

Java shares this fate with other platforms such as .net, node.js and ruby. To make a point, the follwoing table shows a brief overview of the current “alpine” variants (i.e. the smallest) of those platform’s base images:

Image Size(MB)
 adoptopenjdk/openjdk8:jre8u212-b04-alpine 48
mcr.microsoft.com/dotnet/core/runtime:2.2.4-alpine 36
node:12.0.0-alpine 26
ruby:2.6.3-alpine3.9 26

(The sizes are the compresses size of all layers of the images within DockerHub. The image sizes is calculated as described here).

In contrast, nativly compiled apps written in Go or C/C++ can be built from scratch (if statically compiled), starting of at 0 MB.

GraalVM to the rescue

About a year ago, Oracle announced the first release candiadate of GraalVM offering better interoperability for Java Programms with non-JVM languages and precompiled native images with instant start up and low memory footprint (using ahead-of-time (AOT) compilation). In the announcement they stated that Twitter already uses GraalVM in production to efficiently run their Scala workloads.

From the start, this sounded like a innovation to me. As I’m always a bit low on free time to play around with technologies, I started by following the topic by reading articles that crossed my way, such as

Great introductions, though they all have one thing in common: They compile a single “Hello World” Java file into a native image.

This made me wonder if it’s also that easy for more real-life projects and what pitfalls there are. Especially regarding the known limitations for native image generation, like reflection.

Getting the hands dirty

My first approach was to naively adopt the Dockerfiles for two existing projects to produce native Images. Containing the compilation in Dockerfiles has the advantage of not requiring anything installed locally and is also a kind of “infrastructure as code”, providing deterministic results and allowing for version control.

The projects I chose were

BTW – As they both failed in the beginning (with GraalVM 1.0.0-rc14), I had a look on Java frameworks that offically support GraalVM native images. See my article: Short comparison: Building Graal Native Images with Quarkus, Micronaut and Helidon.

When I started, the latest version was GraalVM 1.0.0-rc14 (the 14th release candidate for version 19.0.0 🤔). For both projects I stumbled upon a number of confusing (to me) errors (like NullPointerExceptions) that all magically vanished once I updated to the “ready for production use” version GraalVM 19.0.0 (as soon as it was available).
Nice work by the GraalVM team 👍  (
if anyone is interested in the errors, I carved them into the Dockerfile or Git Commit messages, see fernflower and colander).

Here are my most interesting findings (I will elaborate on them bellow):

  • the fernflower CLI app works with a Docker image of only 5.3 MB in size! This is really revolutionary for a Java app 🎉
    Note that it only compiles statically (which is what I wanted anyway, to be able to use a scratch Docker image)
  • the colander native image also compiles (it’s also only 5 MB). However, the app does not work, because of it’s dependencies. These were the kind of real-world problem findinds I was looking for.

Findings in detail

Happy days

As said, the GraalVM native image build for fernflower works like charm. In fact, I was so charmed I created an automated build at DockerHub, that regularly builds Docker images for the latest version of fernflower, using different base images: schnatterer/fernflower-docker.

So if you’d ever need a Java decompiler just do a docker run --rm -v $(pwd):/src schnatterer/fernflower and 5MB later you’ll have you’re decompiler ready.
These images also allow for a nice comparison of image sizes for different Java base images.

Image Size(MB)
native/scratch image
adoptopenjdk/openjdk8:jre
adoptopenjdk/openjdk8:alpine-jre
gcr.io/distroless/java:8

5 vs 45MB comparing native image to regular JRE. Stunning, isn’t it?

BTW – this is the final Dockerfile to build the native image.

Facing real world challenges

Of course, I love it when a plan comes together. But on the other hand I suspected this wouldn’t always be the case for a such complex a thing as GraalVM native images. Here are the issues I encountered for the colander app (as said before, these issues are well documented known limitations of GraalVM):

  • When starting the native image, there’s not much output to the console.
    Reason: Not surprisingly, unlike the jar, the native image does not contain a logback.xml. In order to fix this, we would have to copy the logback.xml manually into the final docker image during the build.
  • The command line option --help does not show any options.
    Reason: These options are defined in annotations, and are read at runtime using reflection via the JCommander framework. How to fix?
    Configure the reflection for native image generation. Fortunately, some Java CLI frameworks like picocli support generating the config files out of the box. So migrating would also be an option.
  • Colander can’t show its own version name.
    Reason: The version name is read from the MANIFEST.mf file using the cloudogu/versionName library. The file is (again, not surprisingly) is not contained in the native image. This made me wonder if it wouldn’t be much simpler to read the version name from a Java constant. A hard-coded value just feels more read-only than a text file such as the MANIFEST.mf. I added this feature to the library using annotation processors.

Conclusion

The issues encountered definitely proof my hypothesis, that GraalVM native images, while a technological innovation that provides a lot of potential, causes extra efforts during development. So we have to decide if these extra efforts are justified by the advantages.

I presume that, if planned from the beginning, the efforts are a lot less, because we can take GraalVM support into account when making our technical decisions. There are frameworks like Quarkus, Micronaut and Helidon that have native support for GraalVM, minimizing the extra effort. For large existing apps, though this does not help. The colander example is really small (with only about 1k of net LOC) and already causes a lot of effort.

So would I use GraalVM native images in production?

I probably wouldn’t migrate existing applications, except they run at a massive scale and the potential savings (faster horizontal scaling, smaller memory footprint) justify the effort for building the native image in the first place. For new projects, I would assess building GraalVM native images from the start, if sticking with JDK8 is OK.  Native images only support JDK8, as of GraalVM 19.0.0.
In the long run most popular Java frameworks, not only Quarkus, Micronaut and Helidon are likely to support native image generation. For now, Spring is still a WIP, also mentioned in the GraalVM 19.0 release announcement. If I had a teams with profound Spring experience, I would only switch to another framework for a good reason.
On the other hand, if developing using the microservices architecture pattern, first experiences with GraalVM native images could be gained by implementing small services using GraalVM.

Short comparison: Building Graal Native Images with Quarkus, Micronaut and Helidon

The technological innovations of the last years such as the adoption of containers, cloud-native technologies, the microservice architectural style, the inception of GraalVM and the end of JavaEE (as we know it) has energized the Java framework market.

As of May 2019, there are at least three frameworks supporting GraalVM native images out of the box, targeting cloud-native microservices:

  • Quarkus,
  • Micronaut and
  • Helidon.

As building GraalVM native images is a bit challenging, I was curious to find out how these three frameworks keep up with their promises. I worked through the respective getting started guides and wrote down some similarities and differences resulting in this short (and surely incomplete) comparison of the three frameworks. See the following table for an overview.

General comparison

Quarkus Micronaut Helidon
Core Project Source quarkusio/quarkus micronaut-projects/micronaut-core oracle/helidon
Website quarkus.io micronaut.io helidon.io
Started/Backed By RedHat objectcomputing Oracle
First Commit 2018-06-22 2017-03-06 2018-08-28
GitHub Stars (05/2019) 1693 2283 1371
GitHub Contributers (05/2019) 88 120 26
# Commits (05/2019) 3970 5907 617
Supported languages Java, Kotlin Java, Groovy, Kotlin Java
Supported build tools mvn, Gradle mvn, Gradle mvn
Supported APIs for graal native Microprofile, vert.x,

 

Micronaut, ReactiveX/RxJava Helidon SE (Microprofile, without native image)
Programming paradigms for graal native Imperative, reactive Reactive, imperative? Reactive (imperative, without native image)
Code generation via mvn plugin CLI (mn) mvn archetype
Getting Started (Graal native) Guide Docs Blog
Resulting src of Getting Started schnatterer/quarkus-getting-started schnatterer/micronaut-getting-started schnatterer/helidon-getting-started
Size of getting started docker image
Getting started base docker image fedora-minimal alpine-glibc scratch
Getting started uses native image? N N Y
Size of getting started in scratch docker image

All three projects are rather young (grandpa Micronaut is about 2 years old as of 05/2019) but have what looks like extensive documentation at first glance. The only thing that made my stumble a bit was that Helidon’s docs don’t return a result for “graal”. I later found a brand new getting started with graal on oracle’s developers blog. Hopefully, this will be added to the docs soon.

There are a couple of notable differences between the three frameworks:

    • Programming style (reactive vs. imperative)
      • Quarkus explicitly supports both (reactive as an extension),
      • Helidon claims to support both, but only reactive in conjunction with native images right now
      • Micronaut is reactive only From the docs it seems that micronaut focuses on reactive, but blocking approaches are supported (see Graeme Rocher’s comment).
    • Language
      • Micronaut and Quarkus both support Java and Kotlin.
        Micronaut also supports Groovy 🎉 (having Graeme Rocher, the creator of Grails, on board it’s probably a must)
      • Helidon only supports Java
    • Build tool / code generation
      • Micronaut and Quarkus support Maven and Gradle.
        • Quarkus uses a Maven plugin for code generation (bad luck for Gradle users) whereas
        • Micronaut brings its own CLI tool that thankfully can easily be installed using sdkman.
      • Helidon supports only Maven and has only initial code generation support via a Maven archetype.
    • Kubernetes
    • Community
      Hard to tell. The amount of discussions on my tweet about Quarkus makes me think they’re the ones that are most interested in feedback and people getting involved.

GraalVM native Image / Docker Image

  • The Dockerfiles provided by the getting started of Quarkus and Micronaut each require an external Maven build.
    The images base on fedora-minimal (resulting in a 44MB compressed image) or alpine-glibc (resulting in a 32MB compressed image) respectively.
    A base image containing a libc is required because the native image is linked dynamically.
  • Helidon provides a proper self-contained Dockerfile that can be built by simply calling docker build, not requiring anything locally (except Docker, of course).
    Here, the native image is linked statically. Therefore the binary can run in an empty scratch image (resulting in an 8MB compressed image).

Bearing in mind that a Java 8 JRE Image requires about 100MB (debian) or 50 MB (alpine), 44MB or even 32MB for a small webapp is not so bad. OTOH the 8 MB for the statically linked image are a real revelation, leaving me stunned.

The fact that Helidon plays well with GraalVM shouldn’t be too surprising, as they both are official Oracle products.

Beyond getting started

As Quarkus was the first framework I tried, I wondered why they rely on fedora and not just compile a static binary (later, I learned about some of their reasons on twitter). So I tried a couple of other images, eventually setting the switch for creating a static binary and using a scratch image. Voilà: It results in a 7MB image, even a wee bit smaller than the Helidon one. See the table bellow for an overview of images and their features and sizes (taken from the README of my getting started repo).

Base Image Size Shell Package Manager libc Basic Linux Folders Static Binary Dockerfile
fedora 📄
debian 📄
alpine-glibc 📄
distroless-base 📄
busybox 📄
distroless-static 📄
scratch 📄

I applied more or less the same on Micronaut. Here, the scratch image is only 5 MB smaller than the alpine one – 27 MB. This is not too surprising, because the plain alpine-glibc image is only about 6MB. It also felt like the native image generation took longer and needed more memory (observed with docker stats).

As for Helidon’s self-contained, scratch image containing only a static binary, there was not much to be done. I only extend the Dockerfile by a maven cache stage for faster Docker builds.

There’s one last thing I changed in all Dockerfiles: Don’t run as root. I used the USER statement in the Dockerfile. docker run -u ... would also be fine. This way, it’s much more unlikely that possible vulnerabilities (such as CVE-2019-5736 in runc) are exploited.

So summing up: Quarkus and Helidon can be used to create really small docker images, Micronauts are “only” small 😉. It’s worth mentioning that I didn’t look what features are included in those images, so maybe it’s a bit naive to just compare the minimal sizes resulting from the individual getting started guides.

Going even further

If I were to continue my comparison at this point (which I won’t because it’s only a short comparison) I would look into the following features of each framework:

  • integration and unit testing,
  • extensions (e.g. Cloud Native features, Tracing, Monitoring, etc.)

Summary

So, for a new green field project, which one of e frameworks would I use?

As far as I can tell after completing the getting started, all three look promising. As for all architectural decisions, I’d definitely try to build a walking skeleton (technical roundtrip) before finally deciding, in order to gain more field experience and find out what’s beyond getting started.

I’d base this decision on the experience or preferences of the team

  • reactive vs. imperative
  • Maven vs. Gradle
  • Java vs. Kotlin (or even groovy)
  • APIs – Microprofile, vert.x, RxJava

Personally, I like the fact that Quarkus builds on existing APIs such as Microprofile, so existing experience can be reused for faster results. It also seems to me the most flexible of the three, supporting Java, Kotlin, Maven, Gradle, reactive and imperative.

As for native images, I’d definitely either try it from the beginning or stick to a regular JRE. I suppose switching from plain JRE-based to native could be complicated for an existing app, due to the native image limitations. If the app under development does not have the requirement to be scaled horizontally, this could be an argument for skipping the native image part. But this is beyond the scope of this article.

As for the docker image – it’s obviously not only the size that matters. An image without shell and package manager is always more secure but harder to debug.

 

Edits:

  • 2019/05/17: John Clingan pointed out that Quarkus supports Kubernetes resource generation an multiple reactive extensions
  • 2019/05/19: As commentef by Graeme Rocher’s, Micronaut also supports blocking workloads

Android Logging for Java Professionals – SLF4J and Logback in Android

One of my articles was published in Java Magazin 9.17. I wrote it while working on the nusic android app, about how to use SLF4J in Android using logback-android. It also features an example and a small library for android.

Triology GmbH provides an English version of this article, and also acquired the original article PDF (in German), which can be found here: Android Logging für Java-Profis – SLF4J und Logback in Android. I’d like to thank my colleagues there for their support.

Maven: Create a more sophisticated build number

Earlier this year, while working on a project for TRIOLOGY GmbH, I once again used maven to write a version name into an application, using the mechanism described in my post Maven: Create a simple build number. As a more sophisticated version name was required for this project, we expanded it by a time stamp, SCM information (branch and commit), build number and a also created a special name for releases. You can find a how-to here – Version names with Maven: Creating the version name – which is the first part of a small series of blog posts on this topic.

The second part shows how the version name can be read from within the application. While writing the examples for the post, I wondered how many times I must have implemeted reading a version name from a file in Java. Way too often! So I decided that this would be the very last time I had to do it, and extracted the logic into a small library: versionName, availble on GitHub. What it does and how to use it is described in the second part of the post: Version names with Maven: Reading the version name.

Hopefully, this will be useful for someone else. Funny enough, in the new project I’m on, I’m about to reuse it once again. I’m glad I don’t have to write it again. Here’s to reusability 🍺

Android apps – nusic: Find New Music Albums

Have you ever stumbled upon a new album of one of your favorite artist and recognized they released several albums since you last checked?

nusic - your new music

nusic – your new music

Problems like this can now be solved conveniently with nusic for Android.

All you need is an Android device that contains music of all the artists you like.

After installing, nusic regularly checks if there are upcoming releases for the artists on your device and informs you about any news.

Just install the app, start it once and it will keep you up to date about new releases via android notifications. Never again miss any releases of your favorite artists!
 
You can install nusic from Google Play or get the APK from GitHub.

Get it on Google Play
If you should encounter any errors, please report them here instead of just giving a poor rating.

By the way, nusic is open source. Please contribute by forking nusic on GitHub.

 

How does it work?

nusic regularly checks MusicBrainz – the open music encyclopedia – for new releases of the artists on your device.

That’s all there is. You don’t need an account and the app is not pulling any other data from your phone!

Automatically downloading/backing up/dumping/exporting databases from remote hosts via the web

The problem

You operate a database-backed website (e.g. Drupal) where you can’t access cron jobs, cgi, perl and outgoing connections. So one idea to back up your database on a regular basis (which is always a good idea) is to download SQL dumps via a web-based administration tool (such as the backup and migrate plugin for drupal). Unfortunately, these kinds of downloads cannot simply be automated on the shell by using curl or wget, because they require a bit of javascript, for example to outsmart the php timeout.

The solution

Use a headless browser (that is, a browser without graphical user interface) to automate the task. It fetches the desired page, logs in, (virtually) clicks the download button and downloads the dump file.

It should be a command line tool, in order to run it as cron job from a some server (e.g. a NAS).

Personally, I liked the idea of PhantomJS, but it was not available for my Synology DS213+ PowerPC and I didn’t like the idea of building it from source.

So my plan B was to write a small Java program (remoteDbDumper)  that uses the HtmlUnit framework (our headless browser).

How to use

  1. Install drupal plugin backup and migrate.
  2. Download and extract remoteDbDumper.
  3. Start it from the shell.
    remoteDbDumper -u <username> -p <password> -o <output dir> <url to backup and migrate>

    Note that output dir must be an existing directory

    1. Linux example:
      ./remoteDbDumper.sh -u user -p topsecret -o ~/backup https://ho.st/drupal/?q=admin/config/system/backup_migrate
      
    2. Windows example
      remoteDbDumper.bat -u user -p topsecret -o "%HOMEPATH%\backup" https://ho.st/drupal/?q=admin/config/system/backup_migrate
      
  4. Use the scheduling mechanism of your choice to call remoteDbDumper regularly, creating backups.

Example (Synology)

Just a short exemplary scenario on how to use remoteDbDumper on a Synology Diskstation (running DSM 4.2) to regularly back up a drupal database.

  1. (if Java is not installed) install Java:
    If available for your Diskstation, use the Java Manager package. Otherwise, you could use a third party Java package (that’s what I had to do).
  2. Download, extract and copy remoteDbDumper to the NAS (e.g. to \\diskstation\public\, which corresponds to /volume1/public/)
  3. SSH to the NAS and check if it works
    /volume1/public/remoteDbDumper-1.0/remoteDbDumper.sh -u user -p topsecret -o /volume1/someUser/ https://ho.st/drupal/?q=admin/config/system/backup_migrate
    
  4. (optional) Wrap the command line call in a shell script, e.g.
    BASEDIR=$(dirname $0)
    $BASEDIR/remoteDbDumper-1.0/remoteDbDumper.sh -u user -p topsecret -o $1 https://ho.st/drupal/?q=admin/config/system/backup_migrate
    
  5. Either use the web frontend  or the crontab to schedule the back up.
    1. Web frontend:
      Go to http://diskstation:5000, (or whatever combination of host name and port you’re using)
      login as admin,
      click on Control Panel | Task Scheduler.
      Then click on Create | User-defined Script.
      Enter a task name, choose a user (preferably not root), set up a schedule (e.g. every sunday at 8 p.m.).
      Finally enter the path to remoteDbDumpe or the script (4.) respectively. For the example above, the path would look like this:

      /volume1/public/dumpDb.sh /volume1/public/
      
    2. If you insist to do it on foot, here’s what to enter in crontab:
      vi /etc/crontab
      
      #minute hour    mday    month   wday    who              command
      0       20      *       *       0       enterUserHere    /volume1/public/dumpDb.sh /volume1/public/
      
    3. Set a maker in your calender for the next scheduled run, to check if it worked.

Future tasks

At the current state remoteDbDumper can only backup drupal databases. Fair enough.

However, with just a little more effort it would be possible to extend remoteDbDumper to support addition web-based database administration tools, such as  mysqldumper, phpMyBackupPro, phpMyAdmin or phpPhAdmin.

In order to do so, just fork the repo on github and implement the interface DbDump.