Coding Continuous Delivery with Jenkins Pipelines

Starting in their 01/2018 issue, Java aktuell published my four-part articles series Coding Continuous Delivery in German. I’m happy to announce that all parts are now available in English, courtesy of Cloudogu.

The series takes you from zero to continuously delivering your software through a sophisticated Jenkins pipeline. It starts with the fundamentals, heading on to advanced topics such as nightly builds, parallel execution, docker, shared libraries, unit testing, static code analysis with SonarQube and deployment to Kubernetes. All of the topics are described hands-on with examples comparing the scripted with the declarative syntax provided by the Jenkins Pipeline Plugin.

  1. Jenkins pipeline plugin basics | 🖺 original article PDF (German)
  2. Performance optimization for the Jenkins Pipeline | 🖺 original article PDF (German)
  3. Helpful Tools for the Jenkins Pipeline | 🖺 original article PDF (German)
  4. Static Code Analysis with SonarQube and Deployment on Kubernetes et al. with the Jenkins Pipeline Plugin | 🖺 original article PDF (German)

The examples to all articles are contained in this GitHub repository: triologygmbh/jenkinsfile and the builds can be seen in action on this Jenkins server: opensource.triology.de.

My awesome colleagues at Cloudogu GmbH and Triology GmbH – thank you so much for your support. Especially my co-author from the first article, Daniel Behrwind, who got this whole thing started.

Building GitHub projects with Jenkins, Maven and SonarQube 5.2 on OpenShift

Time for an update of the post Building GitHub projects with Jenkins, Maven and SonarQube 4.1.1 on OpenShift, because SonarQube 5.2 is out: It’s the first version since 4.1.1 that can be run on OpenShift. That is, it’s the first version of SonarQube 5 and the first one that contains Elasticsearch and many other features that are now available on OpenShift!
Interested? Then let’s see how to set up SonarQube on OpenShift.

  • If you’re starting from scratch, just skip to this section.
  • If you got a running instance of SonarQube
    • make sure to back up you instance before you continue:
      rhc snapshot save -a --filepath
      or
      ssh @-.rhcloud.com 'snapshot' > sonar.tar.gz
    • Then pull the git repository just like in step 2,
    • wait until the app has started and visit
      https://sonar-.rhcloud.com/setup

      SonarQube will update it’s database during the process.

    • If you followed this post to set up your SonarQube instance and therefore use an SSH tunnel to access the SonarQube database, note that you can now get rid of this workaround. From SonarQube 5.2 the analyses can be run without direct contact to the database.
      That is, you can also remove the database connection from your the configuration of the SonarQube plugin in jenkins.

Install new SonarQube instance

To install SonarQube 5.2, execute the following steps on your machine:

  1. rhc app create sonar diy-0.1 postgresql-9.2

    Make sure to remember the login and passwords!

  2. git rm -r diy .openshift misc README.md
    git remote add upstream -m master https://github.com/schnatterer/openshift-sonarqube.git
    git pull -s recursive -X theirs upstream master
    git push
    
  3. Login to your SonarQube instance at
    http://sonar-.rhcloud.com/

    Note that the initial setup may take some minutes. So be patient.
    The default login and passwords are admin / admin.
    You might want to change the password right away!

Basic installation Jenkins

Basically, the following is an updated (and a lot simpler) version of my post about SonarQube 4.1.1.

  1. Create Jenkins app
    rhc app create jenkins jenkins-1
  2. Install Plugins
    Browse to Update Center

    https://jenkins-.rhcloud.com/pluginManager/advanced

    and hit Check Now (as described here).
    Then go to the Available tab and install

    1. Sonar Plugin,
    2. GitHub plugin,
    3. embeddable-build-status (if you’d like to include those nifty build badges in you README.md).

    Then hit Install without restart or Download and install after restart. If necessary, you can restart your app anytime like so

    rhc app restart -a jenkins
  3. Set up maven settings.xml to a writable location.
    • SSH to Jenkins
      mkdir $OPENSHIFT_DATA_DIR/.m2
      echo -e "<settings><localRepository>$OPENSHIFT_DATA_DIR/.m2</localRepository></settings>" > $OPENSHIFT_DATA_DIR/.m2/settings.xml
      
    • Browse to Configure System
      https://jenkins-.rhcloud.com/configure

      Default settings provider: Settings file in file system
      File path=$OPENSHIFT_DATA_DIR/.m2/settings.xml

  4. Either see my post on how to introduce a dedicated slave node to this setup or
    set up the Jenkins master to run its own builds as follows (not recommended on small gears, as you might run out of memory pretty fast during builds):
    Go to Configure System

    https://jenkins-.rhcloud.com/configure

    and set
    # of executors: 1

  5. Setup sonar plugin (the following bases on SonarQube Plugin 2.3 for Jenkins)
    On the Jenkins frontend, go to Configure System

    https://jenkins-.rhcloud.com/configure
    • Global properties,
      tick Environment variables
      Click Add
      name=SONAR_USER_HOME
      value=$OPENSHIFT_DATA_DIR
      See here for more information.
    • Setup the Runner:
      Navigate to SonarQube Runner
      Click Add SonarQube Runner
      Name=
    • Then set up the plugin itself
      Navigate to SonarQube
      tick Enable injection of SonarQube server configuration as build environment variables
      and set the following
      Name=
      Server URL:

      http://sonar-.rhcloud.com/

      Sonar account login: admin
      Sonar account password: (default: admin)

    • Hit Save

Configure build for a repository

Now lets set up our first build.

  1. Go to
    https://jenkins-.rhcloud.com/view/All/newJob

    Item name:
    (Unfortunately, Maven projects do not work due to OpenShift’s restrictions.)
    Hit OK

  2. On the next Screen
    GitHub project:

    https://github.com///

    Source Code Management:

    https://github.com//.git

    Branch Specifier (blank for 'any'): origin/master
    Build Triggers: Tick Build when a change is pushed to GitHub
    Build Environment: Tick Prepare SonarQube Scanner environment
    Build | Execute Shell

    cd $WORKSPACE
    # Start the actual build
    mvn clean package  $SONAR_MAVEN_GOAL --settings $OPENSHIFT_DATA_DIR/.m2/settings.xml -Dsonar.host.url=$SONAR_HOST_URL
    
  3. I’d also recommend the following actions
    Post-build Actions| Add post-build action| Publish JUnit test result report
    Test report XMLs=target/surefire-reports/TEST-.xml*
    Post-build Actions| Add post-build action| E-mail Notification
    Recipients=
  4. Hit Apply.
  5. Finally, press Save and start you first build. Check Jenkins console output for errors. If everything succeeds you should see the result of the project’s analysis on SonarQube’s dashboard.

Using Custom Maven / JDK version when building with Jenkins on OpenShift

[EDIT (2016-06-02): OpenShift now provides different JDK “alternatives”, e.g.

/etc/alternatives/java_sdk_1.8.0

So you might want to skip the steps bellow, regarding a custom JDK. The steps described for using a custom maven still apply, however.

]

In previous posts I pointed out how to build GitHub projects with Jenkins, Maven and SonarQube and how to run these builds on dedicated Jenkins slaves. The following shows how to replace the “stock” versions of maven and JDK that are provided by OpenShift.

At the time of writing OpenShift features Maven 3.0.4 and OpenJDK Server 1.7.0_85. Why would you want to change those? Best example is a Java8 project to be build on Jenkins. Can we just advise Jenkins to download the newest Oracle JDK and we’re good to go? Nope, it’s not that simple on OpenShift! Jenkins does download the new JDK, sets the JAVA_HOME variable and the correct PATH, but maven is always going to use the stock JDK. Why? Running this command provides the answer

$ cat `which mvn`
#!/bin/sh
prog=$(basename $0)
export JAVA_HOME=/usr/lib/jvm/java
export JAVACMD=$JAVA_HOME/bin/java
export M2_HOME=/usr/share/java/apache-maven-3.0.4
exec $M2_HOME/bin/$prog "$@"

The stock maven is setting its own environment variables that cannot be overridden by Jenkins!

So, in order to exchange the JDK, we need to exchange maven first.

  • SSH to the machines where your builds are executed (e.g. your slave node). The following example show what to do for maven 3.3.3:
    cd $OPENSHIFT_DATA_DIR
    mkdir maven
    cd maven
    wget http://apache.lauf-forum.at/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz
    tar -xvf apache-maven-3.3.3-bin.tar.gz
    rm apache-maven-3.3.3-bin.tar.gz
    
  • Edit maven config
    vi $OPENSHIFT_DATA_DIR/maven/apache-maven-3.3.3/conf/settings.xml
    

    Add the following to the tag (replace by your OpenShift UID first)

    /var/lib/openshift//app-root/data/.m2

    (press i button for edit mode, insert, then press esc button, enter :wq, finally press return button)

  • Browse to
    https://jenkins-.rhcloud.com/configure

    Set Environment variables
    PATH=$OPENSHIFT_DATA_DIR/maven/apache-maven-3.3.3/bin:$PATH
    M2_HOME=$OPENSHIFT_DATA_DIR/maven/apache-maven-3.3.3

  • And that’s it, your builds are now running on the custom maven!
    This allows for using a specific JDK in Jenkins. You could just choose a specific JDK via Jenkins console. This is comfortable, but has one disadvantage: It takes a lot of memory (approx. 600MB per JDK), because the JDK is stored twice – compressed in cache to be sent to slave and again uncompressed to be used on the master. If you got enough memory, you’re done here.

    However, In case you’re running a small gear with only 1GB of memory, you might want to save a bit of your precious memory. The following example shows how to do so for JDK 8 update 51 build 16.
    On SSH:

    cd $OPENSHIFT_DATA_DIR
    mkdir jdk
    cd jdk
    wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u51-b16/jdk-8u51-linux-x64.tar.gz
    tar -xvf jdk-8u51-linux-x64.tar.gz
    rm jdk-8u51-linux-x64.tar.gz
    
  • Then go to Jenkins
    https://jenkins-.rhcloud.com/configure

    JDK installations | JDK
    Name=SlaveOnly-Custom-JDK8u51
    JAVA_HOME=$OPENSHIFT_DATA_DIR/jdk/jdk-8u51-linux-x64

Building GitHub projects on Jenkins slaves on OpenShift

This post showed how to build GitHub projects with Jenkins, Maven and SonarQube 4 on OpenShift. For starters, it used the Jenkins master node for running build jobs. However, when running on a small gear, the master node might run out of memory pretty fast, resulting in a reboot of the node during builds.

In order to resolve this issue, there are two options:

  • limitting the memory of the build or
  • running the build on a slave node.

As spawning additional nodes is easy in a PaaS context such as OpenShift and provides a better performance than running builds with small memory, the slave solution seems to be the better approach.

This post shows how.

  1. Create new DYI app as a slave node (a how-to can be found here), name the node e.g. slave
  2. Create node in Jenkins
    1. Go to Jenkins web UI and create new node:
      https://jenkins-.rhcloud.com/computer/new
    2. Set the following values:
      Remote FS root:/app-root/data folder on slave. Typically this is /var/lib/openshift//app-root/data/jenkins, you can find out by SSHing to the slave node and calling

      echo $OPENSHIFT_DATA_DIR/app-root/data/jenkins

      Labels: Some label to use within builds to refer the node, e.g. OS Slave #1
      Host: the slave’s hostname, e.g. slave-.rhcloud.com

    3. Add Credentials
      username:
      Private Key File:Path to a private key file that is authorized for your OpenShift account. In the <a href="https://itaffinity.wordpress.com/2015/02/26/building-github-projects-with-jenkins-maven-and-sonarqube-4-1-1-on-openshift/”>first post this path was used: /var/lib/openshift//app-root/data/git-ssh/id_rsa. Note: $OPENSHIFT_DATA_DIR seems not to work here.
      BTW: You can change the credentials any time later via this URL

      https://jenkins-.rhcloud.com/credentials/
  3. Prepare slave node: Create same environment as on master in the <a href="https://itaffinity.wordpress.com/2015/02/26/building-github-projects-with-jenkins-maven-and-sonarqube-4-1-1-on-openshift/”>first post
    1. Create folder structure
      mkdir $OPENSHIFT_DATA_DIR/jenkins
      mkdir $OPENSHIFT_DATA_DIR/.m2
      echo -e "<settings><localRepository>$OPENSHIFT_DATA_DIR/.m2</localRepository></settings>" > $OPENSHIFT_DATA_DIR/.m2/settings.xml
      
    2. Copy SSH directory from master to same directory on slave, e.g.
      scp -rp -i $OPENSHIFT_DATA_DIR/.ssh $OPENSHIFT_DATA_DIR/.ssh <slave's UID>@slave-<your account>.rhcloud.com:app-root/data/.ssh
    3. As the different cartridges (jenkins and DIY) have different environment variables for their local IP addresses ($OPENSHIFT_JENKINS_IP vs $OPENSHIFT_DIY_IP) we’ll have to improvise at this point. There are two options: Either
      1. Replace all occurrences of $OPENSHIFT_JENKINS_IP
        In all builds and in

        https://jenkins-.rhcloud.com/configure

        Sonar | Sonar installations
        Database URL: jdbc:postgresql://$OPENSHIFT_DIY_IP:15555/sonarqube
        or

      2. Create an $OPENSHIFT_JENKINS_IP environment variable on your slave machine
        rhc env set OPENSHIFT_JENKINS_IP=<value of  $OPENSHIFT_DIY_IP> -a slave

        You can find out the value of $OPENSHIFT_DIY_IP by SSHing to the slave and execute

        echo $OPENSHIFT_DIY_IP 
      3. I’d love to hear suggesstions that do better 😉
  4. Adapt Build
    Easiest way is to not use the master at all.
    To do so, go to

    https://jenkins-.rhcloud.com/configure

    and set # of executors to 0.
    Hit Apply

  5. Limit memory usage.
    Make sure the slave does not run out of memory (which leads to a restart of the node):
    Global properties | Environment variables
    name: MAVEN_OPTS
    value: -Xmx512m
    Hit Save.
  6. Now run a build. It should run on the slave and hopefully succeed 🙂

See also Libor Krzyzanek’s Blog: Jenkins on Openshift wi… | JBoss Developer

Building GitHub projects with Jenkins, Maven and SonarQube 4.1.1 on OpenShift

Basic installation SonarQube

There are different community-driven sonar cartridges around. There is

  • this one that bases on a Tomcat cartridges and provides SonarQube 3.x and
  • that one that comes with SonarQube 4.0.
  • The most uptodate and flexible one is this, though. It downloads a specific version of SonarQube with each build. At the moment it works with version 4.1.1. I’m still working on getting SonarQube 5 to run on openshift, but haven’t succeeded, yet.

There also is a tutorial that shows how to install SonarQube 3.1.1. It also contains general thoughts on how to bypass OpenShift’s restrictions.

Anyway, to install SonarQube 4.1.1 execute the following steps on your machine:

    1. rhc app create sonar diy-0.1 postgresql-9.2

Make sure to remember the login and passwords!

  1. git rm -r diy .openshift misc README.md
    git remote add upstream -m master https://github.com/worldline/openshift-sonarqube.git
    git pull -s recursive -X theirs upstream master
    git push
    
  2. Login to your SonarQube instance at
    http://sonar-<yourAccount>.rhcloud.com/

    The default login and passwords are admin / admin.
    You might want to change the password right away!

Basic installation Jenkins

A lot of information within this paragraph was taken from here.

  1. Create Jenkins gear with Git-SSH
    rhc create-app jenkins  jenkins-1  "https://cartreflect-claytondev.rhcloud.com/reflect?github=majecek/openshift-community-git-ssh"
  2. Authorize your Jenkins node to communicate with other gears (and with you Git Repository)
    Generate SSH key for your Jenkins node
    SSH to the jenkins node

    ssh-keygen -t rsa -b 4096 -f $OPENSHIFT_DATA_DIR/git-ssh
  3. Add the key to your OpenShift, either
    • via web console
      In SSH console

      cat id_rsa.pub

      then copy and paste the output into web console
      or

    • via rhc
      Download the public key (id_rsa.pub) to your host (e.g. by SFTP) and use the

      rhc sshkey add

      command to authorize the public keys for your OpenShift account.
      If you plan on accessing a private repo or want to allow jenkins committing to your repo (e.g. for generate releases with the maven release plugin) you should also add the key to your repo account. See GitHub Help.

  4. Install Plugins
    Browse to Update Center

    https://jenkins-<yourAccount>.rhcloud.com/pluginManager/advanced

    and hit Check Now (as described here).
    Then go to the Available tab and install

    1. Sonar Plugin,
    2. GitHub plugin,
    3. embeddable-build-status (if you’d like to include those nifty build badges in you README.md).

    While you’re at it, you might as well update the already installed plugins in the Updates tab.
    Then hit Install without restart or Download and install after restart. If necessary, you can restart your app like so

    rhc app restart -a jenkins
  5. Set up maven settings.xml to a writable location.
    • SSH to Jenkins
      mkdir $OPENSHIFT_DATA_DIR/.m2
      echo -e "<settings><localRepository>$OPENSHIFT_DATA_DIR/.m2</localRepository></settings>" > $OPENSHIFT_DATA_DIR/.m2/settings.xml
      
    • Browse to Configure System
      https://jenkins-<yourAccount>.rhcloud.com/configure

      Default settings provider: Settings file in file system
      File path=$OPENSHIFT_DATA_DIR/.m2/settings.xml

  6. Set up main Jenkins node as slave (easy to set up and doesn’t need extra gears).
    Go to Configure System

    https://jenkins-<yourAccount>.rhcloud.com/configure

    and set
    # of executors: 1
    As an alternative, you could also use another gear as dedicated Jenkins slave. To do so, follow the steps described here.

    [EDIT 2015-08-09: As it turned out, memory is too low to run the jenkins master and builds on one node. See my second post on how to introduce a dedicated slave node to this setup]

  7. Setup sonar plugin
    On the Jenkins frontend, go to Configure System

    https://jenkins-<yourAccount>.rhcloud.com/configure
    • Global properties,
      tick Environment variables
      Click Add
      name=SONAR_USER_HOME
      value=$OPENSHIFT_DATA_DIR
      See here for more information.
    • Then set up the plugin itself
      Navigate to Sonar, Sonar installations and set the following
      Name=<be creative>
      Server URL:

      http://sonar-<yourAccount>.rhcloud.com/

      Sonar account login: admin
      Sonar account password: <your pw>, default: admin
      Database URL: jdbc:postgresql://$OPENSHIFT_JENKINS_IP:15555/sonar
      Database login: The admin account that was returned when you first created the sonar application
      Database password: The password that was returned when you first created the sonar application

    • Hit Save

Configure build for a repository

Now lets set up our first build.

  1. Go to
    https://jenkins-<yourAccount>.rhcloud.com/view/All/newJob

    Item name: <your Project name>
    Build a free-style software project (Unfortunately, Maven projects do not work due to OpenShift’s restrictions.)
    Hit OK

  2. On the next Screen
    GitHub project:

    https://github.com/<your user>/<your repo>/

    Source Code Management:

    https://github.com/<your user>/<your repo>.git

    Branch Specifier (blank for ‘any’): origin/master
    Build Triggers: Tick: Build when a change is pushed to GitHub
    Build | Execute Shell

    cd $WORKSPACE
    # Start the actual build
    mvn clean compile test package
    

    Post-build Actions | Add post-build action | Sonar

  3. I’d also recommend the following actions
    Post-build Actions | Add post-build action | Publish JUnit test result report
    Test report XMLs=target/surefire-reports/TEST-.xml*
    Post-build Actions | Add post-build action | E-mail Notification
    Recipients=<your email address>
  4. Hit Apply.
  5. That’s it for the basic build set up. Now for the fun part: We need to find a way for Jenkins to reach sonar’s database.
    We’ll use an SSH tunnel for that.
    Build | Add build step | Execute Shell
    Now enter the following:

    # Make sure Tunnel for Sonar is open
    # Find out IP and port of DB
    OPENSHIFT_POSTGRESQL_DB_HOST_N_PORT=$(ssh -i $OPENSHIFT_DATA_DIR/git-ssh/id_rsa -o "UserKnownHostsFile=$OPENSHIFT_DATA_DIR/git-ssh/known_hosts" <UID>@sonar<yourAccount>.rhcloud.com  '(echo `printenv OPENSHIFT_POSTGRESQL_DB_HOST`:`printenv OPENSHIFT_POSTGRESQL_DB_PORT`)')
    # Open tunnel to DB
    BUILD_ID=dontKillMe nohup ssh -i $OPENSHIFT_DATA_DIR/git-ssh/id_rsa -o "UserKnownHostsFile=$OPENSHIFT_DATA_DIR/git-ssh/known_hosts" -L $OPENSHIFT_JENKINS_IP:15555:$OPENSHIFT_POSTGRESQL_DB_HOST_N_PORT -N <UID>@sonar<yourAccount>.rhcloud.com &
    

    This will tunnel requests from your Jenkins’ local Port 15555 via SSH to your sonar gear, which will forward it to its local PostgreSQL database.
    What is missing is script that explicitly closes the tunnel. But for now I’m just happy that everything is up and running. The tunnel will eventually be closed after a timeout. Let me know if you have any ideas how to improve the tunnel handling.

  6. Finally, press Save and you’re almost good to go.
  7. Before running your first build you should SSH to your Jenkins once more and
    ssh -i $OPENSHIFT_DATA_DIR/git-ssh/id_rsa -o "UserKnownHostsFile=$OPENSHIFT_DATA_DIR/git-ssh/known_hosts" <UID>@sonar<yourAccount>.rhcloud.com

    so the sonar node is added to the list of know hosts.