Configuration

Upgrade policy

Here are some guidelines to triage security vulnerabilities in Jenkins and the plugins we have installed:

  1. Protecting our infra from folks who have access to Jenkins

    → Upgrading quarterly is sufficient.

  2. Protecting our infra from attacks against folks who have access to Jenkins

    For example, XSS that could lead a legitimate user to perform unintended actions with Jenkins credentials (i.e. root in practice).

    → We should stay on top of security advisories and react more quickly than "in less than 3 months".

  3. Protecting our infra from other 3rd-parties that affect Jenkins' security

    For example, say some Jenkins plugin, that connects to remote services, has a TLS certificate checking bug. This could potentially allow a MitM to run arbitrary code with Jenkins orchestrator or workers permissions, i.e. root.

    → We should stay on top of security advisories and react more quickly than "in less than 3 months".

Upgrade steps

  • Preparation:

    • ☐ Go through the changelog, paying attention to changes on how agents connect to controller, config changes that may need update, important changes in plugins, etc.
  • Deployment:

    • Take note of currently running builds before starting the upgrades.

    • Deploy Jenkins upgrade to latest version available using Puppet.

    • Generate a list of up-to-date plugins by running this Groovy script in the Jenkins Script Console. Make sure to update the initial list containing actively used plugins if there were changes.

    • Generate updated Puppet code for tails::jenkins::master using this Python3 script and the output of the above script.

    • Deploy plugin upgrades using the code generated above.

    • Restart all agents.

    • Manually run the Update jobs script (may be needed so XML is valid with current Jenkins):

      sudo -u jenkins /usr/local/sbin/deploy_jenkins_jobs update
      
  • Wrap up:

    • Go through warnings in Jenkins interface.

    • Manually remove uneeded plugins from /var/lib/jenkins/plugins.

    • Restart builds that were interrupted by Jenkins restart.

    • Update the Jenkins upgrade steps documentation in case there were changes.

    • Schedule next update.

Agent to controller connections

These are the steps a Jenkins agent does when connecting to the controller:

  1. Fetch connection info from https://jenkins.dragon:8080 (see the tails::jenkins::slave Puppet class).

  2. Receive the connection URL https://jenkins.tails.boum.org ("Jenkins URL", manually configured in Configure System).

  3. Resolve jenkins.tails.boum.org to 192.168.122.1 (because of libvirt config).

  4. Connect using HTTPS to jenkins.tails.boum.org:443.

  5. Learn about port 42585 (fixed "T***** port for inbound agents", manually configured in Configure Global Security).

  6. Finally, connect using HTTP to jenkins.tails.boum.org:42585.

Generating jobs

We generate automatically a set of Jenkins jobs for branches that are active in the Tails main Git repository.

The first brick extracts the list of active branches and output the needed information:

This list is parsed by the generate_tails_iso_jobs script run by a cronjob and deployed by our puppet-tails tails::jenkins::iso_jobs_generator manifest.

This script output YAML files compatible with jenkins-job-builder. It creates one project for each active branch, which in turn uses several JJB job templates to create jobs for each branch:

  • build_Tails_ISO_*
  • reproducibly_build_Tails_ISO_*
  • test_Tails_ISO_*

This changes are pushed to our jenkins-jobs git repo by the cronjob, and thanks to their automatic deployment in our tails::jenkins::master and tails::gitolite::hooks::jenkins_jobs manifests in our puppet-tails repo, these new changes are applied to our Jenkins instance.

Passing parameters through jobs

We pass information from build job to follow-up jobs (reproducibility testing, test suite) via two means:

  • the Parameterized Trigger plugin, whenever it's sufficient

  • the EnvInject plugin, for more complex cases:

    • In the build job, a script collects the needed information and writes it to a file that's saved as a build artifact.

    • This file is used by the build job itself, to setup the variables it needs (currently only $NOTIFY_TO).

    • Follow-up jobs imported this file in the workspace along with the build artifacts, then use an EnvInject pre-build step to load it and set up variables accordingly.

Builds

See automated builds in Jenkins.

Tests

See automated tests in Jenkins.