- contribute
- working together
- roles
- sysadmins
- Automated ISO/IMG builds and tests on Jenkins
Configuration
master:
a few Jenkins plugins installed with
jenkins::plugin
YAML jobs configuration lives in a dedicated Git repository; Jenkins Job Builder uses it to configure Jenkins
slaves:
web server:
Upgrade policy
Here are some guidelines to triage security vulnerabilities in Jenkins and the plugins we have installed:
Protecting our infra from folks who have access to Jenkins
→ Upgrading quarterly is sufficient.
Protecting our infra from attacks against folks who have access to Jenkins
For example, XSS that could lead a legitimate user to perform unintended actions with Jenkins credentials (i.e. root in practice).
→ We should stay on top of security advisories and react more quickly than "in less than 3 months".
Protecting our infra from other 3rd-parties that affect Jenkins' security
For example, say some Jenkins plugin, that connects to remote services, has a TLS certificate checking bug. This could potentially allow a MitM to run arbitrary code with Jenkins orchestrator or workers permissions, i.e. root.
→ We should stay on top of security advisories and react more quickly than "in less than 3 months".
Upgrade steps
Preparation:
- ☐ Go through the changelog, paying attention to changes on how agents connect to controller, config changes that may need update, important changes in plugins, etc.
Deployment:
- ☐
Take note of currently running builds before starting the upgrades.
- ☐
Deploy Jenkins upgrade to latest version available using Puppet.
- ☐
Generate a list of up-to-date plugins by running this Groovy script in the Jenkins Script Console. Make sure to update the initial list containing actively used plugins if there were changes.
- ☐
Generate updated Puppet code for
tails::jenkins::master
using this Python3 script and the output of the above script. - ☐
Deploy plugin upgrades using the code generated above.
- ☐
Restart all agents.
- ☐
Manually run the Update jobs script (may be needed so XML is valid with current Jenkins):
sudo -u jenkins /usr/local/sbin/deploy_jenkins_jobs update
- ☐
Wrap up:
- ☐
Go through warnings in Jenkins interface.
- ☐
Manually remove uneeded plugins from /var/lib/jenkins/plugins.
- ☐
Restart builds that were interrupted by Jenkins restart.
- ☐
Update the Jenkins upgrade steps documentation in case there were changes.
- ☐
Schedule next update.
- ☐
Agent to controller connections
These are the steps a Jenkins agent does when connecting to the controller:
Fetch connection info from
https://jenkins.dragon:8080
(see thetails::jenkins::slave
Puppet class).Receive the connection URL
https://jenkins.tails.boum.org
("Jenkins URL", manually configured in Configure System).Resolve
jenkins.tails.boum.org
to192.168.122.1
(because of libvirt config).Connect using HTTPS to
jenkins.tails.boum.org:443
.Learn about port
42585
(fixed "T***** port for inbound agents", manually configured in Configure Global Security).Finally, connect using HTTP to
jenkins.tails.boum.org:42585
.
Generating jobs
We generate automatically a set of Jenkins jobs for branches that are active in the Tails main Git repository.
The first brick extracts the list of active branches and output the needed information:
- config/chroot local-includes/usr/lib/python3/dist-packages/tailslib/git.py
- config/chroot local-includes/usr/lib/python3/dist-packages/tailslib/jenkins.py
This list is parsed by the generate_tails_iso_jobs
script run by
a cronjob and deployed by our puppet-tails
tails::jenkins::iso_jobs_generator
manifest.
This script output YAML files compatible with
jenkins-job-builder.
It creates one project
for each active branch, which in turn uses
several JJB job templates
to create jobs for each branch:
build_Tails_ISO_*
reproducibly_build_Tails_ISO_*
test_Tails_ISO_*
This changes are pushed to our jenkins-jobs git
repo by the cronjob, and thanks to their automatic deployment in our
tails::jenkins::master
and tails::gitolite::hooks::jenkins_jobs
manifests in our puppet-tails repo, these new
changes are applied to our Jenkins instance.
Passing parameters through jobs
We pass information from build job to follow-up jobs (reproducibility testing, test suite) via two means:
the Parameterized Trigger plugin, whenever it's sufficient
the EnvInject plugin, for more complex cases:
In the build job, a script collects the needed information and writes it to a file that's saved as a build artifact.
This file is used by the build job itself, to setup the variables it needs (currently only
$NOTIFY_TO
).Follow-up jobs imported this file in the workspace along with the build artifacts, then use an EnvInject pre-build step to load it and set up variables accordingly.
Builds
See automated builds in Jenkins.