- contribute
- Linux kernel
This document is about:
How we keep track of upcoming Linux kernel upgrades we may want, or have to, apply in Tails.
How we apply such upgrades.
Background
… where the reader learns why this page is even needed and a few relevant facts they should be aware of.
APT repositories
Tails is built using a combination of snapshots of the Debian archive and overlay APT suites. This greatly impact the timeline and processes of upgrading any Debian package in Tails, and in particular the Linux kernel. This document assumes a good understanding of this somewhat complex system.
The Debian side of things
We generally ship the latest Linux kernel from Debian stable backports, and occasionally from unstable when a critical security fix is not available in stable backports yet.
The Linux kernel binary packages include an ABI version number, e.g.
for linux-image-4.18.0-2-amd64
the ABI is version 2.
Sometimes, the only way to include security fixes is to upgrade Linux to a major new version. For example, say the current Tails release includes Linux 5.18.8, an important security issue was fixed in 5.18.9 and in 5.19.6. If 5.19.6 was uploaded to unstable but 5.18.9 never was, then our only option to include this security fix is to upgrade to 5.19.6.
We need to decide
Which exact Linux kernel version we ship in a given Tails release is a trade-off between packages availability, security fixes, limiting the risks of hardware support regressions, and adding support for new hardware. The Foundations Team has to make this decision during every Tails release cycle. Due to the way we manage our APT repositories and to the fact we don't publish release candidates for Tails bugfix releases, how this decision is made and implemented depends in great part on what kind of Tails release is upcoming (major or bugfix).
The KERNEL_VERSION
variable in config/variables
determines what version of the kernel will be installed during the
build. This includes the ABI version number, so even our devel
branch does not get Linux point-release (e.g. 4.19.7) upgrades
automatically if the ABI is bumped.
Process
A Foundations Team member (generally the team lead, so far) creates a tracking issue whenever:
A new major version of Linux is released. At this point, that version is generally available only in Debian experimental.
A Linux point-release (e.g. 4.19.7) that includes potentially relevant security fixes or changes the ABI is uploaded to Debian unstable.
Once this new kernel is available in our APT snapshots a Foundations Team member (you!) gathers the data that will inform our decision. There are two aspects to it: how it works for us and what the risk/benefit of the upgrade is.
Determine what kernel package you want to test
Linux kernel in debian is a bit harder to grasp than other packages. So, what
kernel are currently available in debian? linux-signed-amd64 tracker
page will tell us. In the
"versions" block there are the currently available kernel versions. Those
versions are not, however, the package name you need. Click on that, and look
for every linux-image-(.*)-amd64
package. The regex group is what you need
later for KERNEL_VERSION
.
To know if a package fixes a specific CVE, go to the CVE page and see the status of every version.
Also have a look at the changelog if you have any doubt: that will most probably mention the most important CVEs that gets fixed, and which upstream version is used. Also have a look at the date! That can give you a good hint of whether it's possible that you'll find a specific bugfix in it, or not.
Test the new kernel
To learn how the new kernel works for us:
Fork a branch off
devel
calledNNNNN-linux-X.Y-force-all-tests
.Adjust
KERNEL_VERSION
inconfig/variables
. IfKERNEL_VERSION
is unchanged (point-release without ABI bump), then this branch is identical todevel
and its only purpose is to force Jenkins to run our entire test suite (-force-all-tests
). But since this branch has no commit on top ofdevel
, Jenkins will ignore it, so you need to create a dummy commit. Despite the variable name,KERNEL_VERSION
must be set to the package name. For example, if you want to get the kernel version 4.42.1337, most probably4.2.1337
is not the right value for the variable. Instead, what you need to do is determine which kernel package contains the intended kernel version.Check if the pinning is right in
config/chroot_apt/preferences
. We might have pinned the current kernel, or you may need to pin the one you want to install.Push this new branch to our CI.
Quickly test a build from this branch on your hardware.
Compare the Jenkins build and test results to the ones for our
stable
anddevel
branch.Report your findings on the issue.
Gather other data that will inform our decision
Take notes on the issue of the most relevant bits, or lack thereof, for:
Security issues fixed since the version we currently ship
To do so, look for "CVE" in the Changelog entries newer than the kernel we currently ship.
Major security fixes still not fixed in the version we're considering
(in which case, depending on when we are in our own release process, it may be worth delaying the analysis a bit, or exceptionally installing the kernel from unstable)
To do so, use the Debian security tracker for Linux.
Important regressions
… i.e. bugs with severity important or higher recently reported to Debian against this version of the kernel.
To do so, in the Debian BTS, look for bugs with severity important or higher that satisfy one of these criteria:
the bug title includes the kernel version we're considering upgrading to
the bug was reported very recently (the bug title may not say what kernel version is affected, but once you open the bug's page, you'll see that info; for example, Debian bug #960871 is marked as affecting linux/5.6.7-1)
If there are important hardware support regressions, ask someone who has an archive of WhisperBack reports (for example intrigeri) to check the popularity in Tails of the affected hardware: while assessing the impact of regressions, we need to take into account that some devices are much more popular among Tails users than among the general Debian users population.
Debian BTS tips
scanning through udd.debian.org can be boring. The advanced search can't always express the logic you want. So here is some tip:
curl -s 'https://udd.debian.org/bugs.cgi?release=sid&merged=ign&fnewerval=7&flastmodval=7&packages=linux&cseverity=1&sortby=id&sorto=desc&format=json' | jq '.[]|select(.severity != "normal" and .severity != "wishlist" and .severity != "minor")' > linux-bugs.json
jq < linux-bugs.json .
# recent bugs with severity >= important, only ID and title, easy to grep
jq -r < linux-bugs.json 'select(.last_modified >= "2021-10-01") | [.last_modified, .id, .title] |join("\t")'
# this is what you need for the second criterion!
jq -r < linux-bugs.json 'select(.last_modified >= "2021-10-01") | [.last_modified, .id, .title] |join("\t")' | grep -v linux-image-
# this is the first criterion, of course you must select the right version
jq -r < linux-bugs.json '[.last_modified, .id, .title] |join("\t")' | grep -F linux-image-5.10.0-9
# exclude "5.14" from results
jq < linux-bugs.json 'select(.title | test("^(?!.*5\\.14)"; "i"))'
Relevant hardware support improvements
To get a rough guts feeling about it, look for improvements related to graphics cards and Wi-Fi adapters support while skimming over:
the "Drivers" section of Kernelnewbies's human-readable changelog of upstream changes (only relevant for a new major Linux release)
the Debian package's Changelog
Make a decision
With all the information you gathered earlier, use your best judgement to make a decision. The kind of decision you need to make depends on several factors:
If the upcoming Tails release is a major one and upgrading the kernel did not require modifying
KERNEL_VERSION
, then if we do nothing particular, our next release will get the upgrade. So you need to decide whether this kernel upgrade is bad enough for us to opt-out, or important enough for us to fix whatever regressions it may bring. In practice, opting-out of the upgrade is rarely the best choice but YMMV.Otherwise, upgrading Linux in our next release will require work, so you need to decide whether the overall cost/benefit of the upgrade is worth it, factoring in the work needed and all the data you've gathered earlier.
If in doubt, ask your team-mates :)
If the decision is "do nothing", close the issue and stop reading here. Else, read on.
Implement the decision
How to implement the decision depends on what kind of Tails release is upcoming.
Bugfix release
The branch you've used so far to get results from our CI was forked
off devel
so it's not a valid candidate for merging into stable
.
Therefore, create a new
NNNNN-linux-X.Y-stable-force-all-tests
topic branch forked
off stable
and transplant onto it the commits you had to create on
your devel
-based topic branch (git rebase --onto
or git
cherry-pick
).
But the new resulting topic branch will likely not build: a bugfix
release is built from our stable
branch, that uses a set of APT
snapshots frozen during the last major release process, and these old
snapshots probably don't include the version of the kernel you want to
upgrade to. So there are two options:
Either bump the APT snapshot of the
debian
archive to the oldest one that includes this new kernel. When it can reasonably be done, this is the cheapest option so it's worth trying it first:Update
config/APT_snapshots.d/debian/serial
, commit, push and trigger a build on Jenkins.Bump the expiration date, for the snapshot of the
debian
archive that you've switched the branch to, to 6 months from now.Compare the
.build-manifest
and.packages
files generated by building your topic branch with the ones for the current Tails release.If the diff seems reasonable, fine. Otherwise, fall back to the next option (freeze exception).
Or use our freeze exception mechanism i.e. import the new Linux packages into a dedicated overlay suite in our custom APT repository and make your topic branch use it.
The CI results you got with your previous topic branch based on
devel
are not valid for your new branch: the new kernel may work
fine in the former case thanks to corresponding userspace changes, but
cause trouble in the context of our stable
branch. So push your
branch to our CI, trigger a build in Jenkins and analyze the
test results. Once happy:
Follow our usual process to get it reviewed and merged.
Follow the instructions to enable new security features.
Major release
If you decided to opt-out from a kernel upgrade we would otherwise automatically include: piggy-back on our freeze exception mechanism to force the installation of an older kernel.
Else, you're trying to upgrade the kernel. It turns out you already have prepared the very topic branch we need to do that, so:
Follow our usual process to get it reviewed and merged.
Follow the instructions to enable new security features.
Enable new security features
This section assumes we have decided to upgrade to a major new version of Linux.
Major new kernel versions often bring new security features. After each major kernel release, Kees Cook publishes on his blog an article titled "security things in Linux $VERSION" about these improvements.
What you need to do depends on what it takes to benefit from each such improvement:
Enabled by default: nothing to do, profit :)
Guarded by local configuration such as a sysctl or a kernel command line option: file a GitLab issue about it, with the
Core work: Foundations Team
label. Optionally, do the work yourself: once you've got CI results about your topic branch with this new option disabled, add a commit that enables it and compare the results (including test suite total run time, to spot important performance regressions).Needs to be enabled at kernel configuration time: check if it's been enabled in the Debian kernel; if it's not been enabled there yet and enabling it would make sense in a general-purpose distro kernel where UX breakage and performance regressions can be serious problems, file a wishlist bug against the
linux
source package. Point to Kees' post and explain why you think it's worth it.