Verification during release process

How is something verified to be reproducible? It is a matter of definition, which ideally requires that every conceivable compatible system should get the same result. As you'll see below we'll lower this requirement slightly for practical purposes. :)

Assumptions

1. The Release Manager (RM) is not evil! ♥

2. A compromised RM system can block arbitrary outgoing communication (e.g. email) so it cannot be trusted to initiate a check that's precisely meant to verify whether said system is compromised (and compromising our published products).

3. RMs don't plug their smartcard in a computer outside of the release process.

4. Our web site and the webserver that runs is is trusted (so much of other Tails stuff relies on this that we can't reasonably set the bar higher).

5. An attacker publishing a compromised Tails needs to update the IDF, UDFs and/or detached signatures to our website, and possibly replace a Git tag. If this is done through Git, there's a much greater chance we notice it after of our release process than during it. (If this is done directly on the web server, chances are that we don't notice it and it could even survive Git push and web site refresh, but it's out of scope here as said above.)

Goals

Here are our high-level goals and their rationales.

1. Identify reproducibility bugs (ideally before releasing)

   • We are claiming that we release reproducible products. We don't want to lie to the world.

   • We will hopefully involve third-parties in the reproducibility verification process at some point; they'll need confidence that our products are reliably reproducible, otherwise they might be discouraged and stop even trying.

   • A reproducibility bug can be a symptom of another bug.

2. Identify compromised RM system and/or Tails products ASAP

   • Allow RMs to sleep better at night.

   • Avoid letting users use a compromised Tails.

   • OTOH, generally speaking, anything that has to be done in a tight time frame is stressful. Specifically, steps that block the release process are stressful because it's hard to predict when one will need to be available to go through them. This has a big impact on the "P" in "ASAP".

3. Ensure what we've tested (QA) and reproduced matches what is published

Furthermore, let's make the process not depend on a few key persons in a tight, not well defined time frame (e.g. during release process). Availability is a serious constraint, and if e.g. the architects of the reproducibility efforts are the only ones able to follow the process it is an indication that the process probably is too complex, which makes it prone for errors.

Design and implementation

We do two reproducibility checks:

• For Goal 1, we do a sanity check that is only about identifying non-malicious bugs as soon as possible.

  • ISO
  • IUKs (not very useful at the moment since Jenkins doesn't build IUKs)

• For Goals 2 and 3, we do a second reproducibility check by a trusted third third-party (the Trusted Reproducer) after the RM has unplugged their OpenPGP smart card; this implies that the attacker must have had the compromised RM system sign compromised data earlier, and then either publish the compromised product during the confused release process time (when there's less chance for it to be noticed) or later (and then they have a good chance to be caught). This raises the bar for a successful attack a bit, although it leaves a window during which users can download a compromised Tails.

The rest of the time we're all careful about what's going on in the UDFs, IDFs, detached signatures and Git tags. Of course, we should have monitoring of these things and alerts whenever they change.