1. Changes
  2. Automated test suite
  3. Tor Browser
  4. Thunderbird
  5. WhisperBack
  6. Incremental upgrades
  7. Torrents
  8. Real (non-VM) hardware
Read this document from the branch used to prepare the release.

Changes

Packages

This test can only be done by a Tails developer (Foundations Team).

Compare the list of bundled packages and versions with the one shipped last time. The new .packages file should have been attached to the call for testing email sent by the Release Manager after the images have been built.

/usr/bin/diff -u \
        ${MASTER_CHECKOUT:?}/wiki/src/torrents/files/tails-amd64-${PREVIOUS_VERSION:?}.packages \
        ${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.packages \
        | wdiff --diff-input  --terminal

Check the output for:

  • new packages that may cause harm or make the images unnecessarily big

  • packages that were erroneously removed

  • new versions of software we might not have audited yet

Image size

Check the images size has not changed much since the last release.

In a directory with many Tails ISO and USB images (archive):

(
cd "${ISOS:?}"
find -L \( -iname "tails*.iso" -o -iname "tails*.img" \) \
       -type f -ls | sort -n -k 7
)

Automated test suite

  • Point --old-iso to the ISO image of the previous stable release.

  • We want the test suite run to be as "vanilla" and reproducible as possible, so the only allowed options for run_test_suite are:

    • --artifacts-base-uri
    • --capture and --capture-all
    • --tmpdir
    • --view
    • --vnc-server-only
    • --iso and --old-iso (in fact, both of these are mandatory)

    For cucumber itself we only allow the --format option.

See:

Tor Browser

Miscellaneous functionality

  • Test if uBlock works:

    • The uBlock icon must be visible.

    • Visit a website that normally displays ads, such as https://www.nytimes.com/. The ads should not be displayed and the uBlock icon should display a strictly positive number of blocked elements.

Security and fingerprinting

  1. Download the version of Tor Browser included in the Tails release you are testing.

  2. Start that Tor Browser on a non-Tails Linux.

  3. Compare the fingerprint of that Tor Browser with the one of Tails' Tor Browser, using Test your browser on https://coveryourtracks.eff.org/.

    To see the details we're interested in, scroll down to check Detailed results and click on Detailed view.

    • The exposed User-Agent should match the latest Tor Browser's one.

    • Ignore the result of the "blocking tracking ads" and "blocking invisible trackers" tests, which seem unreliable (we've seen different results for the very same version of Tor Browser).

    • Ignore the "bits of identifying information" in the "Your results" summary, it is not clear what this number is (it is not the sum of the results for each section).

    • Instead compare results section-by-section. The only section that is allowed to differ is the Screen Size and Color Depth in case you couldn't test the browsers with the exact same window dimensions.

  4. If the fingerprints differ, ask the Release Manager to update the Browser fingerprint section of the known issues page.

Thunderbird

Only perform this test if one of these conditions is met:

  • We're testing a release candidate, such as Tails 7.42~rc1.

  • The release under testing upgrades to a major Thunderbird version. For example, the last Tails release shipped Thunderbird 87.9, and the one you're testing includes Thunderbird 88.1.

To check that the EHLO/HELO SMTP message is not leaking anything at the application level:

  1. In the Welcome Screen, set an administration password.

  2. Start Thunderbird using the GNOME Applications menu.

  3. Configure the outgoing SMTP server for an email account so it uses STARTTLS on port 587 (Thunderbird will send two EHLO/HELO: one before TLS is initiated; one after; the assumption here is that Thunderbird will send the same both times).

  4. Run sudo t*****dump -n -i lo -w dump.pcap while sending an email to capture the packets before Tor encrypts it, then close t*****dump. Note that the packet containing EHLO/HELO will be sent really early, so even if the email failed (e.g. because the mail server doesn't support plaintext SMTP on port 587) we are ok.

  5. Check the dump for the HELO/EHLO message and verify that it only contains 127.0.0.1: sudo t*****dump -A -r dump.pcap | grep EHLO

WhisperBack

  1. I should be able to send a bug report with WhisperBack.

  2. We receive this encrypted bug report via the [email protected] email address and your email client can decrypt it.

Incremental upgrades

⚠ Wait until the Release Manager removes this warning.

(automate: #18004)

Follow the instructions below for each of these versions of Tails:

⚠ PLACEHOLDER

  1. Install this version of Tails from scratch to a USB stick. A system that was upgraded to that version will not do.

  2. Start from that USB stick.

  3. Set an administration password in the Welcome Screen.

  4. Prepare the system so it can upgrade to the version we're testing:

     sudo sh -c 'sed -i /^TAILS_CHANNEL=/d /etc/os-release &&
             echo TAILS_CHANNEL=\"test\" >> /etc/os-release' && \
 systemctl --user restart tails-upgrade-frontend.service

    Don't wait for the last command to complete: this will happen later on.

  5. Connect to Local Network and to Tor

  6. Accept the suggestion to upgrade to the version we're testing

  7. Once the upgrade has been applied and you're suggested to restart Tails, do that.

  8. Verify that the resulting, upgraded system "works fine":

    • it boots

    • The About Tails app reports the expected version: the one we are testing

    • You can connect to Tor using Tor Connection

    • Tor Browser can browse websites

    • the Unsafe Browser starts

Torrents

⚠ Wait until the Release Manager removes this warning.

Download the images with the Torrent files provided by the Release Manager:

  • ISO

  • USB image

Wait until the downloads complete before calling this done.

Real (non-VM) hardware

UEFI boot

Note that for emergency releases, we do not always optimize the ordering of files in the SquashFS, which might make them boot somewhat slower.

On a 64-bit computer that has Secure Boot enabled and that is configured to prefer UEFI:

  1. Freshly install the version of Tails being tested to a USB stick.

  2. Boot this USB stick on bare-metal a first time to trigger re-partitioning.

    • You should see the "GNU GRUB" title, as opposed to a syslinux menu.

    • Wait until you see the Welcome Screen.

  3. Shutdown Tails

  4. Boot this USB stick a second time, add the login option to GRUB, and measure the boot time (from the GRUB menu until the GNOME desktop is ready). Take note of the boot time you measured.

  5. Go through steps 1-4 again, but this time using the previous Tails version. Use the same computer and the same USB stick.

  6. Compare the boot times you measured at steps 4 and 5. Expected result: the new Tails should not be significantly slower to start than the old one.

BIOS boot

On a computer configured to prefer legacy BIOS boot:

  1. Freshly install the version of Tails being tested to a USB stick.

  2. Boot this USB stick on bare-metal a first time to trigger re-partitioning.

    • You should see the "SYSLINUX" title in the bootloader, as opposed to GRUB.

    • Wait until you see the Welcome Screen.

  3. Shutdown Tails

  4. Boot this USB stick a second time.

  5. You should see the "SYSLINUX" title in the bootloader.

  6. You should see the Welcome Screen appear.