checkbox-cli Man page

CHECKBOX_NG(1) CheckboxNG CHECKBOX_NG(1)

NAME

checkbox_ng – CheckboxNG Documentation

CheckboxNG is a hardware testing tool useful for certifying laptops,
desktops and servers with Ubuntu. It is a new version of Checkbox that
is built directly on top of PlainBox

CheckboxNG replaces Checkbox, where applicable.

WARNING:
Documentation is under development. Some things are wrong, inaccu‐
rate or describe development goals rather than current state.

INSTALLATION
CheckboxNG can be installed from a PPA (recommended) or pypi on Ubuntu
Precise (12.04) or newer.

$ sudo add-apt-repository ppa:checkbox-dev/ppa && sudo apt-get update && sudo apt-get install checkbox-ng

RUNNING STABLE RELEASE UPDATE TESTS
CheckboxNG has special support for running stable release updates tests
in an automated manner. This runs all the jobs from the sru.whitelist
and sends the results to the certification website.

To run SRU tests you will need to know the so-called Secure ID of the
device you are testing. Once you know that all you need to do is run:

$ checkbox sru $secure_id submission.xml

The second argument, submission.xml, is a name of the fallback file
that is only created when sending the data to the certification website
fails to work for any reason.

REPORTING BUGS

To report bugs on the Checkbox project you will need a launchpad
account. You may find instructions on how to create one
useful. Once you
have an account you can report bugs .

On that page you can select the project you wish to file the bug on (we
use a number of projects to coordinate releases and we prefer to have
bugs associated with appropriate part of Checkbox). If you know the
right project to use, just use it and file the bug. If you don’t know
Checkbox internals that much or are in doubt just file it on the base
‘Checkbox’ project (you can use this direct link .) A member of the development team will
review your bug and re-assign it to the appropriate location. The bug
number will not change when that happens.

THE CHECKBOX STACK
The Checkbox Stack is a collection of projects that together constitute
a complete testing and certification solution. It is composed of the
following parts (see table below for extra details). All of the
projects are linked to from the Launchpad project group
.

Architecture Diagram
[image: Architecture Diagram] [image]

This diagram contains a high-level approximation of the current Check‐
box architecture. There are three main “pillars”. On the left we have
end products. Those are actual tools that certification and engineers
are using. On the right we have the test market. This is a open market
of tests vendors and suppliers. The tests are wrapped in containers
known as providers. In the center we have three shared components.
Those implement the bulk of the framework and user interfaces for test
execution. Finally in the bottom-left corner there is a part of check‐
box (a library) that is shared with HEXR for certain tasks. HEXR is a
out-of-scope web application used by part of the certification process.
Arrows imply communication with the shape of the arrow shows who calls
who.

As mentioned before, in the center column there are three main compo‐
nents of shared code (shared by everyone using the end products that
are discussed below). The shared code is composed of plainbox, check‐
box-ng and checkbox-gui. Component responsibilities are discussed in
more detail in the table below. Here we can see that checkbox-gui used
DBus API exposed by checkbox-ng, which in turn uses checkbox-support (a
helper library separated out so share some code with HEXR) and plain‐
box.

In the right hand side column there are various test providers. The
checkbox project is producing and maintaining a number of providers
(see the table below) but it is expected that our downstream users will
also produce their own providers (specific to a customer or project).
Eventually some providers may come from third parties that will adopt
the format.

Lastly in the bottom-left corner, the shared library, this library con‐
tains many parsers of various file formats and output formats. Techni‐
cally this library is a dependency of HEXR, checkbox-ng and of
providers. As an added complexity the library needs to be called from
python3 code and python2 code.

NOTE:
The communication between checkbox-ng and plainbox is bi-direc‐
tional. Plainbox offers some base interfaces and extension points.
Those are all exposed through plainbox (using common APIs) but some
of those are actually implemented in checkbox-ng.

WARNING:
All internal APIs are semi-unstable. The DBus API is more stable in
practice but should not be relied upon. Projects are encouraged to
be merged into lp:checkbox where API transitions can be handled
gracefully. The only stable API is are file format specification
(job definitions and whitelits). Launcher specification will be sta‐
bilized in the next release.

Component Descriptions
┌────────────────────┬─────────────────────────┬─────────────────────┐
│Project │ Responsible for │ Type │
└────────────────────┴─────────────────────────┴─────────────────────┘

│Next Generation │ │ Application │
│Checkbox (GUI) │ · The │ │
│ │ C++/QML │ │
│ │ user │ │
│ │ interface │ │
│ │ │ │
│ │ · The graph‐ │ │
│ │ ical │ │
│ │ launcher │ │
│ │ for │ │
│ │ providers, │ │
│ │ e.g. │ │
│ │ check‐ │ │
│ │ box-certi‐ │ │
│ │ fica‐ │ │
│ │ tion-client │ │
├────────────────────┼─────────────────────────┼─────────────────────┤
│Next Generation │ │ Application │
│Checkbox (CLI) │ · The python │ │
│ │ com‐ │ │
│ │ mand-line │ │
│ │ interface │ │
│ │ │ │
│ │ · the text │ │
│ │ user │ │
│ │ interface │ │
│ │ │ │
│ │ · the SRU │ │
│ │ testing │ │
│ │ command │ │
│ │ │ │
│ │ · Additional │ │
│ │ certifica‐ │ │
│ │ tion APIs │ │
│ │ │ │
│ │ · sending │ │
│ │ data to │ │
│ │ Launchpad │ │
│ │ │ │
│ │ · sending │ │
│ │ data to │ │
│ │ HEXR │ │
│ │ │ │
│ │ · the DBus │ │
│ │ service │ │
│ │ needed by │ │
│ │ GUI │ │
├────────────────────┼─────────────────────────┼─────────────────────┤
│Client Certifica‐ │ │ Provider │
│tion Provider │ · canoni‐ │ │
│ │ cal-certi‐ │ │
│ │ fica‐ │ │
│ │ tion-client │ │
│ │ executable │ │
│ │ │ │
│ │ · client cer‐ │ │
│ │ tification │ │
│ │ whitelists │ │
└────────────────────┴─────────────────────────┴─────────────────────┘

│Server Certifica‐ │ │ Provider │
│tion Provider │ · server cer‐ │ │
│ │ tification │ │
│ │ whitelists │ │
│ │ │ │
│ │ · additional │ │
│ │ server │ │
│ │ whitelists │ │
├────────────────────┼─────────────────────────┼─────────────────────┤
│System-on-Chip │ │ Provider │
│Server Certifica‐ │ · SoC server │ │
│tion Provider │ certifica‐ │ │
│ │ tion │ │
│ │ whitelists │ │
├────────────────────┼─────────────────────────┼─────────────────────┤
│Checkbox Provider │ │ Provider │
│ │ · Almost all │ │
│ │ job defini‐ │ │
│ │ tions │ │
│ │ │ │
│ │ · Most of │ │
│ │ custom │ │
│ │ “scripts” │ │
│ │ │ │
│ │ · Default and │ │
│ │ SRU │ │
│ │ whitelist │ │
├────────────────────┼─────────────────────────┼─────────────────────┤
│Resource Provider │ │ Provider │
│ │ · Almost all │ │
│ │ resource │ │
│ │ jobs │ │
│ │ │ │
│ │ · Almost all │ │
│ │ resource │ │
│ │ “scripts” │ │
├────────────────────┼─────────────────────────┼─────────────────────┤
│Checkbox Support │ │ Library │
│ │ · Support │ │
│ │ code for │ │
│ │ various │ │
│ │ providers │ │
│ │ │ │
│ │ · Parsers for │ │
│ │ many text │ │
│ │ formats │ │
└────────────────────┴─────────────────────────┴─────────────────────┘

│PlainBox │ │ Library and Devel‐ │
│ │ · Almost all │ opment Toolkit │
│ │ core logic │ │
│ │ │ │
│ │ · RFC822 │ │
│ │ (job def‐ │ │
│ │ inition) │ │
│ │ parser │ │
│ │ │ │
│ │ · Configu‐ │ │
│ │ ration │ │
│ │ handling │ │
│ │ │ │
│ │ · Testing │ │
│ │ session │ │
│ │ (sus‐ │ │
│ │ pend/resume) │ │
│ │ │ │
│ │ · Job run‐ │ │
│ │ ner │ │
│ │ │ │
│ │ · Trusted │ │
│ │ launcher │ │
│ │ │ │
│ │ · Depen‐ │ │
│ │ dency │ │
│ │ resolver │ │
│ │ │ │
│ │ · Command │ │
│ │ line han‐ │ │
│ │ dling │ │
│ │ │ │
│ │ · The XML, │ │
│ │ HTML and │ │
│ │ XSLX │ │
│ │ exporters │ │
│ │ │ │
│ │ · and │ │
│ │ more… │ │
│ │ │ │
│ │ · Provider │ │
│ │ development │ │
│ │ toolkit │ │
│ │ │ │
│ │ · ‘plainbox │ │
│ │ start‐ │ │
│ │ provider’ │ │
│ │ │ │
│ │ · ‘man‐ │ │
│ │ age.py’ │ │
│ │ implemen‐ │ │
│ │ tation │ │
└────────────────────┴─────────────────────────┴─────────────────────┘

│Legacy Checkbox (no │ │ Monolithic Applica‐ │
│longer maintained) │ · Applications │ tion Library and │
│ │ │ Data │
│ │ · Qt4 GUI │ │
│ │ │ │
│ │ · Gtk2 GUI │ │
│ │ │ │
│ │ · Urwid (text) │ │
│ │ GUI │ │
│ │ │ │
│ │ · Core │ │
│ │ │ │
│ │ · Plugin and │ │
│ │ Event / Mes‐ │ │
│ │ sage Engine │ │
│ │ │ │
│ │ · Almost Every │ │
│ │ feature │ │
│ │ implemented │ │
│ │ a core plug‐ │ │
│ │ in │ │
│ │ │ │
│ │ · Data │ │
│ │ │ │
│ │ · Jobs and │ │
│ │ whitelists │ │
└────────────────────┴─────────────────────────┴─────────────────────┘

CHANGELOG
NOTE:
This changelog contains only a summary of changes. For a more accu‐
rate accounting of development history please inspect the source
history directly.

CheckboxNG 0.23 (unreleased)
· Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.23

CheckboxNG 0.22
· Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.22

CheckboxNG 0.3
· Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.3

CheckboxNG 0.2
· Bugfixes: https://launchpad.net/checkbox-ng/+milestone/0.2

CheckboxNG 0.1
· Initial release

· Support for displaying configuration

· Support for running SRU tests (automatic regression testing)

TEST SCRIPTS
Test ‘scripts’ are small programs which are used to aid in implementing
tests.

brightness_test
This script tests the brightness of the systems backlight can be
changed by using the kernel interfaces in /sys/class/backlight. There
may be more than one interface to choose from, so the correct interface
to use is selected by using the heuristic prescribed in
https://www.kernel.org/doc/Documentation/ABI/stable/sysfs-class-backlight.
The brightness is manipulated by updating the brightness file of the
interface and the actual_brightness file is checked to see if the value
was modified to the brightness selected.

PROFILES CONFIGURATION
Execution profiles, or launchers, allow specifying a predefined set of
configuration options that allow customization of the welcome screen,
displayed whitelists as well as saving results locally or sending the
submission file to Launchpad or to the Certification database/HEXR, as
well as some other parameters.

The profile settings are part of a launcher script and use either
checkbox-gui or checkbox-launcher (in text-mode/CLI) as a shebang to
interpret the key/values.

This document provides a reference on launcher functionality and syn‐
tax. To understand the design and concepts and see several examples,
you may want to read the tutorial on how to create launchers and their
relationship with legacy Checkbox.

Syntax
As checkbox-gui is a Qt application, settings must follow the INI-style
rules of the QSettings
class.

Multiple-line values are supported but must be enclosed in doubles
quotes and extra lines must start with one space, e.g:

[category] key = “Hello
World”

· From QML:

settings.value(“category/key”, i18n.tr(“default_value”))

· From C++:

settings->value(“category/key”, app.tr(“default_value”))

Conversely, checkbox-launcher-specific launchers must follow Python
ConfigParser syntax.

Also, some settings only make sense for either GUI or CLI, and are thus
not understood by the other. These are noted below.

Supported Settings
welcome/title
QML application title and welcome screen header. Defaults to
System Testing.

welcome/text
Welcome message to display on the first screen (checkbox-gui
supports Rich text allowing HTML-style markup). Defaults to

Welcome to System Testing.

[…]

suite/whitelist_filter
Regular expression to match a subset of whitelist filenames. On
checkbox-gui it defaults to .*. For checkbox-launcher it has no
default and must be defined.

suite/whitelist_selection
Pattern that whitelists need to match to be preselected. Python
regular expression. It has no default and must be defined. (CLI
only)

suite/skip_whitelist_selection
If set to true, user will not receive a choice of whitelist.
Only the preselected ones (see whitelist_selection) will be
selected. (CLI only).

suite/skip_test_selection
If set to true, user will not be allowed to deselect tests prior
to run: all tests in the selected whitelist will be run. (CLI
only)

submission/message
Header text of the submission pop-up , shown to the user after
submission has completed. (GUI only)

submission/input_type
Show a Text input field to enter the secure ID or the LP address
(default). To just save the results to disk, must use the none
value. To validate using a regex, must be regex. (GUI only)

submission/regex
Regular expression to validate input in submission field (e.g.
email, secure_id) if input_type is regex. (GUI only). RegExp‐
Validator, default .*

submission/input_placeholder
Temporary text to put in input field, used to guide the user.
Launchpad E-Mail Address (default) or Secure ID (15 or 18 char‐
acters). (GUI only)

submission/secure_id
Preconfigured secure_id to fill in the text field.

submission/ok_btn_text
The label for the “Send” button. Submit Results (default) or
Save Results. (GUI only)

submission/cancel_warning
Show to the user if he wants to exit without having saved the
report. You are about to exit this test run without saving your
results report. Do you want to save the report? (GUI only)

submission/submit_to_hexr
Boolean, add an extra header to also send the results to HEXR
(works with the certification transport)

exporter/xml_export_path
Location to save the XML submission file, if set to an empty
string will open a file save dialog. Default: /tmp/submis‐
sion.xml (GUI only)

transport/submit_to
Transport endpoint. Defaults to . Supports submission to
LP (the default, value launchpad), certification, or local (save
to disk)

transport/submit_url
URL to submit results to. This allows to upload to different
websites, for example it can upload directly to hexr, or to the
staging sites. Used only with the certification submit_to value.

transport/config_filename
Name of a custom config file to load. Config files are mainly
used to define environment variables. (CLI only)

transport/dont_suppress_output
If set, resources, local jobs and attachments will be output to
screen, this generates a lot of text and is mainly for debug‐
ging. (CLI only)

CHECKBOX/PLAINBOX LAUNCHERS TUTORIAL
This document provides an explanation of why launchers are necessary,
what you can achieve with them, and goes over several examples to bet‐
ter describe their capabilities. For a detailed reference on which set‐
tings are supported by launchers, and specific syntax for launcher
files, look at /profiles.

Legacy checkbox behavior control
In the past, Checkbox’s behavior was controlled by three mechanisms.

First, the functions of checkbox could be augmented by adding plugins.
For example, the ability to submit to certification website was added
by the checkbox-certification package using a plugin. The plugins
included by checkbox-certification and which add new behavior to base
checkbox were:

/usr/share/checkbox-certification/plugins/certify_message.py
/usr/share/checkbox-certification/plugins/submission_info.py
/usr/share/checkbox-certification/plugins/backup.py
/usr/share/checkbox-certification/plugins/certify_prompt.py
/usr/share/checkbox-certification/plugins/certify_report.py
/usr/share/checkbox-certification/plugins/certify_schemas.py

These added the way to prompt the user for submission-specific data,
generate the xml report, and other functions.

Next, the behaviors of the plugins could be configured or controlled
using configuration files, which are “cascaded”. A config file can
include others and those can in turn include others.

This is an example of a project-specific project-qt.ini main config
file. It’s the first file read when the project-specific client is
launched. Some settings are abbreviated:

[DEFAULT] includes = %(checkbox_oem_share)s/configs/checkbox-project-base-qt.ini %(checkbox_project_share)s/configs/checkbox-project-base.ini

[checkbox/plugins/environment_info] repositories = deb http://.*\(archive\|security\).ubuntu.com/ubuntu precise-security
routers = multiple
server_iperf = 10.20.30.40
sources_list = /etc/apt/sources.list
wpa_n_psk = password
wpa_n_ssid = access-point

[checkbox/plugins/user_interface] title = My project System Testing

Notice the includes line, this instructs it to load the config file for
checkbox-project-base-qt and checkbox-project-base. Check‐
box-project-base-qt loads the configs for checkbox-certification and
checkbox-project. Settings are cascaded so the config options near the
top override the ones near the bottom.

Finally, the “binary” used to invoke checkbox is a shell script that
defines where to find the things checkbox needs to run: you can define
a share directory, a specific data directory, point to a configuration
file and define some environment variables that you may need during
testing. Here’s an example for checkbox-project-qt:

#!/bin/bash
export CHECKBOX_DATA=${CHECKBOX_DATA:-~/.checkbox}
export CHECKBOX_SHARE=${CHECKBOX_SHARE:-/usr/share/checkbox}
export CHECKBOX_OPTIONS=${CHECKBOX_OPTIONS:—log-level=debug –log=$CHECKBOX_DATA/checkbox-project.log}
export CHECKBOX_CERTIFICATION_SHARE=${CHECKBOX_CERTIFICATION_SHARE:-/usr/share/checkbox-certification}
export CHECKBOX_OEM_SHARE=${CHECKBOX_PROJECT_BASE_SHARE:-/usr/share/checkbox-project-base}
export CHECKBOX_PROJECT_SHARE=${CHECKBOX_PROJECT_SHARE:-/usr/share/checkbox-project}

# Convenience for defining the PYTHONPATH directory.
if [ “$CHECKBOX_SHARE” != “/usr/share/checkbox” ]; then
export PYTHONPATH=”$CHECKBOX_SHARE:$PYTHONPATH”
fi

python3 $CHECKBOX_SHARE/run “$@” $CHECKBOX_PROJECT_SHARE/configs/$(basename $0).ini

Here you can see that it defines some locations and an important part
is the final python3 line, where it will locate and use the required
.ini config file we saw earlier.

This hierarchical organization was very powerful but was also difficult
to handle, and also had some limitations. Part of the work we did with
checkbox was to integrate all the project-specific plugins into check‐
box trunk, this way all the core code is in one place, and the
project-specific variants only supply jobs, whitelists, data and con‐
figuration, without adding new behavior.

New plainbox behavior control
Unlike checkbox, plainbox’s core is monolythic, and it has no concept
of plugins. This makes it easier to understand and work with. The
plainbox core has implementations for all the functions in the old
checkbox packages, so no additions are necessary to use features such
as submission to certification or report generation.

What we call plainbox is the library that implements all the function‐
ality, as can be seen here.

Plainbox provides tools to help test developers write and package
tests. These are delivered in “providers”, which are entities designed
to encapsulate test descriptions, custom scripts for testing,
whitelists and assorted data. They were designed to allow teams to
write and deliver their custom tests without worrying too much about
the underlying plainbox code.

To get information on how to write tests and providers, see the
Provider Tutorial

However, when actually using these tests to verify a real system, we
wanted to provide something easier and closer to the user experience of
checkbox. We created two clients, checkbox-gui and checkbox-cli, which
had some hardcoded behaviors, and we also started creating other
clients which were based on these but were purpose specific. For
instance, we had a version of checkbox for SRU testing, another for
server certification, and so on.

But then we realized that a lot of the code was duplicated and the
behaviors were common except for a few changes. So we came up with the
concept of “launchers”, which are somewhat similar to checkbox’s con‐
figuration files and shell script launchers.

The idea is that checkbox-gui and checkbox-cli have some very basic
behaviors, since they are the clients that are shipped by default with
ubuntu. They can show all the available whitelists, show a predefined
welcome message, and at the end will let the user see the html report
and submit it to launchpad using their e-mail address, similar to the
version of checkbox that shipped with Ubuntu.

Instead of using complicated command line switches, launchers allow you
to configure some optional behaviors to customize your testing experi‐
ence. A launcher contains settings, and is similar to a shell script,
but the interpreter will be either checkbox-gui or checkbox-launcher.

Here are a few examples of what can be done with launchers.

As a surprise, checkbox-cli is itself a launcher:

#!/usr/bin/env checkbox-launcher
[welcome] text = Welcome to System Testing!
Checkbox provides tests to confirm that your system is working properly.
Once you are finished running the tests, you can view a summary report for
your system.
Warning: Some tests could cause your system to freeze or become
unresponsive. Please save all your work and close all other running
applications before beginning the testing process.

[suite] whitelist_filter = ^default$
whitelist_selection = ^default$
skip_whitelist_selection = True

[transport] submit_to = launchpad

You can see here we customize a few options: it shows a welcome mes‐
sage, automatically selects the default whitelist, and will submit to
launchpad when it’s finished.

A graphical launcher example is canonical-certification-client.

#!/usr/bin/checkbox-gui

[welcome] title = “System Certification”
text = “

Welcome to System Certification!

This application will
gather information from your system. Then you will be asked manual tests to
confirm that the system is working properly. Finally, you will be asked for
the Secure ID of the computer to submit the information to the certification
database.

To learn how to create or locate the Secure ID,
please see here: certification.canonical.com

[suite] whitelist_filter = “^client-(cert|selftest).*”

[submission] input_type = “regex”
input_placeholder = “Secure ID (15 or 18 characters)”
ok_btn_text = “Submit Results”
submit_to_hexr = “true”

[exporter] xml_export_path = “/tmp/submission.xml”

[transport] submit_to = “certification”

Graphical launchers are a bit more complicated, but essentially it’s
similar, what it allows is for you to define some parameters to custom‐
ize your testing experience.

A very simple text-mode launcher is canonical-hw-collection which just
runs the basic hardware information tests and uploads them to a hard‐
ware database:

[welcome] title = Gathering hardware information
text = Gathering hardware information. You may be prompted for your password.
This process will take approximately 30 seconds and you will be provided
with a URL through which you can confirm and register your hardware
submission.

[suite] whitelist_filter = ^hwsubmit$
whitelist_selection = ^hwsubmit$
skip_whitelist_selection = True
skip_test_selection = True

[submission] # A bogus secure_id ensures we don’t ask it
# It can always be overridden in the .conf file.
secure_id = 000

[transport] submit_to = certification
submit_url = https://hardware-server.example.com/

FInally, canonical-driver-test-suite provides both a graphical and a
text mode launcher, which are functionally equivalent:

#!/usr/bin/checkbox-gui

[welcome] title = “Canonical Driver Test Suite”
text = “

Welcome to the Canonical Driver Test Suite.

This program contains automated and manual tests to help you discover
issues that will arise when running your device drivers on Ubuntu.

This application will step the user through these tests in a
predetermined order and automatically collect both system information as
well as test results. It will also prompt the user for input when manual
testing is required.

The run time for the tests is determined by which tests you decide to
execute. The user will have the opportunity to customize the test run to
accommodate the driver and the amount of time available for testing.

To begin, simply click the Continue button below and follow the onscreen
instructions.

[suite] whitelist_filter = “^ihv-.*”

[submission] ok_btn_text = “Save Results”
input_type = “none”

[exporter] xml_export_path = “”

[transport] submit_to = “local”

Text mode:

#!/usr/bin/env checkbox-launcher
[welcome] text = Welcome to the Canonical Driver Test Suite
This program contains automated and manual tests to help you discover
issues that will arise when running your device drivers on Ubuntu.
This application will step the user through these tests in a
predetermined order and automatically collect both system information as
well as test results. It will also prompt the user for input when manual
testing is required.
The run time for the tests is determined by which tests you decide to
execute. The user will have the opportunity to customize the test run to
accommodate the driver and the amount of time available for testing.
To begin, simply click the Continue button below and follow the onscreen
instructions.

[suite] # Whitelist(s) displayed in the suite selection screen
whitelist_filter = ^ihv-.*
# Whitelist_selection is mandatory so we set it to a bogus value so
# no whitelists are preselected.
whitelist_selection = bogus

CHECKBOX RELEASE PROCESS
This page describes the necessary steps for releasing versions of
Checkbox and Checkbox Certification to the stable PPA belonging to the
Hardware Certification team, on a regular basis. Throughout this docu‐
ment the term ‘Checkbox’ is used as a catch-all term to cover all ver‐
sions of Checkbox owned by the Hardware Certification team, currently
Checkbox itself and the Checkbox Certification extensions.

Overview
Currently the process runs on a bi-weekly cadence, with a new release
of Checkbox every two weeks. This covers ten working days, and the
tasks carried out on each day or group of days is described below:

· Days 1-4: Time allowed for new changes to be introduced into trunk.

· Day 5: Changes are merged from the trunk of lp:checkbox and lp:check‐
box-certification to their respective release branches. The changel‐
ogs for both are bumped at this point and revisions are tagged. At
this stage it may also be necessary to copy the package ‘fwts’ from
the FWTS Stable PPA to the Checkbox Release Testing PPA
.

· Days 6-9: Testing is performed by the release manager for the Hard‐
ware Certification team, and a representative of the CE QA team (the
main customer for Checkbox within Canonical)

· Day 9: A release meeting is held between the release manager for the
Hardware Certification team and the representative of the CE QA team.
Potential issues with the release are identified and plans made to
address them.

· Day 10: The tested version of Checkbox is copied to the stable PPA.

Launchpad Branches
The release process requires separate branches in Launchpad containing
a semi-frozen version of the code that was in trunk on day 5 of the
process. This is so that development can continue on trunk without
jeopardising the stability of the to-be released version of Checkbox.
The relationship between all branches involved in the process is as
shown below:

· lp:checkbox/release <- lp:checkbox · lp:checkbox-certification/release <- lp:checkbox-certification · lp:~checkbox-dev/checkbox/checkbox-packaging-release <- lp:~check‐ box-dev/checkbox/checkbox-packaging Auditing milestoned bugs Prior to creating the release candidate the release manager should review the list of bugs milestoned for the next release of Checkbox. They should visit checkbox milestones and locate the milestone dated with the
release date.

· For bugs that are set to In Progress with a branch associated – liase
with the branch owner to see if the merge can be completed before the
deadline.

· For bugs that are in any other non-closed status (except Fix Com‐
mited) – re-milestone them to the following milestone.

Cutting the release
In order to cut the release, we have to merge the changes from trunk
into the release branch, commit them with a suitable message and update
the changelog in trunk so that future changes go under the correct ver‐
sion. For each combination of branches shown above, do the following
(the example uses lp:checkbox and lp:checkbox/release):

bzr branch lp:checkbox/release checkbox-release
bzr branch lp:checkbox checkbox-trunk
cd checkbox-release
current_stable=`head -n1 $(find . -name ‘changelog’) | grep -oP ‘(?<=\().*(?=\))'` bzr merge lp:checkbox at this point if no changes (other than one to debian/changelog) get merged in then we do not perform a release of the package in question. In practice this often happens with checkbox-certification but never with checkbox: bzr commit -m "Merged in changes from rev$(bzr revno -r tag:$current_stable lp:checkbox) to rev$(bzr revno lp:checkbox) from lp:checkbox" bzr push lp:checkbox/release cd `find . -name 'debian'`; cd .. bzr tag `head -n1 debian/changelog | grep -oP '(?<=\().*(?=\))'` dch -r (save modified changelog) dch -i -U 'Incremented changelog' debcommit bzr push lp:checkbox The last step in the process is to perform a build of the packages in the ppa:checkbox-dev/testing PPA. To do this we need to go to the recipe pages for the checkbox and/or checkbox-certification release branches. · checkbox-testing recipe

· checkbox-certification-testing recipe

The Build Now option should be available on the page. Click it to start
a build.

Copying Firmware Test Suite to the Testing PPA
The Firmware Test Suite tool is a test tool for system firmware that is
naturally heavily utilised by Checkbox. To make sure the latest version
which contains fixes and new tests/features needed by Checkbox is
available and also doesn’t break anything in Checkbox, we need to
release it alongside Checkbox. After cutting the release if the
Firmware Testing team have notified that a new version is available and
that this version should be used for certification, we need to copy it
to the Testing PPA. To do this we need to go to the Copy packages view
of the Firmware Test Suite (Stable) PPA
and select the ‘fwts’ packages for all releases back to Precise. We
need to set the ‘Destination PPA’ as ‘Checkbox Release Testing [~check‐
box-dev/testing]’ and the ‘Copy options’ field to ‘Copy existing bina‐
ries’, then click ‘Copy packages’. This step then needs to be repeated
but set the ‘Destination PPA’ field to ‘PPA for Checkbox Developers
[~checkbox-dev/ppa]’.

Next Release of Checkbox e-mail
So that everyone has the opportunity to perform whatever testing is
required in a timely manner, after the PPA builds have been completed
an email should be sent to the following mailing lists:

· hardware-certification-team@lists.canonical.com

· commercial-engineering@lists.canonical.com

The content is typically something like this:

Subject: Next Release of Checkbox (18/11/2013)

Hi,

The next release of Checkbox is available in the
//code.launchpad.net/~checkbox-dev/+archive/testing PPA.
Please test it at your convenience. Checkbox is based on revision 2484 of
lp:checkbox and Checkbox Certification is based on revision 586 of
lp:checkbox-certification.

Thanks,

If one or the other of Checkbox and Checkbox Certification have not
been updated then there is no need to mention that package

Testing the release
Now that the release has been cut, testing should take place prior to
the release meeting. From the point of view of the certification team,
what needs to be tested is checkbox-certification-client and check‐
box-certification-server which form the basis for CE QAs OEM specific
versions of Checkbox. Checkbox certification server is tested in the CI
loop Checkbox certification client needs to be tested manually.

Release Meeting
On the Thursday before the release is made, a meeting is held between a
representative of the Certification team and a representative of the
Commercial Engineering QA team. The meeting is held at 7:30 UTC as
shown in this calendar invite . An agenda for the meeting is included in the
invite.

Publishing the release
To publish the release we simply need to copy a number of packages from
the Checkbox Release Testing PPA to the Hardware Certification Public PPA
. To do
this we go to the Copy packages view of the Checkbox Release Testing
PPA and select all versions of the following list of packages: check‐
box, checkbox-certification, fwts. Make sure that the ‘Destination PPA’
field is set to ‘Public PPA for Hardware Certification [~hardware-cer‐
tification/public]’ and that the ‘Copy options’ field is set to ‘Copy
existing binaries’, then click ‘Copy Packages’.

After that is done an announcement email should be sent to
commercial-engineering@lists.canonical.com . A template for the announcement in included
below:

Hi,

A new release of checkbox has been uploaded to the Hardware
Certification Public PPA
(https://launchpad.net/~hardware-certification/+archive/public). The
release is based on revision 2294 of lp:checkbox

Thanks,

Please attach the most recent part of the changelog as release notes

· genindex

· modindex

· search

AUTHOR

Zygmunt Krynicki

COPRYRIGHT

2013-2014, Zygmunt Krynicki

0.23 January 11, 2016 CHECKBOX_NG(1)