This document describes the basic workflow to follow, after you've made a change to the Linux Kernel in the Chromium OS sources, to submit a your changes to the Chromium OS repository, and to submit your changes upstream to the official Linux Kernel repository.

What commit message should I use?

See the Kernel Design page for some more details.

Code Changes

For changes which cannot be submitted upstream to the official Linux Kernel repository, the commit message is important.  We use the following conventions:

• Begin the commit message with CHROMIUM:
• If it is architecture specific, add the architecture.  The following are samples of supported architectures: ARM: or X86:
• If it is machine specific, add machine-identifying information.  For example, tegra2: or x86-mario:.
• Follow the needed tags with the subject for the commit message.
• Follow the subject line with the body of the commit message.  The message should not only describe what, but also why, you have created the change.  Please include information about the testing that you performed to ensure the code is valid.
• Signed-off-by is required, and our gerrit server is a bit picky about the order.  It appears to require this line immediately before the Change-Id line if present.

An example subject line is: CHROMIUM: ARM: tegra: Add initial support for aebl

Do not include configuration changes (i.e. changes to files within chromeos/config) with other code changes. See the next section for these.

(Note: git send-email requires git-email to be installed on your host ('sudo apt-get install git-email'),
or you will get the message "git: 'send-email' is not a git command. See 'git --help'.".
You also need to configure .gitconfig to use your SMTP server)

If you are sending a series of patches it is nice to include a cover letter. This turns up as patch zero in the series. Pass the --cover-letter flag to 'git format-patch' and it will create a 0000-subject file which you can edit to contain your cover letter. When you use 'git send-email' you can send files 000* to send the cover letter and all your patches as one email set.

Another flow that might work is to send email directly, without going through 'git format-patch'. For example you can email the top five commits to the mailing list with something like:

The --annotate lets you edit them before they go out, which is probably a good idea in this case!

When replying to an email thread with an updated patch, use the something like the following to attach your email to the thread:

You can find the message id under the label <Message-Id> in gmail in the 'Show Original' link in the drop down options for the email you want to reply to.

There is a video here: http://www.youtube.com/watch?v=LLBrBBImJt4

The patch flow throughout the video is:

1. git diff
2. git commit
3. git show
4. git format-patch
5. git send-email

Patch checklist: (at 34:30 of the video)

1. Kernel builds with patch applied
2. Correct "From:" address
3. Concise "Subject:"
4. Explain the patch
5. Signed-off-by
6. Check you have removed Change ID, TEST= and BUG= from the commit message

Which copyright header should I use?

When adding new files to the kernel, please add a regular Google copyright header to them. In particular this is true for any code that will eventually find its way upstream (which should include practically everything we do).

The main reason for this is that there's no concept of "The Chromium OS Authors" outside of our project, since it refers to the AUTHORS file that isn't bundled with the kernel.

How do I check my patches are correct?

There are two aspects of having correct patches to send upstream: not having Chromium OS-specific details, and meeting all the Linux kernel requirements.

For the following sections, you will need to have created a patch file using git format-patch.  Also note that you will have to recreate the patch file, and re-check your patch file each time you check in code to your source tree.

Remove Chromium OS-specific Details

Verifying these details is as simple as loading the patch file in your favorite editor.  Edit the file manually to become compliant; this will, of course, have no affect on the source or commit message stored by git.

Once all of the above is true, you can move on to checking for compliance with the Linux Kernel guidelines.

Check for Compliance with Linux Kernel Requirements

You should use this perl script to check that your patch conforms to the kernel coding standard. It is kept in the linux kernel tree.

Automating the Compliance Checks

This script might be useful also, as it checks a series of patches, checks for Chrome OS-specific commit tags and prints a summary at the end. Put it in your path and run it from anywhere.

How do I backport an upstream patch?

Let's suppose you've spotted a juicy new commit in Linus's upstream linux kernel tree that you just must have.
Instead of creating a new branch and manually applying the changes, use git cherry-pick to do it for you.
In addition, the repository maintainers appreciate it if the cherry-picked commit still contains the original author and git hash of the original upstream commit.
Use git cherry-pick -x to do this automatically:

First, add Linus's tree as a remote to the chromium-os kernel tree (assuming the chromium-os root is ~/chromiumos):

This will take a little while as git fetches all upstream commits.  Luckily, git is smart and won't refetch commits already in the chromium-os tree.

We can view that juicy commit using its upstream hash:

To backport the commit to the chromium-os tree, first start a new branch from the current Tip of Tree (ToT).
Then cherry-pick with -x to preserve the original author and hash, and -s to sign-off-by the commit:

Add TEST= and BUG= lines at the bottom of the patch description. Also, remember to keep the patch subject intact with only an addition of UPSTREAM: or BACKPORT: as a new prefix.  Use UPSTREAM: if you are applying an upstream patch as-is, or BACKPORT: if you had to change the code to make it run with an older kernel version.

Now, the upstream commit is on its own branch, let's upload it to gerrit, like usual:

This will generate a gerrit change for review.

After review, submit the patch in gerrit like usual.

Kernel Configuration

Kernel configuration in Chromium OS has an extra level of indirection from the normal .config file. So do the instructions - see this page for more information.

How to quickly test kernel modifications (the fastest way)

Please take a look at doc on network-based development. While setting up your environment might appear to be harder and more time consuming, in many cases it will allow to test kernel modifications much faster and easier than the ways described below.

How to quickly test kernel modifications (the fast way)

Do an incremental build of the kernel:

The 3.4 kernel is just called chromeos-kernel.

Update the kernel on the target:

Note that using cros_workon_make leaves build artifacts in your source directory under the "build" directory.  When you do a regular emerge of the kernel (and are cros_work'ed on) this will slow things down because the entire source directory gets copied.  So delete the "build" directory when you're done.

Note: Please ensure that verity is disabled on the target before running the update_kernel.sh script, otherwise the script won't be able to copy over kernel modules and the target will be rebooted with just the kernel image updated. Since the network drivers are built as modules, this leaves machine in a state where there is no way to connect to the network after the reboot. Verity can be disabled using the command "/usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --partition <partition number>" on the target followed by a reboot.

Dealing with a bad kernel installation

One problem with this fast approach is that it requires an already installed and booted target system.  If you update with a bad kernel so that it no longer boots, this approach is no longer available. The system is generally recoverable by booting physical media (USB stick or SD card) and copying its kernel blob over your kernel partition:

Dealing with partition corruption due to bad kernel recovery

One time I really screwed up my system by recovering (after bad kernel installation) with 'dd if=/dev/sdb of=/dev/sda'.  I forgot the '2' after each drive specification.
This overwrote my internal partition table with an exact copy of the USB stick's partition table, including the GUIDs.
When I subsequently tried to boot USB, the system always seemed to boot off the internal disk.  'rootdev -s' reported (internal partition) /dev/sda3.
After an hour or so, consultation with Bill showed that I really was booting the kernel from /dev/sda2, but the kernel found the matching GUID on sda before even looking at sdb.
This was recovered with:

which generates and installs a new GUID for sda3.

Dealing with issues - preparing the environment

Below are described some problems you might encounter. If instructions above work, you might skip them. We assume that you want to boot using the most recently built image.

First, to prepare for other steps:

SSHD keys

If sshd on the target machine complains about missing keys:

Public key authorization

update_kernel.sh uses for authorization keys that, depending on your configuration, might not be present in your image. If that's the case, you will be prompted for password during script execution. To fix it, run the following commands in your image directory:

iptables configuration

Iptables - again, depending on your configuration - might be configured to refuse all the incoming connections, in which case update_kernel.sh will be unable to ssh to your target machine. If you encounter this problem, to fix it:

1. Again mount the root filesystem:
   
   (chroot) $ sudo mount -o loop part_3 rootfs_dir/
2. Edit file rootfs_dir/etc/init/iptables.conf. Find the following line:
   
   iptables -P INPUT DROP
3. Change it to:
   
   iptables -P INPUT ACCEPT
4. Save and unmount the filesystem:
   
   (chroot) $ sudo umount rootfs_dir/

Dealing with issues - cleaning up

To build new image after modifications to one or more of the partitions, simply run:

How to test kernel modifications (the slow way)

Note: there is more information (possibly more useful too) in the disk format document, and more specifically here.

Check out the tree somewhere as usual, make the chroot, build packages, build image, blah blah blah. Create a bootable USB key from that image. We'll modify that key with our testing kernel.

At this point you need to cros_workon the kernel (and clone the kernel tree in case you used mini-layout). See the big picture and instructions in Chromium OS Developer Guide, but as a quick reference you are expected to run the following inside chroot:

Then, still inside chroot, run this:

Where

Renaming of the .git directory for the duration for the build is required to prevent mangling the module path by the kernel make. The

make will produce ${BUILD_DIR}/arch/<your_target_arch>/boot/{bzImage|uImage}, which is the kernel image you want to try. The next step varies depending on whether your hardware has an EFI BIOS, legacy BIOS or u-boot. You can ether copy the kernel to your USB stick and tell the bootloader to use your new kernel, possibly with extra debugging arguments, or use netboot/NFS for u-boot equipped targets (see network_based_development for details).

If you need your module to be present on the target, you can scp it from the build location to your target (provided your target is set for ssh access and allows chronos account login).

Testing with an EFI BIOS

Copy the new bzImage file into the /efi/boot/ directory on your USB key's partition 12. The /efi/boot/grub.cfg file will look for the kernel called vmlinuz, but you can edit that config file to add a line to look for your test kernel too. For example, here's my USB key's partition 12:

When the USB key boots, I'll see a menu that lets me select which boot path to use.

Testing with a legacy BIOS

Copy the new bzImage file into the /boot directory on your USB key's partition 3. The /boot/extlinux.conf file will look for the kernel called vmlinuz, but you can edit that config file to add a line to look for your test kernel too. For example, here's my USB key's partition 3:

When the USB key boots, I can hit TAB to see the list of boot choices, and can pick the one I want by entering the label.

Debugging messages

With either bootloader, you can debug early kernel failures by increasing the verbosity and location of kernel debug messages. You can modify the config files without rebuilding anything. The default boot args have this:

Using args like these instead may be helpful:

Working on several kernel issues

git supports multiple branches coexisting in the same directory tree, and kernel make system supports placing the kernel build output in a separate directory (using the O=<path> make command line parameter).

To create separate builds get per kernel git branch, while in the cloned kernel source tree root create a build directory for your current branch, for instance:

and then just add O=../../build/<branch_name> to make invocations described above. Or use the following bash script to take care of all make command line parameters other than make targets:

Modifying H2C Bios kernel command line

Place kernel blob into a file  (<original_kernel>), either using dd on the target or by dismantling chromiumos_image.bin generated by build_image. Store the desired kernel command line in a file <new_cmd_line> and then use the following to change the kernel command line:

Installing onto SSD

Instead of booting the kernel from USB as described above, it can be installed directly on the SSD of the target device.  With modern H2C Bios, this requires signing the blob with the development key and booting with the target machine's development mode switch set appropriately.  Also, since there are two kernel/root partition pairs in our partition scheme, we need to select which one we want to use.  Usually we stay with the current pair.

To sign with the devkey as per the Disk Format doc http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format#TOC-Quick-development:

Transfer new_kern.bin to the target system.  I prefer scp, but it can be placed on USB stick as well.

Identify the preferred kernel partition.  This will be either sda2 or sda4.  rootdev -s will identify the root partition, and that can be used to identify the currently booted kernel partition.

Copy the image to the partition.

dev_debug_vboot can be used to verify the kernel partition has a properly signed image.  Indeed, it will actually tell you in what modes (ie, development, recovery, neither) your kernel will boot.

cgpt can be used as an alternative to rootdev above to find the currently preferred kernel partition.

Getting a Kernel Trace

Run the following commands on the target. This needs to be done just once after an install.

The crashes will then appear in /var/spool/crash.

Loading Kernel modules from outside the root filesystem

If you need to load kernel modules from a location other than the root filesystem, module locking must be disabled. Either a kernel command line option can be used:

Or, on images with dm-verity disabled (--noenable_rootfs_verification), the restriction can be disabled via the exposed sysctl:

Blacklisting Kernel modules for individual overlays

If you need to blacklist kernel modules for specific overlays. Modify the overlay-<name>/chromeos-base/chromeos-bsp-<name>/chromeos-bsp-<name>-<version>.ebuild file.

Add the following two lines to the end of the src_install() function:

The ${FILESDIR} variable points to the files/ directory within the chromeos-bsp-<name>/ directory. Within this directory, add your <blacklist file> (ex cros-blacklist.conf).

For each kernel module you wish to blacklist, add the following line to <blacklist file>:

You can also use # comments within these files to explain why the kernel module needs to be blacklisted.

Building and installing kernel-next on a specific overlay