Documentation/compat

From Driver Backports Wiki
Jump to: navigation, search

This is now deprecated use hacking on backports.

This page is dedicated to compat, a generic Linux kernel backport compatibility module that has been brewing since 2009. The compat module provides functionality introduced in newer kernels to older kernels through a set of header files and exported symbols. The compat module allows code from newer kernel releases to be used on older kernels without modifications with a few exceptions. Exceptions to achieve automatically backporting kernel code through the compat module are dealt with through specific mechanisms documented in the compat-drivers project page and most recently through the slides used at the 2011 Linux Plumbers presentation on automatically backporting the kernel and its follow up presentation at the 2012 Linux Collaboration summit in San Francisco.

By using compat and frameworks such as compat-drivers, and techniques to integrate additional patches into upstream releases that prefer and push for upstream, we hope to ensure developers can prioritize working directly upstream on the Linux kernel and provide deliverables to end users and customers for any target kernel.

compat-drivers today automatically backports the Wireless, Bluetooth and Ethernet drivers directly from the Linux kernel on a daily basis by using linux-next.git. We also make stable backport releases by using the linux-stable.git tree. We are considering expanding this framework to automatically backport more drivers -- patches are welcomed, otherwise, you'll have to wait until someone decides to integrate other subsystems.

Contents

Supported kernels

The goal is to backport new kernel functionality at least down to the oldest stable supported kernel listed on kernel.org, but backporting code further is certainly welcomed. Currently this means supporting backporting down to the 2.6.27 Linux kernel release. Although our goal is to at least support backport efforts down to 2.6.27 the compat module currently has code to support even older kernels, and we welcome code to support older kernels as well.

Mailing list

See the Backports mailing list page.

Presentations

Metrics explaining the amount of code saved by automatically backporting the kernel using compat and current research are explained.

 * 2011 Linux plumbers kernel backport presentation (slides)
 * follow up presentation Automatically backporting the Linux kernel - 2012 Linux Collaboration summit

Estimating deliverables

Once you use compat and kernel framework integrators such as compat-drivers you should have no excuse to work outside of the upstream Linux kernel. In order to help guestimate when your upstream code will be available through a stable release such as the compat-drivers stable releases you can rely on the phb-crystall-ball.

Tracking down when a feature went upstream

The compat module expect you to put functionality into either header files for defines and static inlines or into c files. This functionality must be tucked away under a file for the kernel release that introduced the functionality. To help aid you with your scavenger hunt on tracing down when a specific functionality was introduced into the upstream Linux kernel you can use several techniques. This section documents those techniques.

Git tree setup for backporting

In order to understand how to track a feature in the Linux kernel you must first understand how the development of the Linux kernel happens. Below we provide a basic description of how this works.

Greg Kroah-Hartman maintained the stable series of the Linux kernel. Linus hands off kernel maintainership to Greg after he blesses the first release of a stable kernel. Greg maintains the stable kernel via through the linux-stable.git tree. Each stable release of the kernel has a respective origin/linux-3.x.y branch, where x is the current release number. The "y" here is literal, so for the 3.2 release there is an origin/linux-3.2.y branch that keeps track of the 3.2.1, 3.2.2 and future extra version releases of the 3.2 release. You can use the linux-stable.git tree to verify if code is present on a stable release of a kernel.

Linus maintains the release candidates of the Linux kernel (RC) up to the point he blesses away the first release of a kernel release. So for example Linus maintained the 3.2 kernel from the inception of 3.2-r1 down to 3.2-rc7. During the 3.2-rc1 to 3.2-rc7 release Linus is supposed to only accept regression fixes. After that he releases the first blessed release of 3.2 and handed off maintainership to Greg. All the magic that went into 3.2-rc1 though happens in what we call the development cycle of the Linux kernel. The development cycle of the Linux kernel varies subsystem to subsystem but the idea can be best described in terms of the usage of two subsystem git trees. Subsystem maintainers typically have to maintain two git trees, a subsystem.git tree and a subsystem-next.git tree. The subsystem.git tree contains code that should go into Linus' tree during the regression testing of the RC release of the kernel. The subsystem-next.git contains code is being queued up for the next release of the Linux kernel, hence the "-next" postfix. All code in the subsystem-next.git trees is ephemeral, that is, what you see there today does not mean that it will be present tomorrow, it is code that is simply accepted on that day by the subsystem maintainer, but a lot of it can simply be dropped, and in fact Linus can reject merging it eventually. Linus will start merging all subsystem-next.git trees into his own tree during what we call, the merge window. This typically consists of a 2-3 week window period. For example the merge window for the 3.2 kernel started after Linus released the v3.1 kernel release on Mon Oct 24, 2011, and ended when Linus released the v3.2-rc1 release on Mon Nov 7, 2011, the merge window in this case lasted exactly 2 weeks.

The merge window can often be painful though and to aid with this Stephen Rothwell has decided to take up merging all subsystem-next.git trees into one tree, linux-next.git on a daily basis. He rants to each subsystem maintainer on any issues during his merge, including build breakage and general insanity.

The implications here are that we can not only track code that is already released, but we can also predict what likely will go into the next release of the Linux kernel by using linux-next.git. This means we can backport the Linux kernel proactively through every subsystem development cycle by using linux-next.git.

If this is still a bit fuzzy you can review the wireless development process for a review of how that specific subsystem development works. The development process there even includes a shiny diagram.

Using git grep to identify what file adds new functionality

First you want to identify what file added the functionality you seek to backport. Suppose you want to identify what file adds a new function, foobar(). To do this you can use:

git grep foobar\(

to identify where foobar() is used in the entire Linux kernel. git grep will use the indexed data and at times can often run faster than a regular grep, as it will also skip object file in a regular grep search. Given that the point of compat is to backport functionality and that this functionality is typically shared between modules and subsystems, you can rest assured most functionality has a declaration in the header files of the Linux kernel. You can typically save yourself git grep time by starting to search through the include/ directory of the Linux kernel.

git grep foobar\( include/

Using git blame to identify introduction of a commit

Typically compat will have a lot of features already backported for older kernels, so to help backport new functionality you should be using linux-next.git. To identify at what point a specific feature was added into the Linux kernel you can use git blame to identify the sha1sum of the commit that added the functionality you want to backport. At times though such functionality may have not been introduced via that commit but instead it may have simply been moved from one place to another. You can use

git blame sha1sum~1

to review if the functionality was present before the last commit you identified added that functionality to a file.

Using git describe to identify where an commit was introduced

Once you have identified the commit that introduced a specific functionality you can use git describe --contains sha1sum to give you the first commit tag that contains the commit. You can use the base of the tag to help you figure out the first kernel release that added the commit.

For example, commit 600715dcdf567c86f8b2c6173fcfb4b873e25a19, or short, 600715dc added to the Linux kernel the CONFIG_PHYS_ADDR_T_64BIT feature. We can determine that it was added on the v2.6.28 release by reviewing the results of using

git describe --contains 600715dc

which yields: v2.6.28-rc1~271^2^4~6. This means that in order to backport 600715dc we can edit two files on compat:

  • include/linux/compat-2.6.28.h
  • compat/compat-2.6.28.c

Since this functionality can be dealt with header files the c file is not touched to backport this functionality. You can review the respective compat CONFIG_PHYS_ADDR_T_64BIT backport via gitweb.

Testing across kernels

The goal behind compat is to add features to provide backport but to older kernels but to do this successfully we must also ensure that each new patch added does not break building against a supported kernel. It may be a daunting task to get all supported kernels to and also to build against then, which is why we have started adding support to do all of this for you. Under the bin/ directory you will find two scripts which we document below:

  • get-compat-kernels
  • ckmake

get-compat-kernels

You can run this script to install all known supported kernels. Currently this will only work for Ubuntu, but work is being considered to standardize on the same kernels for any Linux distribution.

ckmake

You can run this to test compiling compat against supported kernels available. Although initially designed to support Ubuntu only the goal is to make this Linux distribution agnostic.

compat modules

compat.git provides a few modules and headers to help with general kernel compatibility.

compat module

Provides all exported symbols implemented in each respective kernel compat-2.6.xy.c files. Upon module load it just initializes the Linux kernel's power management Quality Of Service (aka pm-qos) Interface interface added as of the 2.6.24 kernel. No other things are initialized, the rest of the compat module just acts as a library of exported symbols.

compat_firmware_class module

Another module which compat.git provides is a backport of the firmware_class module which got updated recently newer with a new request_firmware_nowait() to allow better asynchronous firmware uploading. This was added as of the 2.6.33 kernel. The firmware_class module has been backported into a new module called compat_firmware_class. A separate module has been defined instead of a direct replacement for firmware_class since your system may have old drivers which use the old request_firmware_nowait() and would bust if they used the new request_firmware_nowait(). The compat_firmware_class module registers its own sysfs subsystem and as such also gets udev events sent through a separate subsystem. Because of this a new udev rules file is required and provided.

Research

Active research is ongoing which could enhance automatically backporting the Linux kernel further. For details please read up on:

 * Jesper Andersen & Julia L. Lawall - Generic Patch Inference
 * Coccinelle spatch

Consider one set of diffs that can be read by spdiff to produce an spatch. This could in theory allow us to backport all functionality on a kernel if only two or three sets of files were patched and interpreted by spdiff.

Contributions

Contributions to compat follow the same mechanisms as used in the Linux kernel, this means you should provide as Singed-off-by tag as documented on the Developer's Certificate of Origin 1.1.

Submitting patches

compat and compat-drivers contributions follow the contribution model implemented by the Linux kernel. Patches or pull requests for compat and compat-drivers must have be signed-offed. If you don't sign off on them they will not accepted. This means adding a line that says "Signed-off-by: Name email" at the end of each commit, indicating that you wrote the code and have the right to pass it on as an open source patch. For exact definition of what the Signed-off-by tag is you can read the definition of the "Developer's Certificate of Origin 1.1", which you can read here:

http://gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html

You can send patches as follows:

To: mcgrof@kernel.org
Cc: backports@vger.kernel.org
Subject: compat: foo

License

The compat module and code is all licensed under the same license as the Linux kernel and is considered a complete derivative work of the Linux kernel. For more information refer to the COPYRIGHT file.

Personal tools