View source for Documentation/compat

This is now deprecated use [[Documentation/backports/hacking|hacking on backports]].

This page is dedicated to compat, a generic Linux kernel backport compatibility module that has been brewing since 2009. The compat module provides functionality introduced in newer kernels to older kernels through a set of header files and exported symbols. The compat module allows code from newer kernel releases to be used on older kernels without modifications with a few exceptions. Exceptions to achieve automatically backporting kernel code through the compat module are dealt with through specific mechanisms documented in the [[compat-drivers|compat-drivers project page]] and most recently through the [http://www.linuxplumbersconf.com/2011/ocw/system/presentations/771/original/kernel-backport-for-good.odp slides used at the 2011 Linux Plumbers presentation on automatically backporting the kernel] and its [https://docs.google.com/presentation/d/1axVNEGwKZjnzG1ocdd289WMqPxzJ3qfMv70ghGcnUKc/edit follow up presentation] at the 2012 Linux Collaboration summit in San Francisco.

By using compat and frameworks such as [[compat-drivers|compat-drivers]], and techniques to integrate [http://wireless.kernel.org/en/users/Download/stable/#Additional_patches_to_stable_releases additional patches] into upstream releases that prefer and push for upstream, we hope to ensure developers can <b>prioritize</b> working directly <b>upstream</b> on the Linux kernel and provide deliverables to end users and customers for <b>any</b> target kernel.

compat-drivers today automatically backports the [http://wireless.kernel.org/ Wireless], [http://www.bluez.org/ Bluetooth] and [https://www.linuxfoundation.org/collaborate/workgroups/networking/ethernet Ethernet] drivers directly from the Linux kernel on a daily basis by using [http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git;a=summary linux-next.git]. We also make [http://wireless.kernel.org/en/users/Download/stable/ stable backport releases] by using the [http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=summary linux-stable.git] tree. We are considering expanding this framework to automatically backport more drivers -- patches are welcomed, otherwise, you'll have to wait until someone decides to integrate other subsystems.

<h1>Supported kernels</h1>

The goal is to backport new kernel functionality <b>at least</b> down to the oldest stable supported kernel listed on [http://kernel.org kernel.org], but backporting code further is certainly welcomed. Currently this means supporting backporting down to the <i>2.6.27</i> Linux kernel release. Although our goal is to <b>at least</b> support backport efforts down to <i>2.6.27</i> the compat module currently has code to support even older kernels, and we welcome code to support older kernels as well.

<h1>Mailing list</h1>

See the [[mailing_list|Backports mailing list]] page.

<h1>Presentations</h1>

Metrics explaining the amount of code saved by automatically backporting the kernel using compat  and current research are explained.

  * [http://linuxplumbersconf.org/2011/ocw/proposals/771 2011 Linux plumbers kernel backport presentation] ([http://www.linuxplumbersconf.com/2011/ocw//system/presentations/771/original/kernel-backport-for-good.odp slides])
  * [https://docs.google.com/presentation/d/1axVNEGwKZjnzG1ocdd289WMqPxzJ3qfMv70ghGcnUKc/edit follow up presentation Automatically backporting the Linux kernel - 2012 Linux Collaboration summit]

<h1>Estimating deliverables</h1>

Once you use compat and kernel framework integrators such as [[compat-drivers|compat-drivers]] you should have no excuse to work outside of the <b>upstream</b> Linux kernel. In order to help <i>guestimate</i> when your <i>upstream</i> code will be available through a stable release such as the [[releases|compat-drivers stable releases]] you can rely on the [http://phb-crystal-ball.org/ phb-crystall-ball].

<h1>Tracking down when a feature went upstream</h1>

The compat module expect you to put functionality into either header files for defines and static inlines or into c files. This functionality must be tucked away under a file for the kernel release that <b>introduced</b> the functionality. To help aid you with your scavenger hunt on tracing down when a specific functionality was introduced into the upstream Linux kernel you can use several techniques. This section documents those techniques.

<h2>Git tree setup for backporting</h2>

In order to understand how to track a feature in the Linux kernel you must first understand how the development of the Linux kernel happens. Below we provide a basic description of how this works.

Greg Kroah-Hartman maintained the stable series of the Linux kernel. Linus hands off kernel maintainership to Greg after he blesses the first release of a stable kernel. Greg maintains the stable kernel via through the [http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=summary linux-stable.git] tree. Each stable release of the kernel has a respective <i>origin/linux-3.x.y</i> branch, where x is the current release number. The <i>"y"</i> here is literal, so for the <i>3.2</i> release there is an <i>origin/linux-3.2.y</i> branch that keeps track of the <i>3.2.1</i>, <i>3.2.2</i> and future extra version releases of the <i>3.2</i> release. You can use the [http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=summary linux-stable.git] tree to verify if code is present on a stable release of a kernel.

Linus maintains the release candidates of the Linux kernel (RC) up to the point he blesses away the first release of a kernel release. So for example Linus maintained the 3.2 kernel from the inception of <i>3.2-r1</i> down to <i>3.2-rc7</i>. During the <i>3.2-rc1</i> to <i>3.2-rc7</i> release Linus is <i>supposed</i> to only accept <b>regression fixes</b>. After that he releases the first blessed release of <i>3.2</i> and handed off maintainership to Greg. All the magic that went into <i>3.2-rc1</i> though happens in what we call the <b>development cycle</b> of the Linux kernel. The development cycle of the Linux kernel varies subsystem to subsystem but the idea can be best described in terms of the usage of two subsystem git trees. Subsystem maintainers typically have to maintain two git trees, a <i>subsystem.git</i> tree and a <i>subsystem-next.git</i> tree. The <i>subsystem.git</i> tree contains code that should go into Linus' tree during the <i>regression</i> testing of the <i>RC</i> release of the kernel. The <i>subsystem-next.git</i> contains code is being queued up for the next release of the Linux kernel, hence the "<i>-next</i>" postfix. All code in the <i>subsystem-next.git</i> trees is <i>ephemeral</i>, that is, what you see there today does not mean that it will be present tomorrow, it is code that is simply accepted on that day by the subsystem maintainer, but a lot of it can simply be dropped, and in fact Linus can reject merging it eventually. Linus will start merging all <i>subsystem-next.git</i> trees into his own tree during what we call, the <b>merge window</b>. This typically consists of a 2-3 week window period. For example the merge window for the 3.2 kernel started after Linus released the <i>v3.1</i> kernel release on Mon Oct 24, 2011, and ended when Linus released the <i>v3.2-rc1</i> release on Mon Nov 7, 2011, the <i>merge window</i> in this case lasted exactly 2 weeks.

The <b>merge window</b> can often be painful though and to aid with this Stephen Rothwell has decided to take up merging all <i>subsystem-next.git</i> trees into one tree, [http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git;a=summary linux-next.git] on a <i>daily basis</i>. He rants to each subsystem maintainer on any issues during his merge, including build breakage and general insanity.

The implications here are that we can not only track code that is already released, but we can also predict what <i>likely</i> will go into the next release of the Linux kernel by using [http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git;a=summary linux-next.git]. This means we can backport the Linux kernel <i>proactively</i> through every subsystem development cycle by using [http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git;a=summary linux-next.git].

If this is still a bit fuzzy you can review the [http://wireless.kernel.org/en/developers/process wireless development process] for a review of how that specific subsystem development works. The development process there even includes a shiny diagram.

<h2>Using git grep to identify what file adds new functionality</h2>

First you want to identify what file added the functionality you seek to backport. Suppose you want to identify what file adds a new function, <i>foobar()</i>. To do this you can use:

<pre>git grep foobar\(</pre>

to identify where <i>foobar()</i> is used in the entire Linux kernel. <i>git grep</i> will use the indexed data and at times can often run faster than a regular grep, as it will also skip object file in a regular grep search. Given that the point of compat is to backport functionality and that this functionality is typically shared between modules and subsystems, you can rest assured most functionality has a declaration in the header files of the Linux kernel. You can typically save yourself <i>git grep</i> time by starting to search through the <i>include/</i> directory of the Linux kernel.

<pre>git grep foobar\( include/</pre>

<h2>Using git blame to identify introduction of a commit</h2>

Typically compat will have a lot of features already backported for older kernels, so to help backport new functionality you should be using [http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git;a=summary linux-next.git]. To identify at what point a specific feature was added into the Linux kernel you can use <i>git blame</i> to identify the sha1sum of the commit that added the functionality you want to backport. At times though such functionality may have not been introduced via that commit but instead it may have simply been moved from one place to another. You can use

<pre>git blame sha1sum~1</pre>

to review if the functionality was present <i>before</i> the last commit you identified added that functionality to a file.

<h2>Using git describe to identify where an commit was introduced</h2>

Once you have identified the commit that introduced a specific functionality you can use <i>git describe --contains sha1sum</i> to give you the first commit <i>tag</i> that contains the commit. You can use the base of the <i>tag</i> to help you figure out the first kernel release that added the commit.

For example, commit <i>600715dcdf567c86f8b2c6173fcfb4b873e25a19</i>, or short, <i>600715dc</i> added to the Linux kernel the CONFIG_PHYS_ADDR_T_64BIT feature. We can determine that it was added on the v2.6.28 release by reviewing the results of using

<pre>git describe --contains 600715dc</pre>

which yields: <i>v2.6.28-rc1~271^2^4~6</i>. This means that in order to backport <i>600715dc</i> we can edit two files on compat:

<ul>
  <li>include/linux/compat-2.6.28.h</li>
  <li>compat/compat-2.6.28.c</li>
</ul>

Since this functionality can be dealt with header files the c file is not touched to backport this functionality. You can review the respective [https://github.com/mcgrof/compat/commit/4ec0edbd1cc2e6165e29f8ba466324e975e1cd73#include compat CONFIG_PHYS_ADDR_T_64BIT backport via gitweb].

<h1>Testing across kernels</h1>

The goal behind compat is to add features to provide backport but to older kernels but to do this successfully we must also ensure that each new patch added does not break building against a supported kernel. It may be a daunting task to get all supported kernels to and also to build against then, which is why we have started adding support to do all of this for you. Under the bin/ directory you will find two scripts which we document below:

<ul>
  <li>get-compat-kernels</li>
  <li>ckmake</li>
</ul>

<h2>get-compat-kernels</h2>

You can run this script to install all known supported kernels. Currently this will only work for Ubuntu, but [https://lkml.org/lkml/2012/2/6/85 work is being considered] to standardize on the same kernels for any Linux distribution.

<h2>ckmake</h2>

You can run this to test compiling compat against supported kernels available. Although initially designed to support Ubuntu only the goal is to make this Linux distribution agnostic.

<h1>compat modules</h1>

compat.git provides a few modules and headers to help with general kernel compatibility.

<h2>compat module</h2>

Provides all exported symbols implemented in each respective kernel compat-2.6.xy.c files. Upon module load it just initializes the Linux kernel's ''power management Quality Of Service'' (aka '''pm-qos''') Interface interface added as of the 2.6.24 kernel. No other things are initialized, the rest of the compat module just acts as a library of exported symbols.

<h2>compat_firmware_class module</h2>

Another module which compat.git provides is a backport of the firmware_class module which got updated recently newer with a new request_firmware_nowait() to allow better asynchronous firmware uploading. This was added as of the 2.6.33 kernel. The firmware_class module has been backported into a new module called compat_firmware_class. A separate module has been defined instead of a direct replacement for firmware_class since your system may have old drivers which use the old request_firmware_nowait() and would bust if they used the new request_firmware_nowait(). The compat_firmware_class module registers its own sysfs subsystem and as such also gets udev events sent through a separate subsystem. Because of this a new udev rules file is required and provided.

<h1>Research</h1>

Active research is ongoing which could enhance automatically backporting the Linux kernel further. For details please read up on:

  * [http://www.diku.dk/~julia/andersen_ase08.pdf Jesper Andersen & Julia L. Lawall - Generic Patch Inference]
  * [http://coccinelle.lip6.fr/ Coccinelle spatch]

Consider one set of diffs that can be read by spdiff to produce an spatch. This could in theory allow us to backport <b>all</b> functionality on a kernel if only two or three sets of files were patched and interpreted by spdiff.

<h1>Contributions</h1>

Contributions to compat follow the same mechanisms as used in the Linux kernel, this means you should provide as Singed-off-by tag as documented on the [http://gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html Developer's Certificate of Origin 1.1].

<h1>Submitting patches</h1>

compat and compat-drivers contributions follow the contribution model implemented by the Linux kernel. Patches or pull requests for compat and compat-drivers must have be signed-offed. If you don't sign off on them they will not accepted. This means adding a line that says "Signed-off-by: Name email" at the end of each commit, indicating that you wrote the code and have the right to pass it on as an open source patch. For exact definition of what the Signed-off-by tag is you can read the definition of the "Developer's Certificate of Origin 1.1", which you can read here:

http://gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html

You can send patches as follows:

<pre>To: mcgrof@kernel.org</pre>
<pre>Cc: backports@vger.kernel.org</pre>
<pre>Subject: compat: foo</pre>

<h1>License</h1>

The compat module and code is all licensed under the same license as the Linux kernel and is considered a complete derivative work of the Linux kernel. For more information refer to the COPYRIGHT file.