Latest revision as of 05:24, 8 September 2017

NOTE: the backports project's process is being reworked and redefined. Some of the information in this page may be outdated and will be updated when the changes in the process are finalized.

The project has a To-Do List and a License.

[edit] Philosophy

The project aims to optimize the backporting process by not only providing backported drivers but by also aiming to automatically backport the Linux kernel by optimizing backporting collateral evolutions.

[edit] Backports development flow

1. The backports.git "master" development branch always tracks "linux-next", allowing it to track all the development trees.
   • This ensures that, at the close of each merge window, the state of the backports will be very close to the state of the first release candidate.
2. At the close of each merge window, the Backports Project creates a new branch.
   • This new branch tracks the progress of the impending release throughout the release candidate evaluation period, to the major release, and on to its lifetime as a stable kernel.

The backports project thus makes three kinds of backports releases possible:

• those derived from linux-next
• those derived from the most recent release candidate (if any)
• those derived from recent stable kernels.

[edit] Backports git tree tags and branches

The project uses two types of git tags:

• daily tags following linux-next
• stable tags following either Linus' tree for rc releases or Greg's stable tree for stable extra version releases

The project uses a master branch and a series of stable branches, a stable branch to track each stable release of the kernel. The master branch always tags linux-next, for example the backports tag backports-20141114 supports using linux-next tag next-20141114. Once Linus makes a release candidate of the kernel a respective stable backports branch is created based on the version of Linux Linus released, this backports branch will be based on a recent backports tag which was tracking linux-next. Future release candidates of the same version of Linux as well as stable release of Linux under the same version will be backported using the respective backports stable branch created.

If any of this is fuzzy please refer to the Increasing Automation in the Backporting of Linux Drivers Using Coccinelle paper, in particular the sections:

• 1A - A. The Linux kernel development model
• 1B - A brief history of the Linux kernel backports project

[edit] Git trees you will need

The project backports a few subsystems down to older kernels. To be able to synchronize backporting the latest and greatest the linux-next.git tree is used as its main source for kernel updates. backports brings a general kernel compatibility part containing some backported c functions and header files in addition to patches for code in linux-next to make it work on older kernel versions and with the general kernel compatibility part. You will need first the backports.git tree. Once cloned you can run:

./devel/backports-update-manager

That will get you all other trees and sources required for development, which are:

• linux-next - Stephen Rothwell's tree which tracks all development trees daily
• linux - Linus' tree
• linux-stable - Greg's stable tree for stable extra version releases

This will use git clone --reference to minimize disk usage.

You can generate backports based on different trees depending on your needs. We are supporting build based on linux-next.git, linux.git and linux-stable.git. You can also generate builds based on your own tree, but then you probably have to patch backports yourself to add support for that.

[edit] Tool prerequisites

git, python patch and coccinelle are needed to generate a backports release. The tools will complain if you do not have any of the required dependencies.

[edit] Generate new release

To generate a own backport release based on linux-next you need the following git trees:

git://git.kernel.org/pub/scm/linux/kernel/git/backports/backports.git
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git

As linux-next changes often you should start with a working version by using a tagged version from linux-next.git and backports.git with the same date in it like backports-20130607 and next-20130607. This is a combination where someone already tested that the patches are still applying.

After you have checked out some version of these git trees, you can start generating a tar by running ./gentree.py in the backports tree. This will take the path to the already checked out linux-next kernel as the second last argument and the path to a directory where the newly created release should be written to as the last parameter. The releases are not created in place any more, but in a new path specified in the last parameter.

./gentree.py --clean --git-revision v3.x /path/to/your/linux-next /path/to/your/newly/created/release

[edit] Adding new driver

This is a list of steps to be done when adding a new driver. Not all steps are needed for every new driver it depends on what type of driver is added. You can take the patch adding support for the ieee802154 subsystem as a reference [1].

You should target the master branch of the backports repository.

1. Add directories and files that should be copied from the mainline kernel to copy-list
2. Add patches to patches/collateral-evolutions/ if needed, it is better to add or modify a header file in backport/backport-include/ than adding a patch.
3. Add reference to the newly added driver directory in backport/Makefile.kernel if needed (just needed for new substems)
4. Add reference to the newly added driver directory in backport/Kconfig.sources if needed (just needed for new subsystems)
5. Add a defconfig file for this driver in backport/defconfigs/ and modify other defconfig files if appropriate, e.g. if you add a wifi driver add it to backport/defconfigs/wifi
6. If the driver does not build on all kernel versions specify a minimum kernel version in dependencies

Backports currently supports the latest minor version of every mainline kernel starting from 2.6.24. You should check that backports still builds after making your changes. To do so you should install all supported kernel with the script in devel/get-compat-kernels and then run devel/ckmake to build backports against every kernel version.

[edit] Backporting a kernel feature

Features being backported should either be backported into a respective backports-3.x.c file under backports/compat/ and/or their respective header file under backport/backport-include/.

[edit] Backports Kconfig operation

Go read the backports kconfig operation documentation.

[edit] Backports Makefile operation

Go read the backports Makefile operation documentation.

[edit] Sending patches

backports contributions follow the contribution model implemented by the Linux kernel. Patches or pull requests for backports must have be signed-offed. If you don't sign off on them they will not accepted. This means adding a line that says "Signed-off-by: Name <email>" at the end of each commit, indicating that you wrote the code and have the right to pass it on as an open source patch. For exact definition of what the Signed-off-by tag is you can read the definition of the "Developer's Certificate of Origin 1.1", which you can read here:

http://gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html

Remember there are three trees. linux-next itself is a conglomeration of kernel git trees itself, so patches for linux-next.git should be sent to each respective subsystem for which the patches are targeted for. So for example for 802.11 you will want to send them to John Linville and cc linux-wireless, for further guidelines on this see the Submitting Patches guidelines for 802.11. As another example, for bluetooth you will want to send them to Gustavo Padovan and cc the linux-bluetooth mailing list. If your patch touches on others areas of the kernel refer to the MAINTAINERS file on the kernel.

For backports.git please send patches against to:

To: hauke@hauke-m.de, mcgrof@kernel.org
CC: backports@vger.kernel.org
Subject: [PATCH] backports: fix foo

Patches are preferred sent with a clear commit log entry, if unfamiliar with how to send patches please refer to a git guide.

[edit] Requirements for contributions

The code quality, and coding style is precisely the same as followed by the Linux kernel, for that refer to the Linux kernel SubmittingPatches guide. Other than this, since we a supporting backporting you are expected to refer to the respective upstream commit which you are providing a backport for. This should describe both the git commit sha1sum on Linux, and the commit description. The description should also include the Linux kernel version which merged the commit upstream first, this can be used to help understand what target kernels the backport work was for. An example of good commit log follows:

commit 54584122b3279700f5ef2ba4f260a84273215947
Author: Stefan Assmann <sassmann@kpanic.de>
Date:   Fri May 16 13:21:19 2014 +0200

    backports: handle new get_module_info and get_module_eeprom pointers in struct ethtool_ops
    
    In kernel 3.5 struct ethtool_ops received 2 new function pointers
    get_module_info and get_module_eeprom. Address this by putting ifdef around
    the code.
    
    commit 41c3cb6d20f0252308e9796fa4f3dacb4960de91
    Author: Stuart Hodgson <smhodgson@solarflare.com>
    Date:   Thu Apr 19 09:44:42 2012 +0100
    
        ethtool: Extend the ethtool API to obtain plugin module eeprom data
    
    git describe --contains 41c3cb6d20f0252308e9796fa4f3dacb4960de91
    v3.5-rc1~109^2~124^2~2
    
    Signed-off-by: Stefan Assmann <sassmann@kpanic.de>
    Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>

[edit] Backporting preference for EXPORT_SYMBOL_GPL() and proprietary drivers

This framework was designed by intent since its inception only for upstream Linux kernel drivers to avoid excuses about the difficulty to support upstream Linux kernel development while also supporting users on older kernels. To ensure the intent is respected currently all symbols that we do work on to backport are exported via EXPORT_SYMBOL_GPL() as we do work to backport them for the supported kernels. By using EXPORT_SYMBOL_GPL() we make it clear through the only current technical means we have that if you use the backported symbols your software is considered derivative works of the Linux kernel.

[edit] Backporting more subsystems

Feel free to any backport subsystems. The requirement is you at least list its kernel dependency and also that you have tested compilation success with ckmake. If you are adding a new subsystem we are in hopes you will also help maintain it.

[edit] Generating backports stats

To help backports developers evaluate the gains of use of Coccinelle SmPL a tool has been written as part of the work done towards the Increasing Automation in the Backporting of Linux Drivers Using Coccinelle paper, the tool has a temporary git tree on github, integration of the tool into the upstream backports tree is desirable but porting of the tool to Python must be done first. Below are instructions on how to use the tool to get stats on the latest tree. The statistics give a slew of metrics over the gains of using of Coccinelle towards automatically backporting the Linux kernel on each release.

First git clone upstream backports git tree, and have a linux-next tree available (using Linus' tree is possible as well, you just need to use the respective stable backports branch). Then generate a backports release using --gitdebug so that each step of the backports process creates a commit. This lets the tool evaluate the impact of using Coccinelle at each step. Assuming you have backports cloned on ~/backports/ and the latest linux-next tree with the respective linux-next tag associated with the latest backports tree tag you would do:

cd linux-next git reset --hard next-20151218

cd ~/backports/

mkdir -p ~/build/ rm -rf ~/build/backports-20151218 ./gentree.py --gitdebug ~/linux-next ~/build/backports-20151218

Now go get the temporary backports stats tool and copy over the required binary and script:

cd ~/ git clone https://github.com/mcgrof/backports-cocci-stats.git cd backports-cocci-stats make

cp gen-cocci-stats.sh ~/build/backports-20151218/ cp clean ~/build/backports-20151218/

Now generate the stats, you should see something like this:

mcgrof@ergon ~/build/backports-20151218 (git::master)$ ./gen-cocci-stats.sh

   dev-efficiency	  maint-efficiency	 diff-wc	  diffstat	     clean	     SmPL-Patch
         0.555556	           2.33333	      21	         5	         9	0065-ndisc_send_na-argument.cocci
             2.75	                 8	      64	        22	         8	0062-iff-no-queue.cocci
         0.666667	           1.88889	      17	         6	         9	skb_no_xmit_more.cocci
          1.16667	           2.79167	      67	        28	        24	ptp_getsettime64.cocci
         0.142857	                 1	      14	         2	        14	features_check.cocci
          1.18182	           5.45455	      60	        13	        11	0055-netdev-tstats.cocci
          1.55906	           3.55118	     451	       198	       127	0054-struct-proto_ops-sig.cocci
         0.666667	           1.88889	      17	         6	         9	no-pfmemalloc.cocci
         0.634146	            1.4878	      61	        26	        41	set_vf_rate.cocci
             3.75	            10.625	      85	        30	         8	igb_pci_error_handlers.cocci
          1.07692	           4.23077	      55	        14	        13	ethtool_cmd_mdix.cocci
         0.588235	           1.23529	      21	        10	        17	    rxnfc.cocci
         0.285714	           1.53571	      43	         8	        28	get_module.cocci
         0.285714	           1.53571	      43	         8	        28	ethtool_eee.cocci
         0.714286	           2.28571	      16	         5	         7	skb_no_fcs.cocci
             0.25	           1.59375	      51	         8	        32	set_vf_spoofchk.cocci
         0.428571	           2.85714	      40	         6	        14	sriov_configure.cocci
          0.87037	            2.7037	     146	        47	        54	0031-sk_data_ready.cocci
              4.4	              17.9	     179	        44	        10	genl-const.cocci
          7.11111	           49.6667	     447	        64	         9	0019-usb_driver_lpm.cocci
         0.571429	           4.14286	      58	         8	        14	get_ts_info.cocci
          11.5333	              45.4	     681	       173	        15	0001-netlink-portid.cocci
          2.96667	             16.35	     981	       178	        60	0002-no_dmabuf.cocci
         0.444444	           1.77778	      32	         8	        18	0002-gpio-parent.cocci
         0.512821	           1.89744	      74	        20	        39	0002-group_attr_bus.cocci
         0.769231	           2.79487	     109	        30	        39	0001-group_attr_class.cocci
          1.47184	            5.6484	    3711	       967	       657	 all-SmPL.cocci

Patch diff % contribution: 45.3139 SmPL diff % contribution: 54.6861

Now say you wanted to generate stats for a newer release, say for stats based on linux-next tag next-20160122, you would do:

cd ~/linux-next/ git reset --hard next-20160122

cd ~/backports/ ./gentree.py --gitdebug ~/linux-next/ ~/build/backports-20160122

cd ~/build/backports-20160122 cp ~/backports-cocci-stats/gen-cocci-stats.sh . cp ~/backports-cocci-stats/clean . ./gen-cocci-stats.sh mcgrof@ergon ~/build/backports-20160122 (git::master)$ ./gen-cocci-stats.sh

   dev-efficiency	  maint-efficiency	 diff-wc	  diffstat	     clean	     SmPL-Patch
              0.4	               2.6	      13	         2	         5	0067-mdio-addr.cocci
         0.555556	           2.33333	      21	         5	         9	0065-ndisc_send_na-argument.cocci
             2.75	                 8	      64	        22	         8	0062-iff-no-queue.cocci
         0.666667	           1.88889	      17	         6	         9	skb_no_xmit_more.cocci
          1.16667	           2.79167	      67	        28	        24	ptp_getsettime64.cocci
         0.142857	                 1	      14	         2	        14	features_check.cocci
          1.18182	           5.45455	      60	        13	        11	0055-netdev-tstats.cocci
          1.55906	           3.55118	     451	       198	       127	0054-struct-proto_ops-sig.cocci
         0.666667	           1.88889	      17	         6	         9	no-pfmemalloc.cocci
         0.634146	            1.4878	      61	        26	        41	set_vf_rate.cocci
             3.75	            10.625	      85	        30	         8	igb_pci_error_handlers.cocci
          1.07692	           4.23077	      55	        14	        13	ethtool_cmd_mdix.cocci
         0.588235	           1.23529	      21	        10	        17	    rxnfc.cocci
         0.285714	           1.53571	      43	         8	        28	get_module.cocci
         0.285714	           1.53571	      43	         8	        28	ethtool_eee.cocci
         0.714286	           2.28571	      16	         5	         7	skb_no_fcs.cocci
             0.25	           1.59375	      51	         8	        32	set_vf_spoofchk.cocci
         0.428571	           2.85714	      40	         6	        14	sriov_configure.cocci
          0.87037	            2.7037	     146	        47	        54	0031-sk_data_ready.cocci
              4.4	              17.9	     179	        44	        10	genl-const.cocci
          7.11111	           49.6667	     447	        64	         9	0019-usb_driver_lpm.cocci
         0.571429	           4.14286	      58	         8	        14	get_ts_info.cocci
          11.5333	              45.4	     681	       173	        15	0001-netlink-portid.cocci
          2.96667	             16.35	     981	       178	        60	0002-no_dmabuf.cocci
         0.444444	           1.77778	      32	         8	        18	0002-gpio-parent.cocci
         0.512821	           1.89744	      74	        20	        39	0002-group_attr_bus.cocci
         0.769231	           2.79487	     109	        30	        39	0001-group_attr_class.cocci
          1.46375	           5.62538	    3724	       969	       662	 all-SmPL.cocci

Patch diff % contribution: 45.3237 SmPL diff % contribution: 54.6763

[edit] See also

• To-Do List
• License

- This text is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.