(Add See also - License)
(Add license link at top)
|Line 1:||Line 1:|
The project [[Documentation/backports/hacking/todo|TODO list]] .
= Philosophy =
= Philosophy =
Revision as of 01:52, 25 June 2017
The project aims to optimize the backporting process by not only providing backported drivers but by also aiming to automatically backport the Linux kernel by optimizing backporting collateral evolutions.
Backports development flow
The backports.git master development branch always tracks linux-next, allowing it to track all the development trees. This ensures that at the end of each merge window, the state of the backports will be very close to the state of the first release candidate. At this point, the backports project creates a further branch that tracks the progress of the new release over the release candidate evaluation period, to the major release, and on to its lifetime as a stable kernel.
The backports project thus makes three kinds of backports releases possible: those derived from linux-next, those derived from the most recent release candidate if any, and those derived from recent stable kernels.
The project uses two types of git tags:
- daily tags following linux-next
- stable tags following either Linus' tree for rc releases or Greg's stable tree for stable extra version releases
The project uses a master branch and a series of stable branches, a stable branch to track each stable release of the kernel. The master branch always tags linux-next, for example the backports tag backports-20141114 supports using linux-next tag next-20141114. Once Linus makes a release candidate of the kernel a respective stable backports branch is created based on the version of Linux Linus released, this backports branch will be based on a recent backports tag which was tracking linux-next. Future release candidates of the same version of Linux as well as stable release of Linux under the same version will be backported using the respective backports stable branch created.
If any of this is fuzzy please refer to the Increasing Automation in the Backporting of Linux Drivers Using Coccinelle paper, in particular the sections:
- 1A - A. The Linux kernel development model
- 1B - A brief history of the Linux kernel backports project
Git trees you will need
The project backports a few subsystems down to older kernels. To be able to synchronize backporting the latest and greatest the linux-next.git tree is used as its main source for kernel updates. backports brings a general kernel compatibility part containing some backported c functions and header files in addition to patches for code in linux-next to make it work on older kernel versions and with the general kernel compatibility part. You will need first the backports.git tree. Once cloned you can run:
That will get you all other trees and sources required for development, which are:
- linux-next - Stephen Rothwell's tree which tracks all development trees daily
- linux - Linus' tree
- linux-stable - Greg's stable tree for stable extra version releases
This will use git clone --reference to minimize disk usage.
You can generate backports based on different trees depending on your needs. We are supporting build based on linux-next.git, linux.git and linux-stable.git. You can also generate builds based on your own tree, but then you probably have to patch backports yourself to add support for that.
git, python patch and coccinelle are needed to generate a backports release. The tools will complain if you do not have any of the required dependencies.
Generate new release
To generate a own backport release based on linux-next you need the following git trees:
As linux-next changes often you should start with a working version by using a tagged version from linux-next.git and backports.git with the same date in it like backports-20130607 and next-20130607. This is a combination where someone already tested that the patches are still applying.
After you have checked out some version of these git trees, you can start generating a tar by running ./gentree.py in the backports tree. This will take the path to the already checked out linux-next kernel as the second last argument and the path to a directory where the newly created release should be written to as the last parameter. The releases are not created in place any more, but in a new path specified in the last parameter.
./gentree.py --clean --git-revision v3.x /path/to/your/linux-next /path/to/your/newly/created/release
Adding new driver
This is a list of steps to be done when adding a new driver. Not all steps are needed for every new driver it depends on what type of driver is added. You can take the patch adding support for the ieee802154 subsystem as a reference .
You should target the master branch of the backports repository.
- Add directories and files that should be copied from the mainline kernel to
- Add patches to
patches/collateral-evolutions/if needed, it is better to add or modify a header file in
backport/backport-include/than adding a patch.
- Add reference to the newly added driver directory in
backport/Makefile.kernelif needed (just needed for new substems)
- Add reference to the newly added driver directory in
backport/Kconfig.sourcesif needed (just needed for new subsystems)
- Add a defconfig file for this driver in
backport/defconfigs/and modify other defconfig files if appropriate, e.g. if you add a wifi driver add it to
- If the driver does not build on all kernel versions specify a minimum kernel version in
Backports currently supports the latest minor version of every mainline kernel starting from 2.6.24. You should check that backports still builds after making your changes. To do so you should install all supported kernel with the script in
devel/get-compat-kernels and then run
devel/ckmake to build backports against every kernel version.
Backporting a kernel feature
Features being backported should either be backported into a respective backports-3.x.c file under backports/compat/ and/or their respective header file under backport/backport-include/.
Backports Kconfig operation
Go read the backports kconfig operation documentation.
Backports Makefile operation
Go read the backports Makefile operation documentation.
backports contributions follow the contribution model implemented by the Linux kernel. Patches or pull requests for backprots must have be signed-offed. If you don't sign off on them they will not accepted. This means adding a line that says "Signed-off-by: Name <email>" at the end of each commit, indicating that you wrote the code and have the right to pass it on as an open source patch. For exact definition of what the Signed-off-by tag is you can read the definition of the "Developer's Certificate of Origin 1.1", which you can read here:
Remember there are three trees. linux-next itself is a conglomeration of kernel git trees itself, so patches for linux-next.git should be sent to each respective subsystem for which the patches are targeted for. So for example for 802.11 you will want to send them to John Linville and cc linux-wireless, for further guidelines on this see the Submitting Patches guidelines for 802.11. As another example, for bluetooth you will want to send them to Gustavo Padovan and cc the linux-bluetooth mailing list. If your patch touches on others areas of the kernel refer to the MAINTAINERS file on the kernel.
For backports.git please send patches against to:
To: firstname.lastname@example.org, email@example.com CC: firstname.lastname@example.org Subject: [PATCH] backports: fix foo
Patches are preferred sent with a clear commit log entry, if unfamiliar with how to send patches please refer to a git guide.
Requirements for contributions
The code quality, and coding style is precisely the same as followed by the Linux kernel, for that refer to the Linux kernel SubmittingPatches guide. Other than this, since we a supporting backporting you are expected to refer to the respective upstream commit which you are providing a backport for. This should describe both the git commit sha1sum on Linux, and the commit description. The description should also include the Linux kernel version which merged the commit upstream first, this can be used to help understand what target kernels the backport work was for. An example of good commit log follows:
commit 54584122b3279700f5ef2ba4f260a84273215947 Author: Stefan Assmann <email@example.com> Date: Fri May 16 13:21:19 2014 +0200 backports: handle new get_module_info and get_module_eeprom pointers in struct ethtool_ops In kernel 3.5 struct ethtool_ops received 2 new function pointers get_module_info and get_module_eeprom. Address this by putting ifdef around the code. commit 41c3cb6d20f0252308e9796fa4f3dacb4960de91 Author: Stuart Hodgson <firstname.lastname@example.org> Date: Thu Apr 19 09:44:42 2012 +0100 ethtool: Extend the ethtool API to obtain plugin module eeprom data git describe --contains 41c3cb6d20f0252308e9796fa4f3dacb4960de91 v3.5-rc1~109^2~124^2~2 Signed-off-by: Stefan Assmann <email@example.com> Signed-off-by: Hauke Mehrtens <firstname.lastname@example.org>
Backporting preference for EXPORT_SYMBOL_GPL() and proprietary drivers
This framework was designed by intent since its inception only for upstream Linux kernel drivers to avoid excuses about the difficulty to support upstream Linux kernel development while also supporting users on older kernels. To ensure the intent is respected currently all symbols that we do work on to backport are exported via EXPORT_SYMBOL_GPL() as we do work to backport them for the supported kernels. By using EXPORT_SYMBOL_GPL() we make it clear through the only current technical means we have that if you use the backported symbols your software is considered derivative works of the Linux kernel.
Backporting more subsystems
Feel free to any backport subsystems. The requirement is you at least list its kernel dependency and also that you have tested compilation success with ckmake. If you are adding a new subsystem we are in hopes you will also help maintain it.
Generating backports stats
To help backports developers evaluate the gains of use of Coccinelle SmPL a tool has been written as part of the work done towards the Increasing Automation in the Backporting of Linux Drivers Using Coccinelle paper, the tool has a temporary git tree on github, integration of the tool into the upstream backports tree is desirable but porting of the tool to Python must be done first. Below are instructions on how to use the tool to get stats on the latest tree. The statistics give a slew of metrics over the gains of using of Coccinelle towards automatically backporting the Linux kernel on each release.
First git clone upstream backports git tree, and have a linux-next tree available (using Linus' tree is possible as well, you just need to use the respective stable backports branch). Then generate a backports release using --gitdebug so that each step of the backports process creates a commit. This lets the tool evaluate the impact of using Coccinelle at each step. Assuming you have backports cloned on ~/backports/ and the latest linux-next tree with the respective linux-next tag associated with the latest backports tree tag you would do:
git reset --hard next-20151218
mkdir -p ~/build/ rm -rf ~/build/backports-20151218 ./gentree.py --gitdebug ~/linux-next ~/build/backports-20151218
Now go get the temporary backports stats tool and copy over the required binary and script:
git clone https://github.com/mcgrof/backports-cocci-stats.git
cp gen-cocci-stats.sh ~/build/backports-20151218/ cp clean ~/build/backports-20151218/
Now generate the stats, you should see something like this:
mcgrof@ergon ~/build/backports-20151218 (git::master)$ ./gen-cocci-stats.sh
Development and Maintenance efficiency metrics:
dev-efficiency maint-efficiency diff-wc diffstat clean SmPL-Patch 0.555556 2.33333 21 5 9 0065-ndisc_send_na-argument.cocci 2.75 8 64 22 8 0062-iff-no-queue.cocci 0.666667 1.88889 17 6 9 skb_no_xmit_more.cocci 1.16667 2.79167 67 28 24 ptp_getsettime64.cocci 0.142857 1 14 2 14 features_check.cocci 1.18182 5.45455 60 13 11 0055-netdev-tstats.cocci 1.55906 3.55118 451 198 127 0054-struct-proto_ops-sig.cocci 0.666667 1.88889 17 6 9 no-pfmemalloc.cocci 0.634146 1.4878 61 26 41 set_vf_rate.cocci 3.75 10.625 85 30 8 igb_pci_error_handlers.cocci 1.07692 4.23077 55 14 13 ethtool_cmd_mdix.cocci 0.588235 1.23529 21 10 17 rxnfc.cocci 0.285714 1.53571 43 8 28 get_module.cocci 0.285714 1.53571 43 8 28 ethtool_eee.cocci 0.714286 2.28571 16 5 7 skb_no_fcs.cocci 0.25 1.59375 51 8 32 set_vf_spoofchk.cocci 0.428571 2.85714 40 6 14 sriov_configure.cocci 0.87037 2.7037 146 47 54 0031-sk_data_ready.cocci 4.4 17.9 179 44 10 genl-const.cocci 7.11111 49.6667 447 64 9 0019-usb_driver_lpm.cocci 0.571429 4.14286 58 8 14 get_ts_info.cocci 11.5333 45.4 681 173 15 0001-netlink-portid.cocci 2.96667 16.35 981 178 60 0002-no_dmabuf.cocci 0.444444 1.77778 32 8 18 0002-gpio-parent.cocci 0.512821 1.89744 74 20 39 0002-group_attr_bus.cocci 0.769231 2.79487 109 30 39 0001-group_attr_class.cocci 1.47184 5.6484 3711 967 657 all-SmPL.cocci
Patch total diff wc -l: 3075 SmPL total diff wc -l: 3711 Total total diff wc -l: 6786
Patch diff % contribution: 45.3139 SmPL diff % contribution: 54.6861
Now say you wanted to generate stats for a newer release, say for stats based on linux-next tag next-20160122, you would do:
git reset --hard next-20160122
cd ~/backports/ ./gentree.py --gitdebug ~/linux-next/ ~/build/backports-20160122
cd ~/build/backports-20160122 cp ~/backports-cocci-stats/gen-cocci-stats.sh . cp ~/backports-cocci-stats/clean . ./gen-cocci-stats.sh mcgrof@ergon ~/build/backports-20160122 (git::master)$ ./gen-cocci-stats.sh
Development and Maintenance efficiency metrics:
dev-efficiency maint-efficiency diff-wc diffstat clean SmPL-Patch 0.4 2.6 13 2 5 0067-mdio-addr.cocci 0.555556 2.33333 21 5 9 0065-ndisc_send_na-argument.cocci 2.75 8 64 22 8 0062-iff-no-queue.cocci 0.666667 1.88889 17 6 9 skb_no_xmit_more.cocci 1.16667 2.79167 67 28 24 ptp_getsettime64.cocci 0.142857 1 14 2 14 features_check.cocci 1.18182 5.45455 60 13 11 0055-netdev-tstats.cocci 1.55906 3.55118 451 198 127 0054-struct-proto_ops-sig.cocci 0.666667 1.88889 17 6 9 no-pfmemalloc.cocci 0.634146 1.4878 61 26 41 set_vf_rate.cocci 3.75 10.625 85 30 8 igb_pci_error_handlers.cocci 1.07692 4.23077 55 14 13 ethtool_cmd_mdix.cocci 0.588235 1.23529 21 10 17 rxnfc.cocci 0.285714 1.53571 43 8 28 get_module.cocci 0.285714 1.53571 43 8 28 ethtool_eee.cocci 0.714286 2.28571 16 5 7 skb_no_fcs.cocci 0.25 1.59375 51 8 32 set_vf_spoofchk.cocci 0.428571 2.85714 40 6 14 sriov_configure.cocci 0.87037 2.7037 146 47 54 0031-sk_data_ready.cocci 4.4 17.9 179 44 10 genl-const.cocci 7.11111 49.6667 447 64 9 0019-usb_driver_lpm.cocci 0.571429 4.14286 58 8 14 get_ts_info.cocci 11.5333 45.4 681 173 15 0001-netlink-portid.cocci 2.96667 16.35 981 178 60 0002-no_dmabuf.cocci 0.444444 1.77778 32 8 18 0002-gpio-parent.cocci 0.512821 1.89744 74 20 39 0002-group_attr_bus.cocci 0.769231 2.79487 109 30 39 0001-group_attr_class.cocci 1.46375 5.62538 3724 969 662 all-SmPL.cocci
Patch total diff wc -l: 3087 SmPL total diff wc -l: 3724 Total total diff wc -l: 6811
Patch diff % contribution: 45.3237 SmPL diff % contribution: 54.6763
- This text is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.