Documentation/backports/hacking

From Driver Backports Wiki
(Difference between revisions)
Jump to: navigation, search
Line 100: Line 100:
  
 
= TODO =
 
= TODO =
 +
 +
== More subsystems ==
 +
 +
Feel free to any backport subsystems. The requirement is you at least list its kernel dependency and also that you have tested compilation success with ckmake. If you are adding a new subsystem we are in hopes you will also help maintain it.
  
 
== DKMS support ==
 
== DKMS support ==

Revision as of 21:04, 25 July 2013

Contents

Philosophy

The project aims to optimize the backporting process by not only providing backported drivers but by also aiming to automatically backport the Linux kernel by optimizing backporting collateral evolutions.

Git trees you will need

backports backports a few subsystems down to older kernels. To be able to synchronize backporting the latest and greatest the linux-next.git tree is used as its main source for kernel updates. backports brings a general kernel compatibility part containing some backported c functions and header files in addition to patches for code in linux-next to make it work on older kernel versions and with the general kernel compatibility part. This is now in one repository backports.git and not spitted over two any more.

You can generate backports based on different trees depending on your needs. We are supporting build based on linux-next.git, linux.git and linux-stable.git. You can also generate builds based on your own tree, but then you probably have to patch backports yourself to add support for that.

Generate new release

To generate a own backport release based on linux-next you need the following git trees:

git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/backports.git
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git

As linux-next changes often you should start with a working version by using a tagged version from linux-next.git and backports.git with the same date in it like backports-20130607 and next-20130607. This is a combination where someone already tested that the patches are still applying.

After you have checked out some version of these git trees, you can start generating a tar by running ./gentree.py in the backports tree. This will take the path to the already checked out linux-next kernel as the second last argument and the path to a directory where the newly created release should be written to as the last parameter. The releases are not created in place any more, but in a new path specified in the last parameter.

./gentree.py --clean --git-revision v3.x /path/to/your/linux-next /path/to/your/newly/created/release

Adding new driver

This is a list of steps to be done when adding a new driver. Not all steps are needed for every new driver it depends on what type of driver is added. You can take the patch adding support for the ieee802154 subsystem as a reference [1].

You should target the master branch of the backports repository.

  1. Add directories and files that should be copied from the mainline kernel to copy-list
  2. Add patches to patches/collateral-evolutions/ if needed, it is better to add or modify a header file in backport/backport-include/ than adding a patch.
  3. Add reference to the newly added driver directory in backport/Makefile.kernel if needed (just needed for new substems)
  4. Add reference to the newly added driver directory in backport/Kconfig if needed (just needed for new subsystems)
  5. Add a defconfig file for this driver in backport/defconfigs/ and modify other defconfig files if appropriate, e.g. if you add a wifi driver add it to backport/defconfigs/wifi
  6. If the driver does not build on all kernel versions specify a minimum kernel version in dependencies

Backports currently supports the latest minor version of every mainline kernel starting from 2.6.24. You should check that backports still builds after making your changes. To do so you should install all supported kernel with the script in devel/get-compat-kernels and then run devel/ckmake to build backports against every kernel version.

Backporting a kernel feature

Features being backported should either be backported into a respective backports-3.x.c file under backports/compat/ and/or their respective header file under backport/backport-include/.

Backports Kconfig operation

Go read the backports kconfig operation documentation.

Backports Makefile operation

Go read the backports Makefile operation documentation.

Sending patches

backports contributions follow the contribution model implemented by the Linux kernel. Patches or pull requests for backprots must have be signed-offed. If you don't sign off on them they will not accepted. This means adding a line that says "Signed-off-by: Name <email>" at the end of each commit, indicating that you wrote the code and have the right to pass it on as an open source patch. For exact definition of what the Signed-off-by tag is you can read the definition of the "Developer's Certificate of Origin 1.1", which you can read here:

http://gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html

Remember there are three trees. linux-next itself is a conglomeration of kernel git trees itself, so patches for linux-next.git should be sent to each respective subsystem for which the patches are targeted for. So for example for 802.11 you will want to send them to John Linville and cc linux-wireless, for further guidelines on this see the Submitting Patches guidelines for 802.11. As another example, for bluetooth you will want to send them to Gustavo Padovan and cc the linux-bluetooth mailing list. If your patch touches on others areas of the kernel refer to the MAINTAINERS file on the kernel.

For backports.git please send patches against to:

To: Luis R. Rodriguez <mcgrof@kernel.org>
CC: backports@vger.kernel.org
Subject: [PATCH] backports: fix foo

Patches are preferred sent with a clear commit log entry, if unfamiliar with how to send patches please refer to a git guide.

Backporting preference for EXPORT_SYMBOL_GPL() and proprietary drivers

This framework was designed by intent since its inception only for upstream Linux kernel drivers to avoid excuses about the difficulty to support upstream Linux kernel development while also supporting users on older kernels. To ensure the intent is respected currently all symbols that we do work on to backport are exported via EXPORT_SYMBOL_GPL() as we do work to backport them for the supported kernels. By using EXPORT_SYMBOL_GPL() we make it clear that if you use the backported symbols your software is considered derivative works of the Linux kernel.

TODO

More subsystems

Feel free to any backport subsystems. The requirement is you at least list its kernel dependency and also that you have tested compilation success with ckmake. If you are adding a new subsystem we are in hopes you will also help maintain it.

DKMS support

Someone seems to want add support for DKMS. Who was that? What benefits does this bring?

Who's working on this ?

No one

make localmodconfig support

The Linux kernel has make localmodconfig support, random users shouldn't be expected to know what modules they need, so in order to allow users to get only the drivers they need built we'd need something similar to make localmodconfig. There are a few ways to do this. The best way would be to get upstream a mapping of a kernel object of a device to list its respective Kconfig. We'd also need a map of Kconfig changes over time as the kernel advances so that if a user is on an older system their old kobject Kconfig could be mapped to a new one. This should require changes upstream.

Other simpler solutions are to rely on 'lspci -k' output and map this to respective device drivers but this solution would be PCI bus specific. We should review how the upstream make localmodconfig currently works and improve upon that to enable support for it on backports.

Who's working on this?

No one.

Module namespaces

Module namespace can consist of either adding prefixes to existing symbols or modifying the Linux kernel module data structure so that upon request for a symbol it's own namespace can be inspected for its requested namespace. The Linux kernel backports project embraces prefixing backported versions of symbols by prefixing backported versions of symbols with backports_. A macro LINUX_BACKPORT() was added to help with this. For example:

  1. define crc8 LINUX_BACKPORT(crc8)

Andi Kleen introduced the modular structure modification version of module namespaces back in 2007 but Rusty Russell nacked these patches. Because of this we're left with remapping modules to their own namespaces without modifying the in-kernel module structure. It may be possible to somehow provide module namespaces without requiring modifications to the in-kernel module structure, but such mechanisms would end up being hacks. We need to evaluate if we can figure a clever way to support this that is not so error prone or if we want to respinning Andi's patches with the intent to help support kernel backporting.

Motivations

At times Linux distributions may choose, for whatever support-nightmare reasons, to support two versions of a subsystem module and have two separate device drivers use a different version of that subsystem. The reason this is ill-advised is we stand to gain by having development done on the upstream and latest versions of each subsystem and drivers; that backports framework allows you to get that. An example helps to clarify here. Suppose a distribution wants to ship with an old version of mac80211 and have the iwlwifi driver use it, but for ath9k they'd want to ship the newer version of mac80211 and new version of ath9k. This cannot be supported unless module namespaces are created.

Another more reasonable motivation is to avoid regressions with features / symbols being updated only for backported drivers. Non-backported drivers will use the non-backported version of the routine. This should avoid regressions with non-backported drivers using these routines.

Who's working on this?

No one.

Non modular backport support

As it stands the backports project only supports providing backport updates of modules. That is your system gets updated drivers but if a core kernel component that is not modular is updated we can't provide an update for it unless we can technically extract that core kernel feature, map it in modular form and then have the backported version of the modules use a new namespace. The problem with doing this happens when a core kernel component is dependent upon another core kernel component that cannot be backported or extracted out.

There is hope however. The idea consists to not backport live kernels given that ksplice seems limited to what can be done for backporting purposes but instead to allow Linux distributions to pick a stable kernel, pick a backports release and have it merge into that kernel release. Typically Linux distributions cherry pick patches and deal with merge conflicts manually. The idea is to the stable kernel as a vanilla release and deal with the integration of new backported components using the backports framework. Just as you would build a kernel through make menuconfig upon kernel configuration time you'd then have present your vanilla kernel components and also backported features through a menu subsection. You'd then be able to build backported features as in-kernel or modular just as you would typically build a vanilla kernel.

See commit 0935dea for the hints supplied on how to deal with backporting core kernel features. The idea is you should be able to modify the in-kernel's init routine, add our own backport_init() in the series of init calls and throw onto it what we need from future kernels.

This likely will require a separate script other than gentree.py, perhaps mergetree.py.

Motivation

If possible this solution should provide an alternative for users / Linux distributions to cherry pick changes from a future kernel down to older kernels by providing full frameworks / subsystems. One clear benefit is the time reduced to provide a backport and complete and clear separation against what is on a vanilla upstream stable kernel.

Who's working on this?

Luis will try to dive to look into this August 19th - Sept 13th. With a simple proof of concept feature / driver.

SmPL integration

Typically patches may fail if hunk offsets are off by the allowed offset. We upkeep hunk offsets placements up to date by refreshing patches. We used to use quit refresh to refresh patch offsets but today we simply extract the new delta after application of a patch in which hunk offsets were off. At times however patches can fail to apply due to contexual changes within the patch. We are considering using SmPL to help with this.

Its plausible that if you're hacking on the Linux kernel that you may have not yet heard of coccinelle or SmPL patches. Truth is SmPL is already widely used across many subsystem in the Linux kernel and SmPL's usage even extends outside of the Linux kernel. The page that tries to document the impact of SmPL on the Linux kernel is clearly out of date and likely just needs to be automated with git grep. The idea behind the need for SmPL came about studying and formalizing the concept of collateral evolutions on the Linux kernel. Luis elaborates on his blog on how this can be used to help optimize collateral evolutions. While still theoretical, if we want to make backporting more efficient we need to start using SmPL soon. The work should consist of properly supplying expressions that summarize backporting collateral evolutions from patches from which SmPL could likely deduced or written manually. In order accomplish this each collateral evolution must first be atomically reduced as much as possible. We then will need to categorize backported collateral evolution and prioritize working on categories that we could likely infer SmPL from. As it stands patches which have been already reduced as much as possible have been annotated by using a four digit prefix. Patches which can likely be split up further remain with a two digit prefix.

Patches that need to be reviewed are under the patches/collateral-evolutions directory. The way backports works -- any directory name can be used to separate patches into any logical form as required. Integration with SmPL can start with inferring SmPL from the few four-digit prefixed patches and subsequently writing a Python tool which would use the SmPL patch to patch kernel code. The first objective is to prove backporting transformations are equivalent by either using the old series of patches for a specific collateral evolution or by using its expression in SmPL form. A proof can be obtained by computing the sha256 delta on the tree with both approaches.

Who's working on this?

Luis will try to dive to look into this August 19th - Sept 13th. With a simple proof of concept feature / driver. Luis hopes to further collaborate with Julia on this between October 16 - December 15.

Infer SmPL for patch hunks healing

If a collateral evolution does not yet have a formal SmPL expression to represent it and if SmPL inference is plausible it'd be great to infer SmPL on the fly as a patch fails to attempt fixing the patch. SmPL inference however is only possible today with a series of similar hunks, at least 2, across different set of offsets. SmPL hunk healing then should not be possible for addressing singular hunk collateral evolutions.

Who's working on this?

Luis hopes to further collaborate with Julia on this between October 16 - December 15.

Using the inverse of SmPL to backport

If Linux kernel development dealt with collateral evolutions with a strict SmPL form it is theoretically possible that one might be able to use the inverse of a patch to backport a collateral evolution. We need to prove / disprove this. Exceptions to this and corner cases should also be inspected.

Who's working on this?

Luis hopes to further collaborate with Julia on this between October 16 - December 15.

88x31.png - This text is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Personal tools