Documentation/backports/hacking/todo

From Driver Backports Wiki
(Difference between revisions)
Jump to: navigation, search
(Created page with " The Linux kernel backports TODO list. = DKMS support = Someone seems to want add support for [https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support DKMS]. Who was tha...")
 
(re-order todo list)
Line 1: Line 1:
  
 
The Linux kernel backports TODO list.
 
The Linux kernel backports TODO list.
 
= DKMS support =
 
 
Someone seems to want add support for [https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support DKMS]. Who was that? What benefits does this bring?
 
 
== Who's working on this ? ==
 
 
No one
 
  
 
= make localmodconfig support =
 
= make localmodconfig support =
Line 16: Line 8:
 
Other simpler solutions are to rely on 'lspci -k' output and map this to respective device drivers but this solution would be PCI bus specific.
 
Other simpler solutions are to rely on 'lspci -k' output and map this to respective device drivers but this solution would be PCI bus specific.
 
We should review how the upstream ''make localmodconfig'' currently works and improve upon that to enable support for it on backports.
 
We should review how the upstream ''make localmodconfig'' currently works and improve upon that to enable support for it on backports.
 
== Who's working on this? ==
 
 
No one.
 
 
= Module namespaces =
 
 
Module namespace can consist of either adding prefixes to existing symbols or modifying the Linux kernel module data structure so that upon request for a symbol it's own namespace can be inspected for its requested namespace. The Linux kernel backports project embraces prefixing backported versions of symbols by prefixing backported versions of symbols with ''backports_''. A macro LINUX_BACKPORT() was added to help with this. For example:
 
 
<tt>
 
#define crc8 LINUX_BACKPORT(crc8)
 
</tt>
 
 
Andi Kleen introduced the modular structure modification version of [http://thread.gmane.org/gmane.linux.network/78674 module namespaces] back in 2007 but [http://article.gmane.org/gmane.linux.kernel/606885 Rusty Russell nacked these patches]. Because of this we're left with remapping modules to their own namespaces without modifying the in-kernel module structure. It '''may''' be possible to somehow provide module namespaces without requiring modifications to the in-kernel module structure, but such mechanisms would end up being hacks. We need to evaluate if we can figure a clever way to support this that is not so error prone or if we want to respinning Andi's patches with the intent to help support kernel backporting.
 
 
== Motivations ==
 
 
At times Linux distributions may choose, for whatever support-nightmare reasons, to support two versions of a subsystem module and have two separate device drivers use a different version of that subsystem. The reason this is ill-advised is we stand to gain by having development done on the upstream and latest versions of each subsystem and drivers; that backports framework allows you to get that. An example helps to clarify here. Suppose a distribution wants to ship with an old version of mac80211 and have the iwlwifi driver use it, but for ath9k they'd want to ship the newer version of mac80211 and new version of ath9k. This cannot be supported unless module namespaces are created.
 
 
Another more reasonable motivation is to avoid regressions with features / symbols being updated only for backported drivers. Non-backported drivers will use the non-backported version of the routine. This should avoid regressions with non-backported drivers using these routines.
 
  
 
== Who's working on this? ==
 
== Who's working on this? ==
Line 86: Line 58:
  
 
Luis hopes to further collaborate with Julia on this between October 16 - December 15.
 
Luis hopes to further collaborate with Julia on this between October 16 - December 15.
 +
 +
= DKMS support =
 +
 +
Someone seems to want add support for [https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support DKMS]. Who was that? What benefits does this bring?
 +
 +
== Who's working on this ? ==
 +
 +
No one
 +
 +
 +
= Module namespaces =
 +
 +
Module namespace can consist of either adding prefixes to existing symbols or modifying the Linux kernel module data structure so that upon request for a symbol it's own namespace can be inspected for its requested namespace. The Linux kernel backports project embraces prefixing backported versions of symbols by prefixing backported versions of symbols with ''backports_''. A macro LINUX_BACKPORT() was added to help with this. For example:
 +
 +
<tt>
 +
#define crc8 LINUX_BACKPORT(crc8)
 +
</tt>
 +
 +
Andi Kleen introduced the modular structure modification version of [http://thread.gmane.org/gmane.linux.network/78674 module namespaces] back in 2007 but [http://article.gmane.org/gmane.linux.kernel/606885 Rusty Russell nacked these patches]. Because of this we're left with remapping modules to their own namespaces without modifying the in-kernel module structure. It '''may''' be possible to somehow provide module namespaces without requiring modifications to the in-kernel module structure, but such mechanisms would end up being hacks. We need to evaluate if we can figure a clever way to support this that is not so error prone or if we want to respinning Andi's patches with the intent to help support kernel backporting.
 +
 +
== Motivations ==
 +
 +
At times Linux distributions may choose, for whatever support-nightmare reasons, to support two versions of a subsystem module and have two separate device drivers use a different version of that subsystem. The reason this is ill-advised is we stand to gain by having development done on the upstream and latest versions of each subsystem and drivers; that backports framework allows you to get that. An example helps to clarify here. Suppose a distribution wants to ship with an old version of mac80211 and have the iwlwifi driver use it, but for ath9k they'd want to ship the newer version of mac80211 and new version of ath9k. This cannot be supported unless module namespaces are created.
 +
 +
Another more reasonable motivation is to avoid regressions with features / symbols being updated only for backported drivers. Non-backported drivers will use the non-backported version of the routine. This should avoid regressions with non-backported drivers using these routines.
 +
 +
== Who's working on this? ==
 +
 +
No one.
  
 
[[File:88x31.png‎]] - This text is licensed under a [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons Attribution-ShareAlike 3.0 Unported License].
 
[[File:88x31.png‎]] - This text is licensed under a [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons Attribution-ShareAlike 3.0 Unported License].

Revision as of 21:12, 25 July 2013

The Linux kernel backports TODO list.

Contents

make localmodconfig support

The Linux kernel has make localmodconfig support, random users shouldn't be expected to know what modules they need, so in order to allow users to get only the drivers they need built we'd need something similar to make localmodconfig. There are a few ways to do this. The best way would be to get upstream a mapping of a kernel object of a device to list its respective Kconfig. We'd also need a map of Kconfig changes over time as the kernel advances so that if a user is on an older system their old kobject Kconfig could be mapped to a new one. This should require changes upstream.

Other simpler solutions are to rely on 'lspci -k' output and map this to respective device drivers but this solution would be PCI bus specific. We should review how the upstream make localmodconfig currently works and improve upon that to enable support for it on backports.

Who's working on this?

No one.

Non modular backport support

As it stands the backports project only supports providing backport updates of modules. That is your system gets updated drivers but if a core kernel component that is not modular is updated we can't provide an update for it unless we can technically extract that core kernel feature, map it in modular form and then have the backported version of the modules use a new namespace. The problem with doing this happens when a core kernel component is dependent upon another core kernel component that cannot be backported or extracted out.

There is hope however. The idea consists to not backport live kernels given that ksplice seems limited to what can be done for backporting purposes but instead to allow Linux distributions to pick a stable kernel, pick a backports release and have it merge into that kernel release. Typically Linux distributions cherry pick patches and deal with merge conflicts manually. The idea is to the stable kernel as a vanilla release and deal with the integration of new backported components using the backports framework. Just as you would build a kernel through make menuconfig upon kernel configuration time you'd then have present your vanilla kernel components and also backported features through a menu subsection. You'd then be able to build backported features as in-kernel or modular just as you would typically build a vanilla kernel.

See commit 0935dea for the hints supplied on how to deal with backporting core kernel features. The idea is you should be able to modify the in-kernel's init routine, add our own backport_init() in the series of init calls and throw onto it what we need from future kernels.

This likely will require a separate script other than gentree.py, perhaps mergetree.py.

Motivation

If possible this solution should provide an alternative for users / Linux distributions to cherry pick changes from a future kernel down to older kernels by providing full frameworks / subsystems. One clear benefit is the time reduced to provide a backport and complete and clear separation against what is on a vanilla upstream stable kernel.

Who's working on this?

Luis will try to dive to look into this August 19th - Sept 13th. With a simple proof of concept feature / driver.

SmPL integration

Typically patches may fail if hunk offsets are off by the allowed offset. We upkeep hunk offsets placements up to date by refreshing patches. We used to use quit refresh to refresh patch offsets but today we simply extract the new delta after application of a patch in which hunk offsets were off. At times however patches can fail to apply due to contexual changes within the patch. We are considering using SmPL to help with this.

Its plausible that if you're hacking on the Linux kernel that you may have not yet heard of coccinelle or SmPL patches. Truth is SmPL is already widely used across many subsystem in the Linux kernel and SmPL's usage even extends outside of the Linux kernel. The page that tries to document the impact of SmPL on the Linux kernel is clearly out of date and likely just needs to be automated with git grep. The idea behind the need for SmPL came about studying and formalizing the concept of collateral evolutions on the Linux kernel. Luis elaborates on his blog on how this can be used to help optimize collateral evolutions. While still theoretical, if we want to make backporting more efficient we need to start using SmPL soon. The work should consist of properly supplying expressions that summarize backporting collateral evolutions from patches from which SmPL could likely deduced or written manually. In order accomplish this each collateral evolution must first be atomically reduced as much as possible. We then will need to categorize backported collateral evolution and prioritize working on categories that we could likely infer SmPL from. As it stands patches which have been already reduced as much as possible have been annotated by using a four digit prefix. Patches which can likely be split up further remain with a two digit prefix.

Patches that need to be reviewed are under the patches/collateral-evolutions directory. The way backports works -- any directory name can be used to separate patches into any logical form as required. Integration with SmPL can start with inferring SmPL from the few four-digit prefixed patches and subsequently writing a Python tool which would use the SmPL patch to patch kernel code. The first objective is to prove backporting transformations are equivalent by either using the old series of patches for a specific collateral evolution or by using its expression in SmPL form. A proof can be obtained by computing the sha256 delta on the tree with both approaches.

Who's working on this?

Luis will try to dive to look into this August 19th - Sept 13th. With a simple proof of concept feature / driver. Luis hopes to further collaborate with Julia on this between October 16 - December 15.

Infer SmPL for patch hunks healing

If a collateral evolution does not yet have a formal SmPL expression to represent it and if SmPL inference is plausible it'd be great to infer SmPL on the fly as a patch fails to attempt fixing the patch. SmPL inference however is only possible today with a series of similar hunks, at least 2, across different set of offsets. SmPL hunk healing then should not be possible for addressing singular hunk collateral evolutions.

Who's working on this?

Luis hopes to further collaborate with Julia on this between October 16 - December 15.

Using the inverse of SmPL to backport

If Linux kernel development dealt with collateral evolutions with a strict SmPL form it is theoretically possible that one might be able to use the inverse of a patch to backport a collateral evolution. We need to prove / disprove this. Exceptions to this and corner cases should also be inspected.

Who's working on this?

Luis hopes to further collaborate with Julia on this between October 16 - December 15.

DKMS support

Someone seems to want add support for DKMS. Who was that? What benefits does this bring?

Who's working on this ?

No one


Module namespaces

Module namespace can consist of either adding prefixes to existing symbols or modifying the Linux kernel module data structure so that upon request for a symbol it's own namespace can be inspected for its requested namespace. The Linux kernel backports project embraces prefixing backported versions of symbols by prefixing backported versions of symbols with backports_. A macro LINUX_BACKPORT() was added to help with this. For example:

  1. define crc8 LINUX_BACKPORT(crc8)

Andi Kleen introduced the modular structure modification version of module namespaces back in 2007 but Rusty Russell nacked these patches. Because of this we're left with remapping modules to their own namespaces without modifying the in-kernel module structure. It may be possible to somehow provide module namespaces without requiring modifications to the in-kernel module structure, but such mechanisms would end up being hacks. We need to evaluate if we can figure a clever way to support this that is not so error prone or if we want to respinning Andi's patches with the intent to help support kernel backporting.

Motivations

At times Linux distributions may choose, for whatever support-nightmare reasons, to support two versions of a subsystem module and have two separate device drivers use a different version of that subsystem. The reason this is ill-advised is we stand to gain by having development done on the upstream and latest versions of each subsystem and drivers; that backports framework allows you to get that. An example helps to clarify here. Suppose a distribution wants to ship with an old version of mac80211 and have the iwlwifi driver use it, but for ath9k they'd want to ship the newer version of mac80211 and new version of ath9k. This cannot be supported unless module namespaces are created.

Another more reasonable motivation is to avoid regressions with features / symbols being updated only for backported drivers. Non-backported drivers will use the non-backported version of the routine. This should avoid regressions with non-backported drivers using these routines.

Who's working on this?

No one.

88x31.png - This text is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Personal tools