Documentation/backports/hacking/todo

From Driver Backports Wiki
(Difference between revisions)
Jump to: navigation, search
(re-order todo list)
(Kernel integration completed)
(4 intermediate revisions by one user not shown)
Line 13: Line 13:
 
No one.
 
No one.
  
= Non modular backport support =
+
= Module namespaces =
  
As it stands the backports project only supports providing backport updates of modules. That is your system gets updated drivers but if a core kernel component that is not modular is updated we can't provide an update for it unless we can technically extract that core kernel feature, map it in modular form and then have the backported version of the modules use a new namespace. The problem with doing this happens when a core kernel component is dependent upon another core kernel component that cannot be backported or extracted out.
+
Module namespace can consist of either adding prefixes to existing symbols or modifying the Linux kernel module data structure so that upon request for a symbol it's own namespace can be inspected for its requested namespace. The Linux kernel backports project embraces prefixing backported versions of symbols by prefixing backported versions of symbols with ''backports_''. A macro LINUX_BACKPORT() was added to help with this. For example:
  
There is hope however. The idea consists to not backport live kernels given that ksplice seems limited to what can be done for backporting purposes but instead to allow Linux distributions to pick a stable kernel, pick a backports release and have it merge into that kernel release. Typically Linux distributions cherry pick patches and deal with merge conflicts manually. The idea is to the stable kernel as a vanilla release and deal with the integration of new backported components using the backports framework. Just as you would build a kernel through ''make menuconfig'' upon kernel configuration time you'd then have present your vanilla kernel components and also backported features through a menu subsection. You'd then be able to build backported features as in-kernel or modular just as you would typically build a vanilla kernel.
+
<tt>
 +
  #define crc8 LINUX_BACKPORT(crc8)
 +
</tt>
  
See commit [https://git.kernel.org/cgit/linux/kernel/git/mcgrof/backports.git/commit/?id=0935deabd1c7f95ab0ed07dd40291022c7822913 0935dea] for the hints supplied on how to deal with backporting core kernel features. The idea is you should be able to modify the in-kernel's init routine, add our own backport_init() in the series of init calls and throw onto it what we need from future kernels.
+
Andi Kleen introduced the modular structure modification version of [http://thread.gmane.org/gmane.linux.network/78674 module namespaces] back in 2007 but [http://article.gmane.org/gmane.linux.kernel/606885 Rusty Russell nacked these patches]. Because of this we're left with remapping modules to their own namespaces without modifying the in-kernel module structure. It '''may''' be possible to somehow provide module namespaces without requiring modifications to the in-kernel module structure, but such mechanisms would end up being hacks. We need to evaluate if we can figure a clever way to support this that is not so error prone or if we want to respinning Andi's patches with the intent to help support kernel backporting.
  
This likely will require a separate script other than gentree.py, perhaps mergetree.py.
+
== Original motivation for module namespaces ==
  
= Motivation =
+
Here's a brain dump from Andi about the original motivation behind module namespaces:
  
If possible this solution should provide an alternative for users / Linux distributions to cherry pick changes from a future kernel down to older kernels by providing full frameworks / subsystems. One clear benefit is the time reduced to provide a backport and complete and clear separation against what is on a vanilla upstream stable kernel.
+
The original intent was to provide a cleaner export symbol API.
  
== Who's working on this? ==
+
There are roughly two classes of exported symbols:
  
Luis will try to dive to look into this August 19th - Sept 13th. With a simple proof of concept feature / driver.
+
- Generally usable, well designed, interfaces intended to be used by multiple
 +
modules (e.g. functions for drivers to register themselves or
 +
library functions)
  
= SmPL integration =
+
- Special purpose exports that are only really for a single
 +
module, but are not intended as a general interface. These interfaces
 +
are usually not really clean and reusable.
  
Typically patches may fail if hunk offsets are off by the allowed offset. We upkeep hunk offsets placements up to date by refreshing patches. We used to use quit refresh to refresh patch offsets but today we simply extract the new delta after application of a patch in which hunk offsets were off. At times however patches can fail to apply due to contexual changes within the patch. We are considering using SmPL to help with this.
+
The idea was to mark the later special purpose exports with the name of the
 +
module that is supposed to them. They wouldn't be available
 +
to everybody else and wouldn't become part of the general
 +
kernel module API.
  
Its plausible that if you're hacking on the Linux kernel that you may have not yet heard of [http://coccinelle.lip6.fr/ coccinelle] or [http://coccinelle.lip6.fr/sp.php SmPL patches]. Truth is SmPL is already widely used across many subsystem in the Linux kernel and SmPL's usage even extends outside of the Linux kernel. The page that tries to document the [http://coccinelle.lip6.fr/impact_linux.php impact of SmPL on the Linux kernel] is clearly out of date and likely just needs to be automated with ''git grep''. The idea behind the need for SmPL came about studying and formalizing the concept of [http://coccinelle.lip6.fr/ce.php collateral evolutions] on the Linux kernel. Luis elaborates on his blog on how this can be used to help [http://www.do-not-panic.com/2012/08/optimizing-backporting-collateral.html optimize collateral evolutions]. While still theoretical, if we want to make backporting more efficient we need to start using SmPL soon. The work should consist of properly supplying expressions that summarize backporting collateral evolutions from patches from which SmPL could likely deduced or written manually. In order accomplish this each collateral evolution must first be atomically reduced as much as possible. We then will need to categorize backported collateral evolution and prioritize working on categories that we could likely infer SmPL from. As it stands patches which have been already reduced as much as possible have been annotated by using a four digit prefix. Patches which can likely be split up further remain with a two digit prefix.
+
== Motivations ==
  
Patches that need to be reviewed are under the [https://git.kernel.org/cgit/linux/kernel/git/mcgrof/backports.git/tree/patches/collateral-evolutions patches/collateral-evolutions] directory. The way backports works -- any directory name can be used to separate patches into any logical form as required. Integration with SmPL can start with inferring SmPL from the few four-digit prefixed patches and subsequently writing a Python tool which would use the SmPL patch to patch kernel code. The first objective is to prove backporting transformations are equivalent by either using the old series of patches for a specific collateral evolution or by using its expression in SmPL form. A proof can be obtained by computing the sha256 delta on the tree with both approaches.
+
At times Linux distributions may choose, for whatever support-nightmare reasons, to support two versions of a subsystem module and have two separate device drivers use a different version of that subsystem. The reason this is ill-advised is we stand to gain by having development done on the upstream and latest versions of each subsystem and drivers; that backports framework allows you to get that. An example helps to clarify here. Suppose a distribution wants to ship with an old version of mac80211 and have the iwlwifi driver use it, but for ath9k they'd want to ship the newer version of mac80211 and new version of ath9k. This cannot be supported unless module namespaces are created.
  
== Who's working on this? ==
+
Another more reasonable motivation is to avoid regressions with features / symbols being updated only for backported drivers. Non-backported drivers will use the non-backported version of the routine. This should avoid regressions with non-backported drivers using these routines.
 
+
Luis will try to dive to look into this August 19th - Sept 13th. With a simple proof of concept feature / driver. Luis hopes to further collaborate with Julia on this between October 16 - December 15.
+
 
+
= Infer SmPL for patch hunks healing =
+
 
+
If a collateral evolution does not yet have a formal SmPL expression to represent it and if SmPL inference is plausible it'd be great to infer SmPL on the fly as a patch fails to attempt fixing the patch. SmPL inference however is only possible today with a series of similar hunks, at least 2, across different set of offsets. SmPL hunk healing then should not be possible for addressing singular hunk collateral evolutions.
+
  
 
== Who's working on this? ==
 
== Who's working on this? ==
  
Luis hopes to further collaborate with Julia on this between October 16 - December 15.
+
No one.
 
+
= Using the inverse of SmPL to backport =
+
 
+
If Linux kernel development dealt with collateral evolutions with a strict SmPL form it is theoretically possible that one might be able to use the inverse of a patch to backport a collateral evolution. We need to prove / disprove this. Exceptions to this and corner cases should also be inspected.
+
 
+
== Who's working on this? ==
+
 
+
Luis hopes to further collaborate with Julia on this between October 16 - December 15.
+
  
 
= DKMS support =
 
= DKMS support =
  
 
Someone seems to want add support for [https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support DKMS]. Who was that? What benefits does this bring?
 
Someone seems to want add support for [https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support DKMS]. Who was that? What benefits does this bring?
 +
Isn't DKMS for proprietary modules? If so we don't want this support. We don't want proprietary drivers using the Linux kernel backports framework.
  
 
== Who's working on this ? ==
 
== Who's working on this ? ==
  
 
No one
 
No one
 
 
= Module namespaces =
 
 
Module namespace can consist of either adding prefixes to existing symbols or modifying the Linux kernel module data structure so that upon request for a symbol it's own namespace can be inspected for its requested namespace. The Linux kernel backports project embraces prefixing backported versions of symbols by prefixing backported versions of symbols with ''backports_''. A macro LINUX_BACKPORT() was added to help with this. For example:
 
 
<tt>
 
#define crc8 LINUX_BACKPORT(crc8)
 
</tt>
 
 
Andi Kleen introduced the modular structure modification version of [http://thread.gmane.org/gmane.linux.network/78674 module namespaces] back in 2007 but [http://article.gmane.org/gmane.linux.kernel/606885 Rusty Russell nacked these patches]. Because of this we're left with remapping modules to their own namespaces without modifying the in-kernel module structure. It '''may''' be possible to somehow provide module namespaces without requiring modifications to the in-kernel module structure, but such mechanisms would end up being hacks. We need to evaluate if we can figure a clever way to support this that is not so error prone or if we want to respinning Andi's patches with the intent to help support kernel backporting.
 
 
== Motivations ==
 
 
At times Linux distributions may choose, for whatever support-nightmare reasons, to support two versions of a subsystem module and have two separate device drivers use a different version of that subsystem. The reason this is ill-advised is we stand to gain by having development done on the upstream and latest versions of each subsystem and drivers; that backports framework allows you to get that. An example helps to clarify here. Suppose a distribution wants to ship with an old version of mac80211 and have the iwlwifi driver use it, but for ath9k they'd want to ship the newer version of mac80211 and new version of ath9k. This cannot be supported unless module namespaces are created.
 
 
Another more reasonable motivation is to avoid regressions with features / symbols being updated only for backported drivers. Non-backported drivers will use the non-backported version of the routine. This should avoid regressions with non-backported drivers using these routines.
 
 
== Who's working on this? ==
 
 
No one.
 
  
 
[[File:88x31.png‎]] - This text is licensed under a [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons Attribution-ShareAlike 3.0 Unported License].
 
[[File:88x31.png‎]] - This text is licensed under a [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons Attribution-ShareAlike 3.0 Unported License].

Revision as of 18:19, 17 November 2014

The Linux kernel backports TODO list.

Contents

make localmodconfig support

The Linux kernel has make localmodconfig support, random users shouldn't be expected to know what modules they need, so in order to allow users to get only the drivers they need built we'd need something similar to make localmodconfig. There are a few ways to do this. The best way would be to get upstream a mapping of a kernel object of a device to list its respective Kconfig. We'd also need a map of Kconfig changes over time as the kernel advances so that if a user is on an older system their old kobject Kconfig could be mapped to a new one. This should require changes upstream.

Other simpler solutions are to rely on 'lspci -k' output and map this to respective device drivers but this solution would be PCI bus specific. We should review how the upstream make localmodconfig currently works and improve upon that to enable support for it on backports.

Who's working on this?

No one.

Module namespaces

Module namespace can consist of either adding prefixes to existing symbols or modifying the Linux kernel module data structure so that upon request for a symbol it's own namespace can be inspected for its requested namespace. The Linux kernel backports project embraces prefixing backported versions of symbols by prefixing backported versions of symbols with backports_. A macro LINUX_BACKPORT() was added to help with this. For example:

 #define crc8 LINUX_BACKPORT(crc8)

Andi Kleen introduced the modular structure modification version of module namespaces back in 2007 but Rusty Russell nacked these patches. Because of this we're left with remapping modules to their own namespaces without modifying the in-kernel module structure. It may be possible to somehow provide module namespaces without requiring modifications to the in-kernel module structure, but such mechanisms would end up being hacks. We need to evaluate if we can figure a clever way to support this that is not so error prone or if we want to respinning Andi's patches with the intent to help support kernel backporting.

Original motivation for module namespaces

Here's a brain dump from Andi about the original motivation behind module namespaces:

The original intent was to provide a cleaner export symbol API.

There are roughly two classes of exported symbols:

- Generally usable, well designed, interfaces intended to be used by multiple modules (e.g. functions for drivers to register themselves or library functions)

- Special purpose exports that are only really for a single module, but are not intended as a general interface. These interfaces are usually not really clean and reusable.

The idea was to mark the later special purpose exports with the name of the module that is supposed to them. They wouldn't be available to everybody else and wouldn't become part of the general kernel module API.

Motivations

At times Linux distributions may choose, for whatever support-nightmare reasons, to support two versions of a subsystem module and have two separate device drivers use a different version of that subsystem. The reason this is ill-advised is we stand to gain by having development done on the upstream and latest versions of each subsystem and drivers; that backports framework allows you to get that. An example helps to clarify here. Suppose a distribution wants to ship with an old version of mac80211 and have the iwlwifi driver use it, but for ath9k they'd want to ship the newer version of mac80211 and new version of ath9k. This cannot be supported unless module namespaces are created.

Another more reasonable motivation is to avoid regressions with features / symbols being updated only for backported drivers. Non-backported drivers will use the non-backported version of the routine. This should avoid regressions with non-backported drivers using these routines.

Who's working on this?

No one.

DKMS support

Someone seems to want add support for DKMS. Who was that? What benefits does this bring? Isn't DKMS for proprietary modules? If so we don't want this support. We don't want proprietary drivers using the Linux kernel backports framework.

Who's working on this ?

No one

88x31.png - This text is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Personal tools