Difference between revisions of "Android Mainlining Project"

From eLinux.org
Jump to: navigation, search
(Patch/Feature Status Chart: add thread from ce-android-mainline for binder)
(Patch/Feature Status Chart)
(12 intermediate revisions by 3 users not shown)
Line 30: Line 30:
 
* Linaro blueprint for project: https://blueprints.launchpad.net/linux-linaro/+spec/linaro-kernel-android-upstreaming
 
* Linaro blueprint for project: https://blueprints.launchpad.net/linux-linaro/+spec/linaro-kernel-android-upstreaming
  
== Meetings ==
+
== Updates ==
 +
* [https://lwn.net/Articles/514901/ KS2012: Status of Android upstreaming]
 
* Meeting: [[Android mainlining interest group meeting 2012]]
 
* Meeting: [[Android mainlining interest group meeting 2012]]
 
** Summary: 9:30- 12:00 February 10, 2012 at the Sofitel Hotel in Redwood Shores, California.
 
** Summary: 9:30- 12:00 February 10, 2012 at the Sofitel Hotel in Redwood Shores, California.
Line 87: Line 88:
 
|yes
 
|yes
 
|Tim Bird
 
|Tim Bird
|should be non-controversial (though I'm always surprised)
+
|See [[Mainline Android logger project]] for a list of ideas, issues and a project plan for this feature
See [[Mainline Android logger project]] for a list of ideas, issues and a project plan for this feature
+
 
Also see [http://lkml.indiana.edu/hypermail/linux/kernel/1112.2/index.html#02419 this LKML discussion thread]
 
Also see [http://lkml.indiana.edu/hypermail/linux/kernel/1112.2/index.html#02419 this LKML discussion thread]
Some cleanup patches in linux-next, with more extensive changes to be written, tested, reviewed by Tim in Spring 2012.
+
Some cleanup patches in linux-next, with some changes to avoid hardcoding logging channels (see Linus' tree commit: 10b241991fc)
 +
Tim was planning on doing channel allocation from user-space next, but Arnd may have talked him into doing a fs-based solution at LinuxCon Japan. (6/18/2012).
 
|-
 
|-
 
|wakelocks
 
|wakelocks
 
|Power management locking mechanism to prevent opportunistic suspend
 
|Power management locking mechanism to prevent opportunistic suspend
|not mainlined, as of 2012.03.19, work in progress
+
|parts mainlined, as of 2012.06.18, work in progress
 
|yes
 
|yes
 
|Rafael Wysocki
 
|Rafael Wysocki
|Is important due to impact on board support and drivers by 3rd parties.  Last effort was around "autosleep and wakelocks".
+
|John Stultz wrote (2012-10-11):
See http://lwn.net/Articles/479841/ and this thread: http://thread.gmane.org/gmane.linux.kernel/1249726
+
 
 +
What is left here is the driver wakeup_source enablement to utilize the
 +
upstream work.  Currently the Android tree has a wakelock.h shim that
 +
maps legacy wakelock usage to wakeup sources, but we need to migrate
 +
that to direct wakeup_source usage and push those patches (in a number
 +
of subsystems) upstream.
 +
 
 +
older status: Is important due to impact on board support and drivers by 3rd parties.  Last effort was around "autosleep and wakelocks".
 +
See http://lwn.net/Articles/479841/ and this thread: http://thread.gmane.org/gmane.linux.kernel/1249726.  This code apparently made it into 3.4, with more code to appear in 3.5?
 
|-
 
|-
 
|Android alarm timers
 
|Android alarm timers
Line 112: Line 121:
 
|yes
 
|yes
 
|John Stultz
 
|John Stultz
|Working on fadvise volatile alternative implementation that handles part of the ashmem functionality. However, there are additional aspects of ashmem design that need to be addressed(no tmpfs mounts, atomic create/unlink behavior,etc).
+
|Working on fadvise volatile alternative implementation that handles part of the ashmem functionality. However, there are additional aspects of ashmem design that need to be addressed (no tmpfs mounts, atomic create/unlink behavior,etc).
 
|-
 
|-
 
|network security
 
|network security
Line 126: Line 135:
 
|yes
 
|yes
 
|no one
 
|no one
|Generated a fair amount of discussion on last submission. See http://elinux.org/Android_Binder#obstacles_to_mainlining
+
|John Stultz wrote (2012-10-11):
 +
I think the key item to get binder upstream would be to find non-Android
 +
use cases, where there is community interest in solving a similar
 +
problem, and so far that hasn't been very productive (kdbus was maybe
 +
the closest, but I've not seen too much heavy lifting going on there). 
 +
 
 +
Generated a fair amount of discussion on last submission. See http://elinux.org/Android_Binder#obstacles_to_mainlining
 
See also this thread: http://lists.linuxfoundation.org/pipermail/ce-android-mainline/2012-January/thread.html#42
 
See also this thread: http://lists.linuxfoundation.org/pipermail/ce-android-mainline/2012-January/thread.html#42
 
|-
 
|-
Line 138: Line 153:
 
|Android USB gadget driver
 
|Android USB gadget driver
 
|Support for Android as gadget device from host
 
|Support for Android as gadget device from host
|unknown
 
|? (not sure if needed for basic bringup or not, but would be really painful if missing)
 
|no one?
 
 
|
 
|
 +
|
 +
|Andrzej Pietrasiewicz?, Kyungmin Park?.  Mike Lockwood of Google seems to be involved.
 +
|[[#USB|USB]]
 +
 +
Apparently, some Samsung developers made progress getting most of the Android USB Gadget driver
 +
upstream.  There is an issue related to ccg and changing the sysfs interface to configfs.
 +
 +
ccg = configurable composite gadget.  It was moved to staging/ccg in September 2012.
 +
Apparently, this is related to the Android USB driver, which has multiple USB functions.
 +
 +
ccg is in staging and nobody is working on it. The configs interface requires
 +
some cleanup of the current infrastructure to make it usuable with multiple
 +
UDC, multiple functions etc. There are patches for of the configfs part from
 +
Andrzej
 +
 
|-
 
|-
 
|timed gpio
 
|timed gpio
Line 148: Line 175:
 
|no? (is this needed for basic bringup?)
 
|no? (is this needed for basic bringup?)
 
|Shuah Khan
 
|Shuah Khan
|Enhancing ledtrig-timer with activate_once timer - Shuah working on a patch
+
|A new transient trigger added to led triggers in Linux 3.4 to address the need for one shot timer activation. This driver could be removed once the legacy libraries are changed to use the new transient trigger. https://lkml.org/lkml/2012/4/30/288
 
|-
 
|-
 
|timed output
 
|timed output
Line 155: Line 182:
 
|no? (required by timed_gpio)
 
|no? (required by timed_gpio)
 
|Shuah Khan
 
|Shuah Khan
|Enhancing ledtrig-timer with activate_once timer - Shuah working on a patch
+
|A new transient trigger added to led triggers in Linux 3.4 to address the need for one shot timer activation. This driver could be removed once the legacy libraries are changed to use the new transient trigger. https://lkml.org/lkml/2012/4/30/288
 
|-
 
|-
 
|low memory killer
 
|low memory killer
Line 162: Line 189:
 
|yes (but system might work without it)
 
|yes (but system might work without it)
 
|Anton Vorontsov
 
|Anton Vorontsov
|See https://lkml.org/lkml/2011/12/18/173 for discussion about these patches
+
|Anton said (2012-10-11):
 +
The work is ongoing, and now it's much more than 'low memory killer':
 +
we're rethinking the whole userland-assisted memory management for Linux,
 +
and this also slightly touches volatile ranges idea (but not code).
 +
 
 +
This new infrastructure will be suitable for Android (and it is developed
 +
keeping Android needs in mind), but to get it into upstream we need to
 +
make it generic and suitable for servers, desktops and 'other' embedded
 +
use cases: memory-management folks are very intolerant to any ad-hoc
 +
solutions. There's a lot of design work and discussions are still going on
 +
in LKML, on meetings and privately.
 +
 
 +
Kernel parts are developed as part of linux-vmevent tree:
 +
 
 +
git://git.infradead.org/users/cbou/linux-vmevent.git
 +
http://git.infradead.org/users/cbou/linux-vmevent.git
 +
 
 +
Userland side is developed as part of 'userland LMK' proof-of-concept
 +
daemon, but most probably we'll end up merging it into Android/Java. The
 +
git tree is following:
 +
 
 +
git://git.infradead.org/users/cbou/ulmkd.git
 +
http://git.infradead.org/users/cbou/ulmkd.git
 +
 
 +
older status: See https://lkml.org/lkml/2011/12/18/173 for discussion about these patches
 
|-
 
|-
 
|ram console
 
|ram console
 
|ability to save console output to a reserved ram area for diagnostics on a subsequent boot
 
|ability to save console output to a reserved ram area for diagnostics on a subsequent boot
|in drivers/staging/android, in kernel version 3.3 2012.03.19
+
|in drivers/staging/android, in kernel version 3.3 2012.03.19 ; new system pstore RAM backend is mainlined
 +
waiting for use by Google in Android
 
|no
 
|no
|Shuah Khan
+
|Anton Vorontsov
 
|No lkml submission attempt history found so far. pstore fs interface solves the same problem and is in mainline 3.1.5. fs/pstore  Reference - http://lwn.net/Articles/421297/ After investigating and discussing ram console use cases, it is clear that the proposal to use ram-oops and pstore will not be a complete solution to cover all the cases. Hence, the going forward plan is to pursue ram console inclusion in the mainline. ram console is going through re-write splitting it into twp drivers. Waiting for these changes to get into the 3.3 staging area. For now working with the unofficial patches.
 
|No lkml submission attempt history found so far. pstore fs interface solves the same problem and is in mainline 3.1.5. fs/pstore  Reference - http://lwn.net/Articles/421297/ After investigating and discussing ram console use cases, it is clear that the proposal to use ram-oops and pstore will not be a complete solution to cover all the cases. Hence, the going forward plan is to pursue ram console inclusion in the mainline. ram console is going through re-write splitting it into twp drivers. Waiting for these changes to get into the 3.3 staging area. For now working with the unofficial patches.
 +
 +
Anton Vortontsov moved some functionality from the ram_console into the new "persistent_ram" feature, in fs/pstore/ram_core.c (May 17, 2012 - see commit cddb8751c)
 +
 +
In October 2012, Anton said:
 +
 +
I worked on this item. Everything is in mainline now. It is no longer just
 +
'ramconsole', it is now called "pstore filesystem" and "pstore RAM
 +
backend", placed under fs/pstore/. It can do console, panic and trace
 +
recording.
 +
 +
As far as I know, Google team plans look at it/validate next year.
 +
 +
Some Intel and Google folks and I are jointly maintain the code, the GIT
 +
tree of the project is following:
 +
 +
git://git.infradead.org/users/cbou/linux-pstore.git
 +
http://git.infradead.org/users/cbou/linux-pstore.git
 
|-
 
|-
 
|ion graphics memory driver
 
|ion graphics memory driver
Line 177: Line 246:
 
|Jesse Barker
 
|Jesse Barker
 
|
 
|
 +
|-
 +
|fiq glue code
 +
|Support for use of ARM fast IRQ (FIQ) for debugging
 +
|???
 +
|no
 +
|Anton Vorontsov
 +
|Anton wrote (2012-10-11):
 +
 +
The idea of FIQ debugger is accepted by the community, and as of this
 +
merge window, the core support merged into the mainline kernel. ARM
 +
specific and "restricted KDB command set" patches are still under review,
 +
the plan is to get them into the next kernel release.
 +
 +
As it usually happens during upstreaming process, the small "FIQ glue"
 +
project ended up being a generic facility, so this is now called "NMI
 +
KGDB/KDB debugger", not just ARM FIQ glue code.
 +
 +
The generic driver is now drivers/tty/serial/kgdb_nmi.c.
 +
 +
Further development (i.e. ARM-specific bits, and restricted mode) is going
 +
on in the following GIT tree:
 +
 +
git://git.infradead.org/users/cbou/linux-nmi-kdb.git
 +
http://git.infradead.org/users/cbou/linux-nmi-kdb.git
 
|}
 
|}
  
 +
== USB==
 +
* [http://thread.gmane.org/gmane.linux.usb.general/56244 Android Composite Gadget driver discussion]
 +
* http://lxr.free-electrons.com/source/drivers/staging/ccg/ - Configurable Composite Gadget - reincarnation of Android Composite Gdaget
 
== Progress Chart ==
 
== Progress Chart ==
 
This section is intended to show our progress, by showing the patch set size over time.  With any luck, as
 
This section is intended to show our progress, by showing the patch set size over time.  With any luck, as
we get features into mainline, the difference between the Android kernel and the legacy Linux kernel will shrink.
+
we get features into mainline, the difference between the Android kernel and the mainline Linux kernel will shrink.
 +
 
 +
<div style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #dfdfdf; padding:0 1em 1em 1em; background-color:#ffffcc; align:right; ">
 +
''Note: This table is not up-to-date and is hard to measure anyway.  The inclusion of a lot of material into 3.3 obsoletes the need for a per-patch tracking mechanism (maybe?).  Also, there will always be some difference, due to board support packages and new development areas (like ion), so the goal is not really 0 differences, but just a reduction overall.  I'm not sure if we'll maintain this table or not, but I'm leaving it for now as a placeholder.''
 +
</div>
 +
 
  
 
* diff of 2.6.29 kernel.org tree versus kernel
 
* diff of 2.6.29 kernel.org tree versus kernel

Revision as of 22:09, 26 October 2012

This page is for organizing the Android Mainlining Project. It has information and resources associated with this project.

Goal

The goal of this project is to ultimately mainline all patches required to run the current released version of Android. The purpose of mainlining these patches is 3-fold:

  1. to allow a developer to use the latest released version of the Linux kernel to run an Android system, without requiring patches to their kernel
  2. to make it possible to develop drivers and board support features against either an Android kernel release or a kernel.org kernel release, with little or no modifications or conditional code
  3. to reduce or eliminate the burden of maintaining independent patches from release to release for Android kernel developers

To "mainline" a patch means to have it included in Linus Torvalds kernel.org kernel, in a released (non-rc) version.

Process

[This is a draft section, up for discussion]

Overall:

  • identify all patches/features, and categorize into core or non/core
    • core = feature is required or strongly desired for Android operation on a platform
    • non-core = Most of the Android system can run without the feature

Per feature or patch:

  • research any previous submission feedback
  • incorporate feedback, as appropriate
  • negotiate any interface changes with Google Android team
  • submit updated patches to mainline
  • repeat until accepted

Resources

Updates

People

People who have expressed interest in this project are:

  • Tim Bird
  • John Stultz
  • Paul McKenney
  • Deepak Saxena
  • Arnd Bergmann
  • Thomas Gleixner
  • Arjan Van de Ven
  • Brian Swetland
  • Tetsuyuki Kobayashi
  • Andy Green
  • Victor M. Jaquez
  • Jesse Barker
  • Anton Vorontsov
  • Greg Kroah-Hartman
  • Shuah Khan

roles/expertise

This section has miscellaneous notes on roles, capabilities and expertise of group's members

John Stultz is the owner of the Linaro blueprint for mainlining Android features. Tim Bird is the owner of the CE Workgroup project for mainlining Android features. Deepak and Jesse can help make arrangements for a meeting at Linaro Connect. Tim can help make arrangements for a meeting at Android Builders Summit.

  • John Stultz has worked on POSIX Alarm timers
  • Jesse is working on shared memory buffers related to pmem/CMA/parts of ion
  • Anton Vorontsov is looking at the lowmemory killer
  • Greg has put some Android patches into mainline (under drivers/staging/android) previously
    • Greg put some Android patches in mainline under drivers/staging/android in Dec. 2011
  • Paul McKenney - kicking around ideas for dealing with wakelocks single global lock (dec. 2011)

Plans

  • Update this site with information on latest patch status - ongoing, by anyone
    • [FIXTHIS - add sections needing a status or status update]

Patch/Feature Status Chart

Feature/Patch Description Status Part of core? Owner/Interested parties Notes
logger kernel support for Android system logging in drivers/staging/android, in kernel version 3.3 2012.03.19 yes Tim Bird See Mainline Android logger project for a list of ideas, issues and a project plan for this feature

Also see this LKML discussion thread Some cleanup patches in linux-next, with some changes to avoid hardcoding logging channels (see Linus' tree commit: 10b241991fc) Tim was planning on doing channel allocation from user-space next, but Arnd may have talked him into doing a fs-based solution at LinuxCon Japan. (6/18/2012).

wakelocks Power management locking mechanism to prevent opportunistic suspend parts mainlined, as of 2012.06.18, work in progress yes Rafael Wysocki John Stultz wrote (2012-10-11):

What is left here is the driver wakeup_source enablement to utilize the upstream work. Currently the Android tree has a wakelock.h shim that maps legacy wakelock usage to wakeup sources, but we need to migrate that to direct wakeup_source usage and push those patches (in a number of subsystems) upstream.

older status: Is important due to impact on board support and drivers by 3rd parties. Last effort was around "autosleep and wakelocks". See http://lwn.net/Articles/479841/ and this thread: http://thread.gmane.org/gmane.linux.kernel/1249726. This code apparently made it into 3.4, with more code to appear in 3.5?

Android alarm timers Timers that count down during suspended operation, and can wake from suspend Partial: Posix alarm timers were mainlined in kernel version 2.6.38 - see https://lwn.net/Articles/429925/ yes John Stultz Mending patches to convert Android Alarm Timers to utilize the upstreamed alarm timer work are still pending.
ashmem Shared memory implementation that allows unpinned pages to be marked, which can be dropped by the kernel under memory pressure in drivers/staging/android, in kernel version 3.3 2012.03.19 yes John Stultz Working on fadvise volatile alternative implementation that handles part of the ashmem functionality. However, there are additional aspects of ashmem design that need to be addressed (no tmpfs mounts, atomic create/unlink behavior,etc).
network security special permission checks for secure access to network operations not mainlined ? (can run without it, but network security won't be enforced) no one May be very difficult to mainline, as the code is extremely Android-specific with hardcoded GIDs and capabilities.
binder Android inter-process communication mechanism in drivers/staging/android, in kernel version 3.3 2012.03.19 yes no one John Stultz wrote (2012-10-11):

I think the key item to get binder upstream would be to find non-Android use cases, where there is community interest in solving a similar problem, and so far that hasn't been very productive (kdbus was maybe the closest, but I've not seen too much heavy lifting going on there).

Generated a fair amount of discussion on last submission. See http://elinux.org/Android_Binder#obstacles_to_mainlining See also this thread: http://lists.linuxfoundation.org/pipermail/ce-android-mainline/2012-January/thread.html#42

Android pmem driver manages large (1-16+MB) contiguous physical memory regions to be shared between userspace and kernel drivers (dsp, gpu etc.) Removed - no longer in use yes Shuah Khan Investigating existing alternatives. Can CMA (Contiguous Memory Allocator) fill the need? Based on recent lkml discussion, pmem.c is no longer used and will be removed from Android release. Reference: https://lkml.org/lkml/2012/1/23/183 - Patch to remove pmem.c from 3.3-rc1 staging area has been accepted.
Android USB gadget driver Support for Android as gadget device from host Andrzej Pietrasiewicz?, Kyungmin Park?. Mike Lockwood of Google seems to be involved. USB

Apparently, some Samsung developers made progress getting most of the Android USB Gadget driver upstream. There is an issue related to ccg and changing the sysfs interface to configfs.

ccg = configurable composite gadget. It was moved to staging/ccg in September 2012. Apparently, this is related to the Android USB driver, which has multiple USB functions.

ccg is in staging and nobody is working on it. The configs interface requires some cleanup of the current infrastructure to make it usuable with multiple UDC, multiple functions etc. There are patches for of the configfs part from Andrzej

timed gpio perform gpio operations as a result of specified timeouts - mainly used to support vibrate feature. in drivers/staging/android, in kernel version 3.3 2012.03.19 no? (is this needed for basic bringup?) Shuah Khan A new transient trigger added to led triggers in Linux 3.4 to address the need for one shot timer activation. This driver could be removed once the legacy libraries are changed to use the new transient trigger. https://lkml.org/lkml/2012/4/30/288
timed output seems to just be a class for timed gpio items in drivers/staging/android, in kernel version 3.3 2012.03.19 no? (required by timed_gpio) Shuah Khan A new transient trigger added to led triggers in Linux 3.4 to address the need for one shot timer activation. This driver could be removed once the legacy libraries are changed to use the new transient trigger. https://lkml.org/lkml/2012/4/30/288
low memory killer feature to manage application lifecycle in low memory conditions in drivers/staging/android, in kernel version 3.3 2012.03.19 yes (but system might work without it) Anton Vorontsov Anton said (2012-10-11):

The work is ongoing, and now it's much more than 'low memory killer': we're rethinking the whole userland-assisted memory management for Linux, and this also slightly touches volatile ranges idea (but not code).

This new infrastructure will be suitable for Android (and it is developed keeping Android needs in mind), but to get it into upstream we need to make it generic and suitable for servers, desktops and 'other' embedded use cases: memory-management folks are very intolerant to any ad-hoc solutions. There's a lot of design work and discussions are still going on in LKML, on meetings and privately.

Kernel parts are developed as part of linux-vmevent tree:

git://git.infradead.org/users/cbou/linux-vmevent.git
http://git.infradead.org/users/cbou/linux-vmevent.git

Userland side is developed as part of 'userland LMK' proof-of-concept daemon, but most probably we'll end up merging it into Android/Java. The git tree is following:

git://git.infradead.org/users/cbou/ulmkd.git
http://git.infradead.org/users/cbou/ulmkd.git

older status: See https://lkml.org/lkml/2011/12/18/173 for discussion about these patches

ram console ability to save console output to a reserved ram area for diagnostics on a subsequent boot in drivers/staging/android, in kernel version 3.3 2012.03.19 ; new system pstore RAM backend is mainlined

waiting for use by Google in Android

no Anton Vorontsov No lkml submission attempt history found so far. pstore fs interface solves the same problem and is in mainline 3.1.5. fs/pstore Reference - http://lwn.net/Articles/421297/ After investigating and discussing ram console use cases, it is clear that the proposal to use ram-oops and pstore will not be a complete solution to cover all the cases. Hence, the going forward plan is to pursue ram console inclusion in the mainline. ram console is going through re-write splitting it into twp drivers. Waiting for these changes to get into the 3.3 staging area. For now working with the unofficial patches.

Anton Vortontsov moved some functionality from the ram_console into the new "persistent_ram" feature, in fs/pstore/ram_core.c (May 17, 2012 - see commit cddb8751c)

In October 2012, Anton said:

I worked on this item. Everything is in mainline now. It is no longer just 'ramconsole', it is now called "pstore filesystem" and "pstore RAM backend", placed under fs/pstore/. It can do console, panic and trace recording.

As far as I know, Google team plans look at it/validate next year.

Some Intel and Google folks and I are jointly maintain the code, the GIT tree of the project is following:

git://git.infradead.org/users/cbou/linux-pstore.git
http://git.infradead.org/users/cbou/linux-pstore.git
ion graphics memory driver graphics memory drivers thingie not mainlined yes (for 4.0 and later?) Jesse Barker
fiq glue code Support for use of ARM fast IRQ (FIQ) for debugging ??? no Anton Vorontsov Anton wrote (2012-10-11):

The idea of FIQ debugger is accepted by the community, and as of this merge window, the core support merged into the mainline kernel. ARM specific and "restricted KDB command set" patches are still under review, the plan is to get them into the next kernel release.

As it usually happens during upstreaming process, the small "FIQ glue" project ended up being a generic facility, so this is now called "NMI KGDB/KDB debugger", not just ARM FIQ glue code.

The generic driver is now drivers/tty/serial/kgdb_nmi.c.

Further development (i.e. ARM-specific bits, and restricted mode) is going on in the following GIT tree:

git://git.infradead.org/users/cbou/linux-nmi-kdb.git
http://git.infradead.org/users/cbou/linux-nmi-kdb.git

USB

Progress Chart

This section is intended to show our progress, by showing the patch set size over time. With any luck, as we get features into mainline, the difference between the Android kernel and the mainline Linux kernel will shrink.

Note: This table is not up-to-date and is hard to measure anyway. The inclusion of a lot of material into 3.3 obsoletes the need for a per-patch tracking mechanism (maybe?). Also, there will always be some difference, due to board support packages and new development areas (like ion), so the goal is not really 0 differences, but just a reduction overall. I'm not sure if we'll maintain this table or not, but I'm leaving it for now as a placeholder.


  • diff of 2.6.29 kernel.org tree versus kernel
Kernel Version Files changed insertions deletions hunks bytes in diff
2.6.29 187 123506 0 187 3291827