CE Workgroup Device Mainlining Project

This page describes the "Device Mainlining" project of the CE Workgroup.

This purpose of this project is to decrease the amount of out-of-mainline patches for the Linux kernel, in modern consumer electronics products.

This project was proposed at the CE Workgroup Steering Committee meeting in Tokyo, Japan, in June of 2014.

Introduction
As of January, 2015, many devices using Linux have a lot of code that is out-of-mainline.

The following table shows analysis of several mobile phones released in 2014 by major phone vendors. Most of these used 3.4 kernels.

Activities
In terms of overall strategy, the actions being performed for this project fall into 4 categories:
 * communicate and meet to make plans and carry forth objectives
 * identify problems
 * implement solutions
 * train and educate participants

Communicate and meet to make plans and carry forth objectives

 * see meetings section below

Identify problems

 * Survey
 * Conduct a survey to identify obstacles to mainlining - Done, September 2014
 * Present results from survey:
 * Presented [[Media:Overcoming_Obstacles_to_Mainlining-ELCE-2014-with-notes.pdf|"Overcoming Obstacles to Mainlining"]] talk at ELCE 2014
 * Presented [[Media:Overcoming_Obstacles_to_Contributing_to_Linux.pdf|"Overcoming Obstacles to Contributing to Linux"]] talk at ELC 2015
 * Identify most-out-of-mainline areas
 * Analyse source tree from multiple vendors (covering multiple SOCs) for actual products, to see what the delta is in different areas
 * Initial analysis is done, and was presented at BOF in March, 2015
 * More detailed analysis is under way
 * Create white paper describing obstacles
 * [[Media:Overcoming-Obstacles-to-Mainlining-White-Paper-version-0.9.pdf|"Overcoming Obstacles to Mainlining White Paper (v0.9)]] - June 3, 2015
 * Greg Kroah-Hartman gave a talk at Linaro Connect 2015 in Hong Kong (February 2015), describing the out-of-tree problem for ARM SoCs
 * see https://www.youtube.com/watch?v=tE3804cOtXA

Implement solutions

 * Decide on specific projects
 * Identify kernel sub-systems that are the most out-of-tree
 * Identify what the problems are
 * Push patches to mainline to address problem areas found
 * Assist weary maintainers

Train and Educate participants

 * Collect resource for mainlinining training
 * See Kernel Mainlining
 * Convince management to provide funding and/or resources for this work
 * Produce a white paper on obstacles (see above)
 * Determine cost of out-of-tree code, and publish information for managers

Device Mainlining BOF - October 16, 2015

 * Tim Bird held a Birds-of-a-Feather meeting to discuss issues with this during Linux Plumbers in Dusseldorf, Germany, on October 16, 2014
 * Kevin Hillman attended, and provided some good insights into SOC mainline efforts in the past

[should put some more meeting notes here]


 * some key ideas
 * TI got serious about mainlining when Nokia adopted the upstream kernel version over their in-house kernel version, for their phone releases
 * take-away: would be good if Google pushed vendors harder, but they don't have incentive to do so
 * Linaro teams are focused on specific technology areas, not on SoC mainlining
 * With early Android releases, they showed android patches supported on dev boards for various SoCs
 * This was for demonstration and was not production grade Linux
 * Probably this was the closest these trees ever were to mainline

Device Mainlining meeting - March 24, 2015
This meeting will be held Tuesday, March 24, at 11:00 am in the Guadalupe meeting room at the San Jose Marriott.

Attendees: [should list attendees here]

Topics:

Some of the issues I want to discuss are:
 * Tim reports on his SOC research
 * Where are we, really, with product-grade mainline support for SOCs?
 * Based on recent work on my part, I've come to believe we're not nearly as close to mainline support for many, many processors, as we should be. I'm in the middle of doing some research on multiple SOCs used in mobile phones, and it appears that there are all kinds of fundamental problems that make using a mainline kernel on a production phone a fleeting dream.
 * What sub-systems have weakest support in mainline?
 * For each sub-system, is the poor support because of:
 * framework deficiencies
 * DT blockage
 * dependencies on out-of-tree code
 * lack of documentation
 * maintainer bandwidth, or
 * something else?
 * Are there things the Linux Foundation, other industry groups, or companies can do to make interacting with the community easier?
 * Would it be helpful to produce a scorecard, for evaluating where each SOC is
 * Scorecard would have: what subsystems have been mainlined, what's outstanding, etc.
 * Can git tools be developed to automate this analysis?
 * Can Linaro and LF work together on some things?
 * keep a list of "maintainer quirks" - what things individual maintainers like/dislike, insist on, what timing they like (when to submit relative to merge window).

Calls and meetings - Spring/Summer 2015

 * Tim held another meeting Special Interest Group meeting at ELC, to identify areas that need work (and could use CEWG/LF funding)
 * Tim prepared some information about the out-of-tree status of various SOC and discussed this in the meeting
 * April 21, 2015 - Tim presented Obstacles to Mainlining talk at Genivi All-Members-Meeting
 * April 28, 2015 - Conference call was held with Linaro (Tim Bird, Kate Stewart and Mark Brown)
 * different items were discussed, with main goal to be publication of upstream analysis tools
 * Tim Bird met with Ibrahim Hadad on April 28, 2015 in San Jose
 * presented basic goals, got information about Samsung and their practices
 * May 12, 2015 - Conference call with Linaro (Kate Stewart and Mark Brown)
 * had not published upstream-analysis-tools, but discussed possible names, and other topics

LinuxCon North America BOF - August 19, 2015

 * Mark, Kate and Tim held BOF at LCNA
 * Here's the presentation from the BOF: [[media:Device-mainlining-LCNA-BOF-2015-08.pdf]]
 * Notes from the meeting are being written

Obstacles (or problems to overcome)

 * Version gap
 * Dependency on out-of-tree code
 * Inability to test (inability to generalize)
 * Low-quality code
 * Specialized code
 * Immature upstream frameworks
 * No time allocated by management
 * Difficult (or missing) internal approval process
 * Overloaded (or slow) maintainers
 * Lack of perceived business value
 * Rate of hardware churn greater than resources allocated to upstream effort
 * Difficult upstream process
 * Lack of English proficiency
 * Fear of rejection
 * Suppliers providing out-of-tree code
 * No incentive for management (or counter-incentives)

Categories:
 * intrinsic delays in the product pipeline
 * lack of corporate will
 * skill deficiency
 * sourcing woes??

Tools
This section has information about tools provided by this project:

Upstream Analysis Tools
Some tools that have been developed are available at: https://github.com/tbird20d/upstream-analysis-tools The purpose of these tools is multiple:
 * to allow a company to analyze their own upstream status (check their own out-of-tree-ness)
 * to allow a company to score the source from potential suppliers
 * to try to identify areas where multiple companies have code for the same hardware or feature out-of-tree
 * this is intended to help identify where shared work makes sense
 * also, it might help identify a sub-system or framework weakness that needs addressing upstream

Some tools (also part of the release above) can be used for the following:
 * to track the status of contributions from different parties to the Linux kernel over time
 * this is intended to allow an organization to monitor the progress of their mainlining activities
 * to identify what their code is dependent on (commit-dep.py)
 * this is to assess the potential difficulty of mainlining a particular patch, to help in prioritizing mainlining work
 * also, this should help to identify dependencies between commits, to help in prioritizing individual patches

Training and tutorials

 * Kernel Mainlining - main page for learning about kernel mainlining

Coordination pages

 * Mainlining improvement ideas - ideas for how to overcome some of the osbstacles mentioned
 * This mainly covers the non-technical issues, like market forces, institutional barriers, incentives, training, etc.
 * Kernel areas of focus for mainlining - major areas of the kernel where many SoCs have code out-of-tree

Pages for specific processors

 * Qualcomm SOC Mainlining Project
 * Allwinner mainlining effort

Mailing list
See http://lists.linuxfoundation.org/mailman/listinfo/device-mainlining archive: http://lists.linuxfoundation.org/pipermail/device-mainlining/

Mobile product source code repositories
See Phones Processors and Download Sites