Difference between revisions of "Kernel areas of focus for mainlining"

From eLinux.org
Jump to: navigation, search
(Media Stream processors)
 
(57 intermediate revisions by 5 users not shown)
Line 1: Line 1:
This page has a list of areas of the kernel where multiple SoCs have code out-of-tree, or that appear to need some work in order to accommodate contribution of device-related software.
+
This page has a list of areas of the kernel where multiple SoCs have code out-of-tree, or that appear to need some work in order to accommodate contribution of device-related software. There is also a maling list for discussion of these issues at http://lists.linuxfoundation.org/mailman/listinfo/device-mainlining
 +
 
  
 
= Overview =
 
= Overview =
The following table shows major out-of-tree areas of code, for mobile phone products based on the 3.4 Linux kernel.
+
The following table shows major out-of-tree areas of code, for mobile phone products based on the 3.4 Linux kernel. For
 +
sources see [[Phones Processors and Download Sites]].
 
Although this kernel is quite old, it was the one currently shipping in phones in late 2014 and early 2015, when this
 
Although this kernel is quite old, it was the one currently shipping in phones in late 2014 and early 2015, when this
 
analysis was done.  It demonstrates the major technical areas where vendors used out-of-tree kernel code, for that
 
analysis was done.  It demonstrates the major technical areas where vendors used out-of-tree kernel code, for that
Line 48: Line 50:
 
= Technical areas =
 
= Technical areas =
 
Here are some lists of things to get mainlined, in different technical areas:
 
Here are some lists of things to get mainlined, in different technical areas:
 +
  
 
== Media ==
 
== Media ==
Line 56: Line 59:
 
action item:  Find out if any vendors have the same IP blocks for wireless
 
action item:  Find out if any vendors have the same IP blocks for wireless
 
Here are the findings for 3.4:
 
Here are the findings for 3.4:
* LG/G3/qcom
+
* LG/G3/qcom - Broadcom 4330/4334/43341/4335/4339
* Samsung/Galaxy 4/exynos
+
** source include drivers/net/wireless/bcmdhd
 +
*** this code is organized differently from the Acer bcmdhd code
 +
 
 +
* Samsung/Galaxy 4/exynos - Broadcom 4335
 +
** source includes drivers/net/wireless/bcmdhd
 +
** source includes drivers/net/wireless/ipsecdrvtl
 +
 
 
* Acer/E2/mediatek - Broadcom 4329/30 wireless chipsets
 
* Acer/E2/mediatek - Broadcom 4329/30 wireless chipsets
 
** source includes drivers/net/wireless/bcmdhd
 
** source includes drivers/net/wireless/bcmdhd
Line 64: Line 73:
 
*** code under this tree is not in linux format (it has vendor style)
 
*** code under this tree is not in linux format (it has vendor style)
 
*** need to see if this is some mediatek on-chip thing, and Acer added external broadcom chip for E2
 
*** need to see if this is some mediatek on-chip thing, and Acer added external broadcom chip for E2
* Asus/Zenfone 6/Atom
+
 
 +
* Asus/Zenfone 6/Atom - Broadcom 4329/30
 +
** source include drivers/net/wireless/bcmdhd
 +
 
 
* Huawei/P6/Hisilicon
 
* Huawei/P6/Hisilicon
* Sony/Z3/Qcom
+
** defines CONFIG_WL12XX=m
 +
 
 +
* Sony/Z3/Qcom -
 +
** Sony has historically used broadcom 4339 and qcom 3660 and qcom 3680.
 +
** Sony did internal testing of the brcmfmac driver on a phone using a 3.4 kernel with the BCM 4339 chipset
 +
*** Results: 1) the chipset appeared to be working, 2) the Wi-Fi can be enabled/disabled from the UI, 3) STA connections worked 4) WiFi direct did not work, 5) Wi-Fi diplay did not work 6) there were hangs on rmmod
 +
*** A driver from the automatic wireless backport system was used: see http://drvbp1.linux-foundation.org/~mcgrof/rel-html/backports/
 +
*** See this page: [[Sony Xperia Wireless backport test]] for more details
 +
 
 +
Random notes:
 +
* Google got burned in early Android - TI had a bad wireless implementation (it was cross-platform and didn't work well in Linux).  Google now integrates broadcom code themselves, at drivers/net/wireless/brcmdhd, and requires current Android vendors to use broadcom.  This explains why everyone has it.
 +
* broadcom driver in mainline is a drivers/net/wireless/brcm80211/*
 +
** this is the one people should try to use and extend if it doesn't work
 +
 
 +
Possible strategy:
 +
* backport brcm80211 code to 3.18 or 3.14.  These are the versions of kernels likely to be used in the next generation of Android products.  Having the brcm80211 code available in that version means that phone vendors can test the driver.  If it's close to product grade, they can help improve it - if not, then they can report the issues upstream.
 +
** Is Google or someone else already doing this - AI - check 3.18 src from SoC vendor for brcm80211 code
 +
 
 +
=== action items ===
 +
* try brcm80211 driver on 3.14 Android device
 +
** identify a 3.14 Android device using a broadcom chipset for wifi
 +
* try brcm80211 driver on dev board running mainline
 +
** identify a dev board running mainline
 +
** make sure the board can run the brcm80211 driver
 +
** test to make sure all required features work
 +
*** figure out what the required features are (is there a wireless test suite?)
 +
* try brcm80211 driver on phone hardware (Xperia Z3) running mainling top-of-tree (in progress)
  
 
== Sound ==
 
== Sound ==
 +
* HDMI integration with graphics for EDID and hotplug information.
  
 
== Input ==
 
== Input ==
 +
 +
== MD ==
 +
 +
* [https://www.codeaurora.org/cgit/quic/la/kernel/msm-3.18/tree/drivers/md/dm-req-crypt.c?h=aosp/android-3.10&id=00e6ac21c6c2b1b0a83cf806daf74d1b9e30767e|dm-req-crypt] Optimisations to make dm-crypt play better with hardware crypto modules.
 +
** Worked on at Linaro, req based approach not going to fly upstream.
 +
** Basic idea is to use large blocks for crypto but the block layer really doesn't want to use anything other than PAGE_SIZE as the size, will require changes in the block core.
 +
** Also working on optimising the crypto layer to be more efficient with smaller block sizes.
 +
 +
== Sensors ==
 +
 +
The [http://developer.android.com/guide/topics/sensors/sensors_overview.html Android sensor stack] is one of these places where vendors are expected to put in their own sensor library providing some middleware (in userspace) to provide sensor input. Some vendors will utilize IIO (industrial I/O, <tt>drivers/iio</tt>) as their backend for this, but several still have non-upstream custom char devices to provide the sensor input. The Android x86 project has [http://git.android-x86.org/?p=platform/hardware/libsensors.git;a=blob;f=iio-sensors.cpp;hb=HEAD some initial work] on IIO-based sensor input, providing polled (non-buffered) sensory data, but it is not part of the official Android git/repo forest.
 +
 +
Migrating custom sensor driver code to use IIO and provide an IIO library by default in upstream android can advance development to address this problem.
 +
 +
* LG/G3/qcom -
 +
* Samsung/Galaxy 4/exynos -
 +
* Acer/E2/mediatek -
 +
* Asus/Zenfone 6/Atom -
 +
* Huawei/P6/Hisilicon -
 +
* Sony/Z3/qcom - older qualcomms have a custom kernelspace driver but could be migrated to IIO, the newer ones utilize a special sensor processor, the communication to this coprocessor is in userspace and presented directly to Android instead of going through the kernel and IIO, these could also be handled in kernelspace but require the off-chip communication to be handled in-kernel first.
  
 
== Camera ==
 
== Camera ==
  
 
== GPU ==
 
== GPU ==
 +
* HDMI integration with sound for EDID and hotplug information.
 +
 +
=== Mali ===
 +
 +
The [https://en.wikipedia.org/wiki/Mali_%28GPU%29 Mali GPU] is used on SoCs from Allwinner, Amlogic, CSR, InfoTM, HiSilicon, Leadcore, MediaTek, NetLogic, Nufront, Rockchip, Samsung, Socle-Tech, Spreadtrum, ST-Ericsson, ST Microelectronics, Telechips and Wondermedia (according to [https://en.wikipedia.org/wiki/Mali_%28GPU%29 the Wikipedia page]).
 +
Sudip Mukherjee suggested adding Mali 400 for ODROID-U3 to staging.  The Odriod-U3 has lots of stuff
 +
upstream and uses a Mali 400 and Exynos 4412.
 +
 +
GregKH says he won't take a DRM kernel driver into staging if it doesn't have an open user-space driver.
 +
The Mali driver from ARM is apparently closed, and they appear to have no intention to open it.  As it stands,
 +
Greg would not take the driver into staging.
 +
 +
It was suggested that possibly the [http://limadriver.org/ Lima driver] could be used to fulfill the requirement for an open-source
 +
user space.  However, the Lima driver project seems to not be active any more.
 +
 +
Heiko Stuebner said:
 +
An additional point would be the possible (in-)stability of the userspace
 +
interface. While I could sucessfully use a r6 midgard kernel driver with a r5
 +
userspace lib yesterday on my Rockchip Chromebook using Debian, I don't think
 +
that is guaranteed to work everytime or that special care is taken for the
 +
kernel driver to prevent needing upgrades in lockstep.
 +
 +
And that is not even taking into account, that the mali userspace libs seem to
 +
be implementator-specific somehow too. For example for Rockchip I currently get
 +
my libs from the ChromeOS tree [0] which can talk to X11 [although not plasma5
 +
it seems] and ARM seems to provide a variant for the firefly board using the
 +
legacy fbdev interface of the 3.10-based Android kernel. And then there are
 +
different variants for Samsung, etc too.
 +
 +
...
 +
 +
[The Lima project is] probably not entirely defunct, but still on life-support:
 +
 +
For one, if you click on the "Source"-link on limadriver.org you get the
 +
gitorious "we're migrating old projects to archive.org" page.
 +
 +
And secondly there is supposed to be an actual mesa driver for mali400 and
 +
even bits for the midgard gpus (Mali-T) around, but limited to Luc's harddrive
 +
for now [1].
 +
 +
* [0] https://github.com/mmind/mali-driver
 +
* [1] http://libv.livejournal.com/27461.html
 +
 +
=== Vivante ===
 +
The [https://en.wikipedia.org/wiki/Vivante_Corporation#Products Vivante GPU] is used in Intel Atom, CuBox, Rockchip, Actions Semiconductor, Freescale, HiSilicon and Marvell SoCs, according to [https://en.wikipedia.org/wiki/Vivante_Corporation#Products the Wikipedia page].
 +
 +
The [https://github.com/etnaviv/etna_viv Etnaviv project] has [https://lwn.net/Articles/639270/ proposed] initial GPU driver support to the upstream kernel.
 +
 +
The Linux kernel driver developed by the [https://github.com/etnaviv/etna_viv Etnaviv project] has been
 +
[https://lwn.net/Articles/672344/ merged into Linux 4.5]. Lucas Stach gave a [https://fosdem.org/2016/schedule/event/etnaviv/ presentation on etnaviv at FOSDEM 2016].
 +
See the [https://fosdem.org/2016/schedule/event/etnaviv/attachments/slides/1206/export/events/attachments/etnaviv/slides/1206/FOSDEM2016_Etnaviv.pdf slides]
 +
and watch the [http://video.fosdem.org/2016/k3401/ video] (not available yet).
 +
 +
=== Adreno ===
 +
Rob Clark has been working on freedreno, an open source driver for the Adreno hardware used on QualComm SoCs.
 +
He gave a presentation on the status at ELC 2015. See the [http://people.freedesktop.org/~robclark/freedreno-elc-2015.html slides],
 +
[https://www.youtube.com/watch?v=ruHJnfeZlgs video]
 +
and [http://elinux.org/images/c/c1/07_Clark--edit_150317.pdf showcase poster].
 +
 +
Rob reported (by private e-mail in September 2015) that there is now the GLES 3.0 code has support for mesa 11.
 +
(Much more detail about the status of the project (as of March 2015) is available in the slides.)
 +
 +
=== PowerVR ===
 +
 +
(eLinux has a wiki page about how to [[Create Free Software PowerVR GPU driver]])
 +
 +
The [https://en.wikipedia.org/wiki/PowerVR PowerVR] GPU by Imagination Technologies is used by a host of embedded SoCs including Texas Instruments OMAP, DaVinci and DM3730, MediaTek, some Intel, Apple iPhone, Samsung S5P and Exynos 3110 and 5410, Actions Semiconductor, Rockchip, and some Renesas systems.
 +
 +
Writing a free software driver for PowerVR [https://libreplanet.org/wiki/Group:PowerVR_drivers is a priority for the Free Software Foundation] since 2012 and the project has a [http://powervr.gnu.org.ve/doku.php web page]. Progress has been sparse.
 +
 +
=== Nvidia ===
 +
 +
== Media Stream processors ==
 +
 +
A lot of embedded processors have slave DSP-like or other signal processing accelerators.
 +
 +
Usually these take the form of a stand-alone system with its own custom DSP with a custom ISA and firmware that need to be loaded from the outside. In some cases (like the Qualcomm Hexagon DSP) this processor may in turn run another instance of Linux.
 +
 +
Once the media stream processor is loaded and parametrized with the right firmware commands, it performs a high-speed media stream conversion, such as decoding H.264 video or MP3 audio on-the-fly.
 +
 +
Some work has been done to manage these processors from drivers/remoteproc and drive the interaction with the streams using rpmsg. rpmsg however require that the firmware of the media stream processor is also augmented to respond to the rpmsg protocol and thus assumes ownership over both sides of the link.
 +
 +
Recently, Björn Andersson [http://lwn.net/Articles/699057/ sent out a patch series] augmenting rpmsg to accomodate other shared-memory transports than virtio, making virtio just one possible back-end and adding the Qualcomm SMD transport protocol an alternative native rpmsg back-end. With this, other transport protocols can be added.
 +
 +
To funnel high-speed datastreams, the dmabuf mechanism need to be utilized in the kernel, and the CMA memory allocator is also often necessary to employ due to memory segmentation constraints. Thus several kernel mechanisms come into play to deliver a full end-to-end upstream solution.
 +
 +
== MMC ==
 +
 +
* [https://www.codeaurora.org/cgit/quic/la/kernel/msm-3.18/tree/drivers/mmc/card/mmc_block_test.c?h=aosp/android-3.10&id=00e6ac21c6c2b1b0a83cf806daf74d1b9e30767e|MMC block test] code
  
 
== Power ==
 
== Power ==
 +
 +
===Battery charging===
 +
 +
Battery charging to be handled in drivers/power. These can be subdivided into two classes:
 +
 +
* Smart batteries, resembling laptop batteries, which have a hardware-provided state machine to charge and monitor the status of the battery. These have good framework support upstream already in the <tt>struct power_supply_desc</tt> and <tt>enum power_supply_property</tt> found in <tt>&lt;linux/power_supply.h&gt;</tt>
 +
* Batteries requireing an external charging algorithm. It was at one point unclear what part of a charging algorithm should go into kernelspace and what parts should possibly go into userspace akin to how Wireless have policy in userspace. It is nowadays widely agreed that any battery handling that may cause damage to the hardware if not attended to need to go into kernelspace, and that includes the major part of the charging algorithm.
 +
 +
The latter class of batteries exist upstream but does not have shared code and frameworks in place to implement the charging algorithm. These algorithms are quite physical and close to standardized, the most well-known concepts are CC/CV (Constant-Current/Constant-Voltage) and trickle charging. These algorithms are then modified to handle critical events such as overtemperature or overvoltage, and for this reason they periodically perform temperature and voltage and current measurements. They are mostly implemented in the form of a state machine. Interaction with IIO drivers for ADC and/or temperature is ideally needed to provide the necessary input.
 +
 +
We may not see the battery charging code in vendor kernels because it has been shimmed to userspace, even though it should technically be inside the kernel.
 +
 +
* LG/G3/qcom -
 +
* Samsung/Galaxy 4/exynos -
 +
* Acer/E2/mediatek -
 +
* Asus/Zenfone 6/Atom -
 +
* Huawei/P6/Hisilicon -
 +
* Sony/Z3/qcom - using the kernel frameworks with a whole suite of alien battery charger drivers in drivers/power, extensions to the power subsystem to handle things like battery aging, shutdown on low battery and battery ID string. These could probably be mainlined easily. Apart from this each battery driver seems to spawn a nexus of custom sysfs files which probably each have a non-standard userspace daemon to talk to them.
  
 
== USB ==
 
== USB ==
Line 82: Line 248:
 
** Currently in progress at Linaro.
 
** Currently in progress at Linaro.
 
* Code for external vbus and ID detection (looks like extcon was created for stuff like this - ish, it's mainly a mainlining of the Android switch API though it probably does fit)
 
* Code for external vbus and ID detection (looks like extcon was created for stuff like this - ish, it's mainly a mainlining of the Android switch API though it probably does fit)
 +
** The ID pin may have different resistance across it (between it and ground), indicating different charger configurations.
 +
*** See https://en.wikipedia.org/wiki/USB_On-The-Go#OTG_micro_plugs for an overview of this feature
 +
** For this reason, on some platforms the ID pin is not routed to the controller, but to the PMIC, so it can do the charger detection.
 +
** The mainline chipidea controller code doesn't handle charging correctly, as it only looks at the ID pin in the PORTSC register
 +
 +
=== USB controllers for different SoCs ===
 +
Need to look at other controllers, and see how they handle the ID pin for their OTG ports.
 +
* LG/G3/qcom
 +
** HS = chipidea (dual role) controller
 +
** SS = dwc3 controller
 +
* Samsung/Galaxy 4/exynos -
 +
** config uses CONFIG_USB_EHCI_HCD, CONFIG_USB_EHCI_S5P, CONFIG_S5P_DEV_USB_EHCI
 +
** config uses CONFIG_USB_DWC3_DEBUG, CONFIG_USB_EXYNOD_SS_UDC, CONFIG_USB_XHCI_HCD, CONFIG_USB_XHCI_EXYNOS
 +
** code can use CONFIG_USB_S3C_OTGD
 +
S5P is related to USB switching - see arch/arm/mach-exynos/dev-usb-switch.c
 +
exynos4_device_usb3_drd1 has name "exynos-dwc3"
 +
** S3C = UDC for OTG for some Samsung SoCs
 +
 +
* Acer/E2/mediatek -
 +
* Asus/Zenfone 6/Atom -
 +
* Huawei/P6/Hisilicon -
 +
 +
See [[Tims_USB_Notes]] for more information
 +
 +
== GPS (Global Positioning System) ==
 +
[https://en.wikipedia.org/wiki/Global_Positioning_System GPS] or any other positioning system for that matter, seems to be driven from userspace. In mobile handsets [https://en.wikipedia.org/wiki/Assisted_GPS Assisted GPS] utilizing cell tower data is commonly used. These chips usually consists of a "baseband" (in the modem terminology, the digital chip) and a RF (radio frequency) frontend that connects to the antenna (also called an air interface). If the "baseband" is a stand-alone chip, it tends to have a large firmware and complex interactions over some local bus such as simple UART. The power states of the chip is also driven from userspace, making power-saving scenarious uncoordinated with kernel power management for the most part. Assisted GPS requires network connectivity over cellular data and special interactions with a cellular modem to work.
 +
 +
Stand-alone GPS chips for devices without cellular modems (such as tablets) also exist. Garmin has [http://developer.garmin.com/open-source/linux/ open source releases] for all their Linux-based GPS devices, based on "basebands" such as OMAP, PXA and Telechips.
 +
 +
Whether the positioning code should be in the kernel or in userspace is a question of debate.
 +
 +
== Bluetooth/NFC/GPS ==
 +
Many parts of these systems use a combination of code from user-space and code in line discipline modules that communicates with the code through the kernel via a serial interface.
 +
 +
The problem of communicating with devices connected to a serial (UART, TTY) bus had been addressed with no obvious solution in sight. As of writing [http://lwn.net/Articles/700156/ this writeup by Neil Brown] gives a good summary of the problems involved in creating a proper device bus for serial devices.
 +
 +
Here are some problems (quoted from the ksummit discuss list, and not very organized at the moment - my apologies if the attribution is wrong):
 +
 +
Linus Walleij wrote:
 +
It seems BT chips more often than not sit on a UART connection
 +
(often very high speed) and since there is no "uart bus" or "serial bus"
 +
akin to what we have for I2C or SPI, it is actually impossible to
 +
instantiate them properly in the driver model. Instead BT drivers
 +
are poked and peeked from userspace using the line discipline as
 +
if they were some kind of modem, just in-kernel.
 +
 +
Rob Herring appears to have written:
 +
Yes, but then there's always some side band signals for regulator
 +
control, reset, clocks, wake-up, rf-kill, etc. that needs a glue
 +
driver. There's been some work by Neil Brown to create a UART slave
 +
bus[1] and I've gotten several other UART device bindings recently,
 +
but they all suffer from being one off device bindings done in
 +
different ways and I want to see something common here.
 +
 +
Someone wrote:
 +
It also seems the BT folks are working on moving more of the setup
 +
into the kernel and creating a proper subsystem. I'm not sure about
 +
the details on that.
 +
 +
Marcel Holtmann wrote:
 +
We are indeed moving a lot of the initialization and handling of the
 +
Bluetooth controller into the kernel. So that this can also be shared
 +
between USB and UART based controllers.
 +
 +
The work on UART slaves (or whatever it will be called eventually) is
 +
important for Bluetooth and most likely GPS and NFC in the future. It
 +
then allows to define all the nasty behind the curtain details of that
 +
UART via DT or ACPI in vendor drivers.
 +
 +
The Bluetooth driver in theory should not care at all and generally
 +
speaking it doesn't since the Bluetooth SIG has defined a Host Controller
 +
Interface (HCI) that is standard. Meaning all operation are well defined.
 +
Just the transport over UART has many vendor specific tweaks. The two most
 +
obvious ones are deep sleep modes and running the UART at high speeds.
 +
 +
In a perfect world I would prefer we are not using the Bluetooth HCI line
 +
discipline at all. The problem right now is that everybody wants to enable
 +
the UART as /dev/ttyFOO and then move on. However in reality they are not
 +
general purpose TTY devices. The only thing you can ever do with them is
 +
tell the Bluetooth subsystem that there is a TTY device and attach its line
 +
discipline to it.
 +
 +
The majority of these high speed UARTs in phones, tablets and also laptops
 +
with Bluetooth chips attached to it should just expose a special bus that
 +
we can then enumerate. Pretending to a general purpose UART is actually
 +
cumbersome and you will just see more and more of these.
 +
 +
While Bluetooth supports USB and SDIO as well, the most common other low
 +
power interface is the UART (using H:4 or 3-Wire Bluetooth transports).
 +
The usage of PCI or SPI or I2C as interface to Bluetooth never really happened.
 +
 +
Current Bluetooth/WiFi chips usually come in these flavors.
 +
  * WiFi on PCIe + Bluetooth on USB,
 +
  * WiFi + Bluetooth on USB,
 +
  * WiFi + Bluetooth on SDIO and
 +
  * WiFi on SDIO + Bluetooth on UART.
 +
Now if you are building a mobile phone, you can choose the low power version
 +
you like. And this often turns into SDIO for WiFi and UART for Bluetooth.
 +
 +
Or and did I mention that some manufactures actually put FM radio and GPS
 +
controls behind the Bluetooth chip. So access to these slave devices goes
 +
via the Bluetooth HCI. This means that you have this fun dependency:
 +
 +
  FM radio -> Bluetooth HCI -> UART -> UART slave
 +
 +
Some nasty solution here are to double stack the line disciplines and have
 +
some sort of shared transport driver in between. I am trying to actually fix
 +
that so that Bluetooth drivers can expose proper platform buses with correctly
 +
assigned resources that the FM radio driver would just enumerate on. So that
 +
part is in the works as well.
 +
 +
We can also talk about A4WP and the corner cases where your battery is so
 +
drained that the Bluetooth Low Energy portion of your hardware is the only
 +
one that gets powered first before you get your device back to life. In case
 +
this is not obvious, the A4WP wireless charging technology using Bluetooth Low
 +
Energy for its communication.
 +
 +
== Embedded modems ==
 +
 +
The modems used in phones and tablets are not connected using UARTs or USB as is the case with desktop modems. Instead they are mostly connected with high-speed custom links using shared memory buffers.
 +
 +
=== Physical link ===
 +
 +
There have been several attempts to put drivers into drivers/rpmsg (using virtio) and drivers/remoteproc. RPMSG is problematic to use because as of writing it requires using virtio to be used as transport protocol and therefore assumes we can alter the firmware ends of the shared memory transport link, i.e. that we may modify also the modem firmware.
 +
 +
Some work has been done by Björn Andersson to augment remoteproc to accomodate the Qualcomm modem manager so remoteproc is now used to manage some modems.
 +
 +
Recently, Björn also [http://lwn.net/Articles/699057/ sent out a patch series] augmenting rpmsg to accomodate other shared-memory transports than virtio, making virtio just one possible back-end and adding the Qualcomm SMD transport protocol an alternative native rpmsg back-end.
 +
 +
=== Protocols ===
 +
 +
Some embedded modems talk Hayes commands as the modems of the past would. But more often than not they will use a binary custom protocol to communicate with the modem.
 +
 +
If the kernel shall properly set up transparent TCP/IP transport links, it needs to be aware of this protocol.
 +
 +
The only in-tree protocol is phonet that can be found in net/phonet in the kernel.
 +
 +
== PCIe ==
 +
 +
=== Endpoint / nontransparent bridge ===
 +
 +
The kernel multiple implementations for communicating directly over PCIe, but none of them are easy to use or generic enough to cover multiple vendors. Examples of existing upstream code include
 +
 +
* drivers/ntb (initial code meant to be generic, but in fact still specific to Intel hardware)
 +
* drivers/misc/mic (specific implementation for Intel MIC)
 +
* drivers/misc/spear13xx_pcie_gadget.c (broken driver for designware pcie)
 +
* drivers/remoteproc/ (generic framework that could be extended for this)
 +
 +
out of tree examples include
 +
 +
* an old prototype virtio implementation for PLX NTB (ask Arnd)
 +
* Roadrunner/Triblade NTB
 +
* Marvell endpoint mode
 +
* APM endpoint mode
 +
 +
A requirement for a framwork would be that it can support both arbitrary hardware implementations of NTB or endpoint through device driver modules, and arbitrary protocols (network, rdma, block, file system, console, socket, ...).
 +
 +
== template list ==
 +
skeleton for other comparisons:
 +
* LG/G3/qcom -
 +
* Samsung/Galaxy 4/exynos -
 +
* Acer/E2/mediatek -
 +
* Asus/Zenfone 6/Atom -
 +
* Huawei/P6/Hisilicon -
 +
* Sony/Z3/qcom -
  
 
= Notes from ksummit-discuss list =
 
= Notes from ksummit-discuss list =

Latest revision as of 01:36, 16 September 2016

This page has a list of areas of the kernel where multiple SoCs have code out-of-tree, or that appear to need some work in order to accommodate contribution of device-related software. There is also a maling list for discussion of these issues at http://lists.linuxfoundation.org/mailman/listinfo/device-mainlining


Overview

The following table shows major out-of-tree areas of code, for mobile phone products based on the 3.4 Linux kernel. For sources see Phones Processors and Download Sites. Although this kernel is quite old, it was the one currently shipping in phones in late 2014 and early 2015, when this analysis was done. It demonstrates the major technical areas where vendors used out-of-tree kernel code, for that generation of products.

The table shows different phones, and the number of lines they have out-of-mainline for several different kernel areas. The analysis was done using the upstream-analysis-tools (See https://github.com/tbird20d/upstream-analysis-tools). Since this task was mostly automated, the results here should not be considered definitive. That is, most of the categorization of diffs was done using regex matches on the file paths. Although this can result in some mis-categorizations of diffs, the results appear to show general patterns which may be of interest.

Note that the table shows the number of insertions (or changes). The number of deletions was found to be inconsequential for this analysis.

Manufacturer/phone/SoC => LG/G3/Qcom Samsung/Galaxy 4/Exynos Acer/E2/Mediatek Asus/Zenfone 6/Atom Huawei/P6/Hisilicon Sony/Z3/Qcom
Kernel area
Mach specific 417411 89118 ? ? ? 393767
Media 131811 163540 ? 1466714 153032 124973
Video 221550 176787 37876 38785 238267 214027
Wireless 110946 123308 387214 91449 122501 81502
Sound 216643 86127 12932 74057 19250 213137
Input 107334 51160 2752 53321 35410 62930
Camera 66989 1342 7900 1024348 51065 56744
GPU 53345 172300 114932 36545 198323 53468
Power 61912 7635 94059 66629 45127 46173
USB 99295 35366 30795 34074 121075 98663
Notable other Prima - 565210 firmware - 101069 i2c - 105744 reg - 90034

Technical areas

Here are some lists of things to get mainlined, in different technical areas:


Media

Video

Wireless

action item: Find out if any vendors have the same IP blocks for wireless Here are the findings for 3.4:

  • LG/G3/qcom - Broadcom 4330/4334/43341/4335/4339
    • source include drivers/net/wireless/bcmdhd
      • this code is organized differently from the Acer bcmdhd code
  • Samsung/Galaxy 4/exynos - Broadcom 4335
    • source includes drivers/net/wireless/bcmdhd
    • source includes drivers/net/wireless/ipsecdrvtl
  • Acer/E2/mediatek - Broadcom 4329/30 wireless chipsets
    • source includes drivers/net/wireless/bcmdhd
    • mainline 4.2 has drivers/net/wireless/brcm80211/brcmfmac,brcmsmac,brcmutil
    •  !! also lots of code in mdtk/krnl/drivers/combo/drv_wlan/mt66s*/wlan/mgmt,nic,os/linux
      • code under this tree is not in linux format (it has vendor style)
      • need to see if this is some mediatek on-chip thing, and Acer added external broadcom chip for E2
  • Asus/Zenfone 6/Atom - Broadcom 4329/30
    • source include drivers/net/wireless/bcmdhd
  • Huawei/P6/Hisilicon
    • defines CONFIG_WL12XX=m
  • Sony/Z3/Qcom -
    • Sony has historically used broadcom 4339 and qcom 3660 and qcom 3680.
    • Sony did internal testing of the brcmfmac driver on a phone using a 3.4 kernel with the BCM 4339 chipset

Random notes:

  • Google got burned in early Android - TI had a bad wireless implementation (it was cross-platform and didn't work well in Linux). Google now integrates broadcom code themselves, at drivers/net/wireless/brcmdhd, and requires current Android vendors to use broadcom. This explains why everyone has it.
  • broadcom driver in mainline is a drivers/net/wireless/brcm80211/*
    • this is the one people should try to use and extend if it doesn't work

Possible strategy:

  • backport brcm80211 code to 3.18 or 3.14. These are the versions of kernels likely to be used in the next generation of Android products. Having the brcm80211 code available in that version means that phone vendors can test the driver. If it's close to product grade, they can help improve it - if not, then they can report the issues upstream.
    • Is Google or someone else already doing this - AI - check 3.18 src from SoC vendor for brcm80211 code

action items

  • try brcm80211 driver on 3.14 Android device
    • identify a 3.14 Android device using a broadcom chipset for wifi
  • try brcm80211 driver on dev board running mainline
    • identify a dev board running mainline
    • make sure the board can run the brcm80211 driver
    • test to make sure all required features work
      • figure out what the required features are (is there a wireless test suite?)
  • try brcm80211 driver on phone hardware (Xperia Z3) running mainling top-of-tree (in progress)

Sound

  • HDMI integration with graphics for EDID and hotplug information.

Input

MD

  • [1] Optimisations to make dm-crypt play better with hardware crypto modules.
    • Worked on at Linaro, req based approach not going to fly upstream.
    • Basic idea is to use large blocks for crypto but the block layer really doesn't want to use anything other than PAGE_SIZE as the size, will require changes in the block core.
    • Also working on optimising the crypto layer to be more efficient with smaller block sizes.

Sensors

The Android sensor stack is one of these places where vendors are expected to put in their own sensor library providing some middleware (in userspace) to provide sensor input. Some vendors will utilize IIO (industrial I/O, drivers/iio) as their backend for this, but several still have non-upstream custom char devices to provide the sensor input. The Android x86 project has some initial work on IIO-based sensor input, providing polled (non-buffered) sensory data, but it is not part of the official Android git/repo forest.

Migrating custom sensor driver code to use IIO and provide an IIO library by default in upstream android can advance development to address this problem.

  • LG/G3/qcom -
  • Samsung/Galaxy 4/exynos -
  • Acer/E2/mediatek -
  • Asus/Zenfone 6/Atom -
  • Huawei/P6/Hisilicon -
  • Sony/Z3/qcom - older qualcomms have a custom kernelspace driver but could be migrated to IIO, the newer ones utilize a special sensor processor, the communication to this coprocessor is in userspace and presented directly to Android instead of going through the kernel and IIO, these could also be handled in kernelspace but require the off-chip communication to be handled in-kernel first.

Camera

GPU

  • HDMI integration with sound for EDID and hotplug information.

Mali

The Mali GPU is used on SoCs from Allwinner, Amlogic, CSR, InfoTM, HiSilicon, Leadcore, MediaTek, NetLogic, Nufront, Rockchip, Samsung, Socle-Tech, Spreadtrum, ST-Ericsson, ST Microelectronics, Telechips and Wondermedia (according to the Wikipedia page). Sudip Mukherjee suggested adding Mali 400 for ODROID-U3 to staging. The Odriod-U3 has lots of stuff upstream and uses a Mali 400 and Exynos 4412.

GregKH says he won't take a DRM kernel driver into staging if it doesn't have an open user-space driver. The Mali driver from ARM is apparently closed, and they appear to have no intention to open it. As it stands, Greg would not take the driver into staging.

It was suggested that possibly the Lima driver could be used to fulfill the requirement for an open-source user space. However, the Lima driver project seems to not be active any more.

Heiko Stuebner said:

An additional point would be the possible (in-)stability of the userspace 
interface. While I could sucessfully use a r6 midgard kernel driver with a r5 
userspace lib yesterday on my Rockchip Chromebook using Debian, I don't think 
that is guaranteed to work everytime or that special care is taken for the 
kernel driver to prevent needing upgrades in lockstep.

And that is not even taking into account, that the mali userspace libs seem to 
be implementator-specific somehow too. For example for Rockchip I currently get 
my libs from the ChromeOS tree [0] which can talk to X11 [although not plasma5 
it seems] and ARM seems to provide a variant for the firefly board using the 
legacy fbdev interface of the 3.10-based Android kernel. And then there are 
different variants for Samsung, etc too.

...

[The Lima project is] probably not entirely defunct, but still on life-support:

For one, if you click on the "Source"-link on limadriver.org you get the 
gitorious "we're migrating old projects to archive.org" page.

And secondly there is supposed to be an actual mesa driver for mali400 and 
even bits for the midgard gpus (Mali-T) around, but limited to Luc's harddrive 
for now [1].

Vivante

The Vivante GPU is used in Intel Atom, CuBox, Rockchip, Actions Semiconductor, Freescale, HiSilicon and Marvell SoCs, according to the Wikipedia page.

The Etnaviv project has proposed initial GPU driver support to the upstream kernel.

The Linux kernel driver developed by the Etnaviv project has been merged into Linux 4.5. Lucas Stach gave a presentation on etnaviv at FOSDEM 2016. See the slides and watch the video (not available yet).

Adreno

Rob Clark has been working on freedreno, an open source driver for the Adreno hardware used on QualComm SoCs. He gave a presentation on the status at ELC 2015. See the slides, video and showcase poster.

Rob reported (by private e-mail in September 2015) that there is now the GLES 3.0 code has support for mesa 11. (Much more detail about the status of the project (as of March 2015) is available in the slides.)

PowerVR

(eLinux has a wiki page about how to Create Free Software PowerVR GPU driver)

The PowerVR GPU by Imagination Technologies is used by a host of embedded SoCs including Texas Instruments OMAP, DaVinci and DM3730, MediaTek, some Intel, Apple iPhone, Samsung S5P and Exynos 3110 and 5410, Actions Semiconductor, Rockchip, and some Renesas systems.

Writing a free software driver for PowerVR is a priority for the Free Software Foundation since 2012 and the project has a web page. Progress has been sparse.

Nvidia

Media Stream processors

A lot of embedded processors have slave DSP-like or other signal processing accelerators.

Usually these take the form of a stand-alone system with its own custom DSP with a custom ISA and firmware that need to be loaded from the outside. In some cases (like the Qualcomm Hexagon DSP) this processor may in turn run another instance of Linux.

Once the media stream processor is loaded and parametrized with the right firmware commands, it performs a high-speed media stream conversion, such as decoding H.264 video or MP3 audio on-the-fly.

Some work has been done to manage these processors from drivers/remoteproc and drive the interaction with the streams using rpmsg. rpmsg however require that the firmware of the media stream processor is also augmented to respond to the rpmsg protocol and thus assumes ownership over both sides of the link.

Recently, Björn Andersson sent out a patch series augmenting rpmsg to accomodate other shared-memory transports than virtio, making virtio just one possible back-end and adding the Qualcomm SMD transport protocol an alternative native rpmsg back-end. With this, other transport protocols can be added.

To funnel high-speed datastreams, the dmabuf mechanism need to be utilized in the kernel, and the CMA memory allocator is also often necessary to employ due to memory segmentation constraints. Thus several kernel mechanisms come into play to deliver a full end-to-end upstream solution.

MMC

Power

Battery charging

Battery charging to be handled in drivers/power. These can be subdivided into two classes:

  • Smart batteries, resembling laptop batteries, which have a hardware-provided state machine to charge and monitor the status of the battery. These have good framework support upstream already in the struct power_supply_desc and enum power_supply_property found in <linux/power_supply.h>
  • Batteries requireing an external charging algorithm. It was at one point unclear what part of a charging algorithm should go into kernelspace and what parts should possibly go into userspace akin to how Wireless have policy in userspace. It is nowadays widely agreed that any battery handling that may cause damage to the hardware if not attended to need to go into kernelspace, and that includes the major part of the charging algorithm.

The latter class of batteries exist upstream but does not have shared code and frameworks in place to implement the charging algorithm. These algorithms are quite physical and close to standardized, the most well-known concepts are CC/CV (Constant-Current/Constant-Voltage) and trickle charging. These algorithms are then modified to handle critical events such as overtemperature or overvoltage, and for this reason they periodically perform temperature and voltage and current measurements. They are mostly implemented in the form of a state machine. Interaction with IIO drivers for ADC and/or temperature is ideally needed to provide the necessary input.

We may not see the battery charging code in vendor kernels because it has been shimmed to userspace, even though it should technically be inside the kernel.

  • LG/G3/qcom -
  • Samsung/Galaxy 4/exynos -
  • Acer/E2/mediatek -
  • Asus/Zenfone 6/Atom -
  • Huawei/P6/Hisilicon -
  • Sony/Z3/qcom - using the kernel frameworks with a whole suite of alien battery charger drivers in drivers/power, extensions to the power subsystem to handle things like battery aging, shutdown on low battery and battery ID string. These could probably be mainlined easily. Apart from this each battery driver seems to spawn a nexus of custom sysfs files which probably each have a non-standard userspace daemon to talk to them.

USB

  • USB-gadget to charger interface
    • Currently in progress at Linaro.
  • Code for external vbus and ID detection (looks like extcon was created for stuff like this - ish, it's mainly a mainlining of the Android switch API though it probably does fit)
    • The ID pin may have different resistance across it (between it and ground), indicating different charger configurations.
    • For this reason, on some platforms the ID pin is not routed to the controller, but to the PMIC, so it can do the charger detection.
    • The mainline chipidea controller code doesn't handle charging correctly, as it only looks at the ID pin in the PORTSC register

USB controllers for different SoCs

Need to look at other controllers, and see how they handle the ID pin for their OTG ports.

  • LG/G3/qcom
    • HS = chipidea (dual role) controller
    • SS = dwc3 controller
  • Samsung/Galaxy 4/exynos -
    • config uses CONFIG_USB_EHCI_HCD, CONFIG_USB_EHCI_S5P, CONFIG_S5P_DEV_USB_EHCI
    • config uses CONFIG_USB_DWC3_DEBUG, CONFIG_USB_EXYNOD_SS_UDC, CONFIG_USB_XHCI_HCD, CONFIG_USB_XHCI_EXYNOS
    • code can use CONFIG_USB_S3C_OTGD

S5P is related to USB switching - see arch/arm/mach-exynos/dev-usb-switch.c exynos4_device_usb3_drd1 has name "exynos-dwc3"

    • S3C = UDC for OTG for some Samsung SoCs
  • Acer/E2/mediatek -
  • Asus/Zenfone 6/Atom -
  • Huawei/P6/Hisilicon -

See Tims_USB_Notes for more information

GPS (Global Positioning System)

GPS or any other positioning system for that matter, seems to be driven from userspace. In mobile handsets Assisted GPS utilizing cell tower data is commonly used. These chips usually consists of a "baseband" (in the modem terminology, the digital chip) and a RF (radio frequency) frontend that connects to the antenna (also called an air interface). If the "baseband" is a stand-alone chip, it tends to have a large firmware and complex interactions over some local bus such as simple UART. The power states of the chip is also driven from userspace, making power-saving scenarious uncoordinated with kernel power management for the most part. Assisted GPS requires network connectivity over cellular data and special interactions with a cellular modem to work.

Stand-alone GPS chips for devices without cellular modems (such as tablets) also exist. Garmin has open source releases for all their Linux-based GPS devices, based on "basebands" such as OMAP, PXA and Telechips.

Whether the positioning code should be in the kernel or in userspace is a question of debate.

Bluetooth/NFC/GPS

Many parts of these systems use a combination of code from user-space and code in line discipline modules that communicates with the code through the kernel via a serial interface.

The problem of communicating with devices connected to a serial (UART, TTY) bus had been addressed with no obvious solution in sight. As of writing this writeup by Neil Brown gives a good summary of the problems involved in creating a proper device bus for serial devices.

Here are some problems (quoted from the ksummit discuss list, and not very organized at the moment - my apologies if the attribution is wrong):

Linus Walleij wrote:

It seems BT chips more often than not sit on a UART connection
(often very high speed) and since there is no "uart bus" or "serial bus"
akin to what we have for I2C or SPI, it is actually impossible to
instantiate them properly in the driver model. Instead BT drivers
are poked and peeked from userspace using the line discipline as
if they were some kind of modem, just in-kernel.

Rob Herring appears to have written:

Yes, but then there's always some side band signals for regulator
control, reset, clocks, wake-up, rf-kill, etc. that needs a glue
driver. There's been some work by Neil Brown to create a UART slave
bus[1] and I've gotten several other UART device bindings recently,
but they all suffer from being one off device bindings done in
different ways and I want to see something common here.

Someone wrote:

It also seems the BT folks are working on moving more of the setup
into the kernel and creating a proper subsystem. I'm not sure about
the details on that.

Marcel Holtmann wrote:

We are indeed moving a lot of the initialization and handling of the
Bluetooth controller into the kernel. So that this can also be shared
between USB and UART based controllers.

The work on UART slaves (or whatever it will be called eventually) is
important for Bluetooth and most likely GPS and NFC in the future. It
then allows to define all the nasty behind the curtain details of that
UART via DT or ACPI in vendor drivers.

The Bluetooth driver in theory should not care at all and generally
speaking it doesn't since the Bluetooth SIG has defined a Host Controller
Interface (HCI) that is standard. Meaning all operation are well defined.
Just the transport over UART has many vendor specific tweaks. The two most
obvious ones are deep sleep modes and running the UART at high speeds.

In a perfect world I would prefer we are not using the Bluetooth HCI line
discipline at all. The problem right now is that everybody wants to enable
the UART as /dev/ttyFOO and then move on. However in reality they are not
general purpose TTY devices. The only thing you can ever do with them is
tell the Bluetooth subsystem that there is a TTY device and attach its line
discipline to it.

The majority of these high speed UARTs in phones, tablets and also laptops
with Bluetooth chips attached to it should just expose a special bus that
we can then enumerate. Pretending to a general purpose UART is actually
cumbersome and you will just see more and more of these.

While Bluetooth supports USB and SDIO as well, the most common other low
power interface is the UART (using H:4 or 3-Wire Bluetooth transports).
The usage of PCI or SPI or I2C as interface to Bluetooth never really happened.

Current Bluetooth/WiFi chips usually come in these flavors.
  * WiFi on PCIe + Bluetooth on USB,
  * WiFi + Bluetooth on USB,
  * WiFi + Bluetooth on SDIO and
  * WiFi on SDIO + Bluetooth on UART.
Now if you are building a mobile phone, you can choose the low power version
you like. And this often turns into SDIO for WiFi and UART for Bluetooth.

Or and did I mention that some manufactures actually put FM radio and GPS
controls behind the Bluetooth chip. So access to these slave devices goes
via the Bluetooth HCI. This means that you have this fun dependency:

  FM radio -> Bluetooth HCI -> UART -> UART slave

Some nasty solution here are to double stack the line disciplines and have
some sort of shared transport driver in between. I am trying to actually fix
that so that Bluetooth drivers can expose proper platform buses with correctly
assigned resources that the FM radio driver would just enumerate on. So that
part is in the works as well.

We can also talk about A4WP and the corner cases where your battery is so
drained that the Bluetooth Low Energy portion of your hardware is the only
one that gets powered first before you get your device back to life. In case
this is not obvious, the A4WP wireless charging technology using Bluetooth Low
Energy for its communication.

Embedded modems

The modems used in phones and tablets are not connected using UARTs or USB as is the case with desktop modems. Instead they are mostly connected with high-speed custom links using shared memory buffers.

Physical link

There have been several attempts to put drivers into drivers/rpmsg (using virtio) and drivers/remoteproc. RPMSG is problematic to use because as of writing it requires using virtio to be used as transport protocol and therefore assumes we can alter the firmware ends of the shared memory transport link, i.e. that we may modify also the modem firmware.

Some work has been done by Björn Andersson to augment remoteproc to accomodate the Qualcomm modem manager so remoteproc is now used to manage some modems.

Recently, Björn also sent out a patch series augmenting rpmsg to accomodate other shared-memory transports than virtio, making virtio just one possible back-end and adding the Qualcomm SMD transport protocol an alternative native rpmsg back-end.

Protocols

Some embedded modems talk Hayes commands as the modems of the past would. But more often than not they will use a binary custom protocol to communicate with the modem.

If the kernel shall properly set up transparent TCP/IP transport links, it needs to be aware of this protocol.

The only in-tree protocol is phonet that can be found in net/phonet in the kernel.

PCIe

Endpoint / nontransparent bridge

The kernel multiple implementations for communicating directly over PCIe, but none of them are easy to use or generic enough to cover multiple vendors. Examples of existing upstream code include

  • drivers/ntb (initial code meant to be generic, but in fact still specific to Intel hardware)
  • drivers/misc/mic (specific implementation for Intel MIC)
  • drivers/misc/spear13xx_pcie_gadget.c (broken driver for designware pcie)
  • drivers/remoteproc/ (generic framework that could be extended for this)

out of tree examples include

  • an old prototype virtio implementation for PLX NTB (ask Arnd)
  • Roadrunner/Triblade NTB
  • Marvell endpoint mode
  • APM endpoint mode

A requirement for a framwork would be that it can support both arbitrary hardware implementations of NTB or endpoint through device driver modules, and arbitrary protocols (network, rdma, block, file system, console, socket, ...).

template list

skeleton for other comparisons:

  • LG/G3/qcom -
  • Samsung/Galaxy 4/exynos -
  • Acer/E2/mediatek -
  • Asus/Zenfone 6/Atom -
  • Huawei/P6/Hisilicon -
  • Sony/Z3/qcom -

Notes from ksummit-discuss list

  • RGB leds - not the same as 3 leds, and sometimes have hardware acceleration
  • battery over-discharge protection
  • charger (there are 3 different chargers on N900)
  • voice link to modem (Nokia says ALSA is not suitable for modem audio)
    • No need for this, ALSA is in active use for rather a lot of production phones...
  • userspace (mini-distro?) for testing
  • suspend-to-ram