Embedded linux status

Here is an outline for a presentation on the status of embedded Linux:

This is essentially Tim Bird's private collection of interesting notes about the status of embedded Linux. It tends to get updated right before a Linux conference.

 NEWS: Tim Bird's [[Media:Status-of-embedded_Linux-2013-10-ELCE-v2.pdf|Status of Embedded Linux talk (PDF)]] (October 2013) presentation, and video from ELCE 2013 is now available.

Process for adding information
Anyone can add information to this page. I used to maintain the information at the Technology Watch List, but the table format there is a bit constrictive. (It would be nice if MediaWiki had a table editor!!)

Since I have to form this stuff into a "State of Embedded Linux" presentation several times a year, keeping the information in wiki outline format is convenient for me. It's easier to put directly into a presentation.

Please place information in bullet form, with a link to a supporting article, in the appropriate sub-section below.

"Best Of" page
Please see the Best of Embedded Linux page for my collection of interesting products at the extremes of embedded Linux.

Page History
I'll let MediaWiki store historical versions of this page. If you want to see what the hot issues were from a last year or a few years ago, please see look at the page history. (Although, updates of this page have historically been a bit spotty).

Presentation History
Here's my presentation history:
 * [[Media:Status-of-Embedded-Linux-2014-10-JJ50-v2.pdf|Jamboree 50 Status of Embedded Linux (PDF)]] (October 2014)
 * [[Media:Status-of-Embedded-Linux-2014-07-JJ49.pdf|Jamboree 49 Status of Embedded Linux (PDF)]] (July 2014)
 * [[Media:Status-of-Embedded-Linux-2013-12-JJ47.pdf|Jamboree 47 Status of Embedded Linux (PDF)]] (December 2013)
 * [[Media:Status-of-embedded_Linux-2013-10-ELCE-v2.pdf|ELC Europe 2013 Status of Embedded Linux talk (PDF)]] (October 2013) Video
 * [[Media:Status-of-Embedded-Linux-2013-09-JJ46.pdf|Jamboree 46 Status of Embedded Linux (PDF)]] (September 2013) Video
 * [[Media:Status-of-Embedded-Linux-2013-06-JJ45.pdf|Jamboree 45 Status of Embedded Linux (PDF)]] (June 2013)
 * [[Media:Status-of-Embedded-Linux-2013-03-JJ44.pdf|Jamboree 44 Status of Embedded Linux (PDF)]] (March 2013)
 * [[Media:Status-of-Embedded-Linux-2012-09-JJ42.pdf|Jamboree 42 Status of Embedded Linux (PDF)]] (September 2012)
 * [[Media:Status-of-embedded-Linux-2012-06-JJ41.pdf|Jamboree 41 Status of Embedded Linux (PDF)]] (June 2012)
 * [[Media:Status-update-2012-03-Jamboree40.pdf|Jamboree 40 Status update (PDF)]] (March 2012)
 * [[Media:Status-of-embedded-Linux-2012-02-ELC-v2.pdf|ELC 2012 Status of Embedded Linux (PDF)]] (February 2012)
 * [[Media:Status-of-embedded-Linux-2011-11-LCB.pdf‎|LinuxCon Brazil Status of Embedded Linux talk (PDF)]] (November 2011)
 * [[Media:Status-of-Embedded-Linux-2011-10-ELCE.pdf|ELC Europe 2011 Status of Embedded Linux talk (PDF)]] (October 2011)
 * [[Media:Status-of-embedded-Linux-2011-09-Jamboree38.ppt|Jamboree 38 Status of Embedded Linux talk (PPT)]] (September 2011)
 * [[Media:Status-of-embedded-Linux-2011-06-LCJ.pdf|LinuxCon Japan 2011 Status of Embedded Linux talk (PDF)]] (June 2011)
 * [[Media:Status-of-embedded-Lnux-2011-05-J37.ppt|Jamboree 37 Status of Embedded Linux talk (PPT)]] (May 2011)
 * [[Media:Status2-of-embedded-Linux-2010-12-Jamboree35.ppt|Jamboree 35 Status of Embedded Linux talk (PPT)]] (December 2010)
 * LinuxCon Japan 2010 Status of Embedded Linux BOF (PDF) (September 2010)
 * [[Media:Status-of-embedded-Linux-2010-04-ELC.pdf|ELC 2010 Status of Embedded Linux talk (PDF)]] (April 2010)
 * [[Media:State-of-embedded-linux-oct-2009.pdf|Japan Linux Symposium 2009 Status of Embedded Linux talk (PDF)]] (October 2009)
 * [[Media:Embedded-Linux-Status-OLS2009.pdf|OLS 2009 Embedded Linux Status talk (PDF)]] (July 2009)
 * [[Media:Embedded-Linux-BOF-OLS2008.ppt|OLS 2008 Embedded Linux BOF slides (PPT)]] (July 2008)
 * [[Media:Status-of-embedded-Linux-ELC2008.ppt|ELC 2008 Status of Embedded Linux talk (PPT)]] (April 2008)
 * [[Media:ELCE-BOF_State_of_Embedded_linux.pdf|ELC Europe 2007 - Status of Embedded Linux BOF slides (PDF)]] (October 2007)
 * [[Media:Embedded_Linux_BOF_OLS_2007.pdf|OLS 2007 Embedded Linux BOF slides (PDF)]] (July 2007)
 * [[Media:OLS2006-Embedded-BOF-2.ppt|OLS 2006 Embedded Linux BOF slides (PPT)]] (July 2005)
 * OLS 2005 Status of Embedded Linux BOF [need presentation]

Uncategorized info
This is where I put stuff I haven't had time to analyze or sort into the appropriate category:

December 2014

 * kernel
 * Kernel version 3.18
 * major features??
 * Things to watch
 * tinification!
 * android patches: Binder
 * https://lwn.net/Articles/618421/
 * kdbus
 * systemd in embedded
 * technologies
 * Bootup time ??
 * graphics
 * file systems
 * power management
 * real time
 * system size
 * splice syscall elimination
 * http://lwn.net/Articles/624258
 * size selftest
 * toolchains
 * testing
 * CEWG projects
 * IOT
 * standard distribution
 * social infrastructure
 * device mainlining
 * LTSI
 * LTSI testing
 * Other stuff
 * Events
 * ELC = Drones, Things and Automobiles
 * March 23-25
 * Initiatives
 * Dronecode
 * AllSeen Alliance news
 * PRPL Foundation
 * Distros
 * Android
 * Tizen
 * other?
 * Build Systems
 * Yocto project
 * Tizen

February 2014

 * http://www.networkworld.com/slideshow/139614/?t51hb&hpg1=mp
 * list of 16 weird places Linux shows up

January 2014

 * Google create automotive alliance for Android - http://www.openautoalliance.net/#about
 * Spark - wireless, dumb board for home automation or IOT.
 * http://www.engadget.com/2014/01/17/spark-shows-how-to-build-a-nest-like-open-source-thermostat/
 * http://blog.spark.io/2014/01/17/open-source-thermostat/
 * http://www.engadget.com/2013/05/03/spark-core-adds-wifi-to-everything/

November 2013

 * things to follow up on:
 * device tree schema validator

general

 * Valve launches SteamOS
 * Good boot time presentation: http://www.slideshare.net/righiandr/linux-boottime-23817352

graphics

 * Nvidia helps with Nouveau effort: http://lwn.net/Articles/568038/

device tree pain
Linus quote about undiscoverable busses: https://lkml.org/lkml/2013/9/10/366

"Ok. I still really despise the absolute incredible sh*t that is non-discoverable buses, and I hope that ARM SoC hardware designers all die in some incredibly painful accident. DT only does so much.

So if you see any, send them my love, and possibly puncture the brake-lines on their car and put a little surprise in their coffee, ok?"

what's coming next

 * transactional memory
 * non-volatile main memory
 * mirasol (color reflective displays)
 * qualcomm watch
 * wearables
 * watches from everyone (Samsung, Sony, Qualcomm)
 * modular pieces (mix-and-match)
 * Example: Sony QX cameras

cpu announcements

 * quark
 * SOC standings:
 * Intel, Qualcomm, MediaTek, NVidia, Samsung, Broadcom
 * GPUs: Mali (Intel, from ARM), Adreno (qcom, Radeon from ATI), PowerVR (Imagination), Intel's own GPU, Videocore (broadcom, from Alphamosaic)

July 2013

 * ARM SOCS moving to device-tree based pinctrl
 * hacker releases leaked exFAT filesystem code
 * http://www.wired.com/wiredenterprise/2013/07/samsung_code/
 * lwn.net/Articles/560424/
 * part of the code has strings identical to code submitted to Linux kernel in 2006

February 2012

 * lttng 2.0 was in mainline for about 2 weeks
 * CTF exists, as well as babeltrace
 * babeltrace (library for trace conversion) exists, but has no serious converters yet
 * TMF (eclipse viewer) support for CTF coming real soon (if not already) (was predicted to be January 2012)
 * TMF support for LTTng 2.0 planned for q2 2012

May 2011
Quote:
 * good interview with Linus on 20th anniversary
 * http://linuxfr.org/nodes/85904/comments/1230981

LinuxFR : What is your opinion about Android ? Are you mostly happy they made cellphones very usable or sad because it's really a kernel fork ?

Linus Torvalds : I think forks are good things, they don't make me sad. A lot of Linux development has come out of forks, and it's the only way to keep developers honest - the threat that somebody else can do a better job and satisfy the market better by being different. The whole point of open source to me is really the very real ability to fork (but also the ability for all sides to then merge the forked content back, if it turns out that the fork was doing the right things!)

So I think the android fork forced the mainline developers to seriously look at some of the issues that android had. I think we've solved them in mainline, and I hope (and do think) that android will eventually end up merging to mainline. But it will probably take time and further effort.

I think the more serious long-term issue we have in the kernel is the wild and crazy embedded platform code (and mostly ARM - not because ARM is in any way fundamentally crazier, but because ARM is clearly the most successful embedded platform by far). The embedded world has always tended to eschew standardized platforms: they've been resource constrained etc, so they've done very tailored chip and board solutions, and felt that they couldn't afford a lot of platform abstraction.

That causes a big maintenance headache, because then all those crazy platforms look slightly different to the kernel, and we have all that silly code just to support all those variations of what is really just the same thing deep down, just differently hooked up and with often arbitrary small differences.

But that's something that happens both within and outside of Android, it's in no way android-specific.

LinuxFR : What about the technical differences between Android and mainline ? Do you think the "wakelock" controversy is solvable ?

Linus Torvalds : I think it is technically largely solved (ie "details to be fixed, but nothing fundamentally scary"), but practically once you have an interface and existing code, it just is a fair amount of work to change. And there perhaps isn't quite enough motivation to make those changes very quickly. So it will take time, and probably several releases (both mainline and adroid) to actually happen.

LinuxFR : Can you explain why you're not happy with the ARM patches sent to you during merge windows ? Is there an obvious solution for this fragmentation problem ?

Linus Torvalds : Obvious solution? No. The problem is the wild variety of hardware, and then in many cases the Linux ARM platform code (not the ARM CPU support, but the support for certain chips with all the glue issues around the CPU core) has been mostly ugly "copy-and-paste" from some previous ARM platform support file, with some minimal editing to make it match the new one.

And it just results in this unmaintainable mess. It becomes painful when somebody then fixes some core infrastructure, and you end up with a hundred different ARM files all using that infrastructure. That happened with the IRQ chip driver cleanups Thomas did recently (well, has been doing over along time, the recent part is really just the final removal of some nasty old interfaces).

It results in other maintainability issues too - patches being big just means that people won't look at them as carefully etc etc. So it's just a bad situation. Many of the cases should be solvable by having better generic solutions and then plugging in just some per-platform numbers for those solutions.

April 2011

 * http://www.linuxfordevices.com/c/a/News/OIN-announces-new-members/?kc=LNXDEVNL042011 - OIN grows 28%
 * Intel working on supporting Android 3.0, according to Paul Ottellini
 * Android in focus at Intel - http://news.cnet.com/8301-13924_3-20055868-64.html
 * Mobile patent lawsuit cheatsheet - http://technologizer.com/2011/04/19/mobile-lawsuits/
 * Linus fed up with ARM sub-architecture chaos
 * Linaro organizing group to help resolve issues (possibly hire sub-arch coordinator)
 * information about the longterm release (2.6.35)
 * how are we deciding the next one
 * information about CELF LTS kernel??

Other

 * CONFIG_PM being eliminated - improved PM configuration
 * ftrace using -mfentry (feb 9, 2011 lkml, steven rostedt)

April 2010

 * HP buys Palm - now owns Webos
 * Japanese companies (with LIMO) announce new mobile stack - http://www.linuxfordevices.com/c/a/News/Consortium-from-NTT-DoCoMo-Renesas-Fujitsu-NEC-Panasonic-and-Sharp/?kc=LNXDEVNL042810

Previous to April 2010

 * SystemTap 1.0 now includes support for cross-compilation.
 * Patches for dynamic printks
 * writing to /dev/kmsg to generate a printk (not new, but I just discovered it)
 * LZO kernel compression is in 2.6.33
 * Arjan van de Ven's timer-slack code - http://lwn.net/Articles/369361/

Bootup Time

 * U-boot bootgraph.pl support (see e-mail from Andrew Murray)
 * Android has problems
 * many people are addressing it with Snapshot boot
 * readahead is getting lots of attention
 * See Tim's presentation on Android boot time with readahead
 * snapshot boot (see above)
 * See ELC 2010? and ABS 2011 presentation on snapshot booting
 * embedded bootchart
 * busybox bootchart
 * bootchart in Android init
 * bootchart2 project
 * C collector, python visualizer


 * filesystem speedups
 * CELF funding UBI logging
 * CELF funding read-only block filesystems on flash (MTD)


 * XIP
 * Almost removed from kernel
 * versions in kernel were broken, use of XIP on out-of-tree platforms doesn't help keep XIP in the tree


 * Bootloader improvements
 * Coreboot on x86
 * See Really Fast x86 boot talk (video) from Rudolf Marek at Fosdem 2011
 * U-Boot ARM caching enhancements

Bootloader

 * fastboot support in U-Boot
 * supposedly TI did this?
 * See remark by John Rigby about possible Linaro project for this (on U-Boot list):http://mailrepository.com/u-boot.lists.denx.de/msg/3672644/
 * DFU support in U-Boot
 * [todo: get status update on this]

Memory Management

 * anything new happening?

Power Management

 * suspend blockers? (aka wakelocks)
 * device PM

then and now
2003 - wanted:
 * operating points
 * frequency scaling
 * tickless idle
 * device pm

2011 - have:
 * tickless idle
 * device pm

System Size

 * CELF reviving Linux-tiny project
 * bloatwatch is still running, but who looks at it?
 * http://www.selenic.com/bloatwatch/
 * big increase (500K) from 2.6.37-rc6 to 2.6.37-rc7
 * Xi Wang's talk at ELC about optimizing memory usage throughout system (kernel, libs, application)


 * OOM killer - dealing with memory pressure:
 * [RESEARCH: OOM killer news]
 * Android has its own thing
 * cgroup memory notifications

then and now
2003 - wanted (shrink kernel to ???k) 2011 - current size = ?
 * see bloatwatch
 * growth is in user space
 * compare with platform size growth over same period

Security

 * virtualization for Android
 * Samsung using vmware for Android (to separate personal and business use of phone)
 * Android and attacks
 * Android has different security
 * has there been a root exploit from a java app?

then and now
2003 - wanted trusted root (TPM), guard against exploits 2011 - have??

File Systems

 * YAFFS2
 * Mainline effort by Charles Manning
 * LogFS
 * Joern disappeared again
 * Squashfs
 * Now supports LZMA2 in mainline
 * CELF funding SquashFS on MTD work
 * Arnd Bergmann's work on optimizing Linux FS for cheap flash media
 * See ELC presentation and Linaro page
 * Tim Bird's treadahead work
 * See ABS presentation

Legal Issues (licensing and patents)

 * mobile patent wars
 * Google buys Motorola for patent portfolio
 * Apple blocks Samsung tablet introduction in Europe

Graphics

 * OpenGL ES
 * whither fbdev?
 * 2D - Android doesn't have a native 2D API (or it's changing?)
 * [RESEARCH - did Android drop it's native 2D API? (skia?)]


 * Latest intel Atom (cedarview) to have PowerVR graphics core
 * Core is PowerVR SGX545-based
 * Source: http://news.softpedia.com/news/Next-Intel-Atom-Has-Stronger-PowerVR-Graphics-199342.shtml
 * patch submission to staging: http://thread.gmane.org/gmane.linux.kernel/1103793


 * GoogleTV 1.0 (logitech and Sony) use Sodaville, which is an Atom with a SGX535 core


 * Alan Cox submitted GMA500 kernel driver to kernel mailing list
 * Source: http://www.h-online.com/open/news/item/Rudimentary-open-source-driver-for-Intel-s-GMA500-Poulsbo-1195125.html


 * Intel work on graphics in kernel:
 * See Kieth Packard's video from September 2010: [Meego Graphics under the hood: http://www.youtube.com/watch?v=YRYTCQqrFcA] OSCON 2010
 * Working on grphics KMS and DRI in the kernel
 * good for faster booting
 * good for less flicker on transitions
 * higher performance 3d graphics
 * better memory management
 * kernel can pull memory back from graphics processor on low-memory conditions
 * [RESEARCH - status of memory allocation discussions at ELC?]
 * per CRTC pixmaps
 * allows for zero-copy rotations
 * support for larger screens
 * support for multiple screens
 * wayland (non-X-windows)
 * moving away from X on Meego
 * support for multiple APIs
 * EGL
 * OpenGL 2.1 supported now, OpenGL 3.0 support coming soon
 * geometry shaders, tesselation
 * architecture:

user space: Open GL  (compiler for shaders) mesa i915 driver --- graphics interface (is not opengl, but a device-specific abstraction) --- kernel space: DRI i915 driver

kernel space driver is a small driver Combination of DRI and GPU-specific driver is called the Graphics Execution Manager (GEM)


 * Android is using LLVM for compiler shaders
 * Android renderscript
 * http://android-developers.blogspot.com/2011/02/introducing-renderscript.html
 * [Google IO 2011 talk: Accelerated Android Rendering http://www.youtube.com/watch?v=v9S5EO7CLjo&NR=1]


 * ABS Khronos standards talk? (no slides available)

Multimedia

 * GStreamer - held first ever GStreamer conference in October (co-located with ELC Europe)
 * Khronos is working on their stuff

infrared remote control support

 * LIRC kernel drivers were mainlined as of 2.6.26
 * IR: add lirc support to ir-core LWN.net LKML reference
 * LIRC web site: http://www.lirc.org/
 * LIRC FAQ: http://www.lirc.org/faq.html

Debuggers

 * 2012-11 - ARM FIQ work (activate KDB with ARM FIQ), by Anton Vorontsov
 * See http://sourceforge.net/mailarchive/message.php?msg_id=30134183

build systems

 * OpenEmbedded/Poky (Yocto is umbrella project)
 * Appears to be gaining momentum - lots of companies (including Sony) using it now
 * lots of new interesting features: graphical tool, can build an image for self-hosting a build, new compatibility badging program for meta-data
 * Will provide LTSI kernels as part of offering

Tracing

 * 2010 - Perf vs. ftrace (LKML flamewar)
 * 2011- big issue is whether tracepoints create a stable ABI
 * Ftrace, perf, and the tracing ABI LWN.net article


 * Common Trace format
 * 2011 - lttng is shipped by most major distros (including yocto)
 * 2011 - would be nice to get in Android?

Toolchains

 * Linaro publishing toolchains - one directly for Android use
 * see https://wiki.linaro.org/WorkingGroups/ToolChain, and
 * https://launchpad.net/gcc-linaro

eclipse stuff

 * Yocto uses CDT remote launch, org.eclipse.cdt.launch.remote and TCF file/shells to transfer binaries and launch applications
 * CDT = (C Development Toolkit)
 * See: http://www.yoctoproject.org/projects/eclipse-ide-plug
 * support communication with emulator or real device, via Yocto Eclipse TCF
 * emulated devices use NFS rootfs so host and target access same filesystem
 * debugging is via cross-gdb (gdbserver and gdb client on host)


 * required plugins: (see http://www.yoctoproject.org/docs/adt-manual/adt-manual.html section 4.1.2)
 * CDT 7.0
 * RSE 3.2 (Remote System Explorer)
 * Autotools
 * Yocto Plug-in


 * Android uses Android Development Tools plugin for eclipse
 * http://developer.android.com/sdk/eclipse-adt.html
 * Supports communication with emulator or real device
 * has DDMS and traceview, as well as debugging via jdwp

Contract Work

 * eMMC tuning guide - should be published in January
 * Is coming along nicely
 * has explanations about benchmarking, analysis of different tunable items in the filesystem stack
 * has lots of results testing the following filesystems: ext3/4, BTRFS, F2FS, with different I/O schedulers (including the new ROW I/O Scheduler)
 * has specific suggestions for what options are most effective to tune your filesystem, based on the characteristics of your flash device


 * Kernel dynamic memory analysis
 * Project is complete
 * see http://elinux.org/Kernel_dynamic_memory_analysis
 * Will be reported on at ELC 2013

Distributions

 * Poky or Angstrom (OpenEmbedded/Yocto Project)

Distribution tools

 * Yocto Project

Mobile Phone distros

 * Tizen - what's up now that Nokia bailed? ( see MeeGo Moblin Maemo )
 * http://www.rethink-wireless.com/2013/01/03/mobile-linux-oss-2013.htm
 * Ubuntu mobile
 * see above link again (says native apps are a boon for speed)
 * Mozilla Firefox mobile??

More at http://elinux.org/Category:Mobile

Android

 * Versions news
 * (2012-06) Android 4.1 (Jelly Bean) announced in June, shipped in July
 * (2012-11) Android 4.2 (Jelly Bean update?) announced and shipped
 * See http://en.wikipedia.org/wiki/Android_version_history


 * (2012-12) Market share:
 * prediction from August that in 2014, installed base of Android will exceed that of all windows platforms (phone, PC and tablet) counted together. (wow!)
 * see http://communities-dominate.blogs.com/brands/2012/08/smartphone-market-shares-q2-full-numbers-samsung-and-android-solidifying-their-leads.html


 * Tablets
 * Some nice tablets based on 3.0 are coming out (will move to Android 3.1 over summer)
 * phone activations? (350,000 per day, as of April 8, 2011)
 * GoogleTV
 * Every device will have 3.1 and adb, which means every device can be used for development
 * will have market
 * good talks at Google I/O about app and web development for TVs


 * non-Android support for Android apps:
 * Dalvik ported to Meego - myriad Group AG announced Myriad Alien Dalvik, for running Android apps on other Linux platforms
 * See http://www.linuxfordevices.com/c/a/News/Myriad-Group-Myriad-Alien-Dalvik/?kc=LNXDEVNL020911
 * IcedRobot implementation of Android (using OpenJDK)
 * BlueStack to run Android apps on Windows
 * see http://www.linuxfordevices.com/c/a/News/BlueStacks/?kc=LNXDEVNL052611

Unbuntu TV

 * see http://www.ubuntu.com/tv
 * announced at CES 2012 - http://blog.canonical.com/2012/01/09/canonical-to-showcase-ubuntu-tv-at-ces/

CPU support

 * ARM architecture issues over past year
 * First, there was the
 * ARM Wrestling LWN.net article on ARM issues
 * Linaro working on methods to upstream code from vendors
 * device tree for ARM status:
 * Grant's tree is included in linux-next
 * See http://groups.google.com/group/linux.kernel/browse_thread/thread/d084d006ad2212c6/06ebf8c4ad4a7b27?lnk=raot
 * ATAG deprecation - it can now be configured off, see http://lists.infradead.org/pipermail/linux-arm-kernel/2012-August/116699.html

Chip vendor news

 * Texas Instruments leaves mobile and tablet market
 * November 2012, announces layoffs of 1700, drops future OMAP lines

Miscellaneous

 * unlockable bootloaders
 * Announced by Motorola and Sony/Ericsson
 * can unlock bootloader to install custom firmward
 * wipes the phone to remove DRM-protected content
 * Motorola says you can re-lock by going back to a vendor-supplied image
 * version update support
 * Announced at Google I/O
 * multiple partners agree to provide version updates at regular intervals

Industry organizations or projects

 * LiMo - anything happening?
 * CELF => CEWG under Linux Foundation
 * CELF projects
 * Linux Foundation
 * Lots of stuff going on in embedded:
 * Yocto
 * Meego
 * CE Working Group
 * Linaro - 1 year in
 * see David Rusling's presentation from ELC

Trends

 * movement to eclipse (both Yocto and Android support eclipse plugins for tools)
 * movement to emulators
 * application portability
 * Android apps will be able to run just about anywhere
 * Will an IOS emulator appear, or will apple strangle any attempts
 * application stores
 * multiple app stores will create competition (good)
 * more open platforms
 * Google TV
 * lots of stuff that no one expects will appear


 * development trends:
 * new tools??

Resources

 * lwn.net
 * elinux.org
 * stackoverflow.com
 * kernelnewbies
 * linuxfordevices.com