Session:Embedded is not special KR 2012

Jump to: navigation, search

Session Details

Kernel Recipes Paris 2012
September 21, 2012
David Woodhouse


How embedded Linux isn’t so special, and most of the « embedded requirements » are also requirements in « big iron » too


David is an 'embedded maintainer' for the Linux kernel, and is a member of Intel's Open Source Technology Centre.



Transcribed by
Chris Dudding
Verified by

0:00 - 1:00:

[Slide 1 - Embedded Linux and the mainline kernel]

>> DAVID WOODHOUSE: Um. I'm going to talk about "Embedded Linux" and what it means to me. Um. And how I dislike talking about "Embedded Linux" and the mainline kernel and how to work with it. Um. So.. [Speaker attempts to switch slide] Oh come on.. Please.. I'm going to talk about Open Office and how I hate it [Laughter from audience]

[Slide 2 - Where does Linux run?]

Wahey! So, Linux is everywhere. You see it on all kinds of devices: control devices, phones, all the way up to supercomputers, mainframes. Everybody runs Linux. Um. And we talk about "Embedded Linux" but what does it mean?

1:00 - 2:00:

You know.. Some people talk about "Embedded Linux" and include this kind of thing

[Speaker points to server image on slide]

It doesn't have a screen and a keyboard. It doesn't have a user with a graphical desktop. It must be embedded right?

Um.. So we use the term "Embedded Linux" to cover all manner of evils.

[Slide 3 - "Embedded"...?]

Um.. And a lot of the time when you hear sort of people in the linux development community talking about "Embedded Linux" mostly its a code word for stuff we don't want to have to care about. Um.. But its increasingly important. There are millions of android devices out there.. really.. you know.. pushing the focus on "Embedded Linux" so a lot more people admit that we have to care about it than used to admit this.

[Slide 4 - "Embedded"...?]

And.. You know.. We can consider a lot of things to be embedded even up to perhaps.. um.. devices like netbooks, ultrabooks, tablets. They aren't necessarily the.. um.. typical thing you might have thought of when you think of "Embedded Linux"

2:00 - 3:00:

But they have a lot of the same requirements. Um..

So what makes an embedded device? Is it the fact that it is headless? That it doesn't have a screen and a keyboard? Is it the fact that it is hand-held battery powered? Um.. Cares about power management deeply. Um.. Is it the physical size? I mean is it the constraints of the hardware? The RAM? The storage? The CPU? What else makes embedded?

[gesture to audience]

What do you think?

[inaudible answer from audience]

Dedicated usage? Yes, that's common. That's a common criteria

[answer from audience: non x86 processor]

Yes, so my Mac G5 is embedded? Actually it is, i moved a month ago, it is still packed in boxes. But, yeah. non x86. That's common. Anything else?

3:00 - 4:00:

[inaudible answer from audience]


[answer from audience: No X11]

No X11. Yes, that's true. So there are all kinds of things that people mean when they talk about "Embedded Linux". Um.. But there are a number of things that are quite common.

[Slide 5 - Embedded needs]

Power management is a very common requirement. When people talk about "Embedded Linux" often they think about power management. We need to conserve battery at all costs. We need to shutdown any circuitry which isn't in use right now. We need aggressive run-time power management.

We also think about fast startup. You take your phone out of your pocket, turn it on, there are legal requirements about how soon you must be able to make an emergency call and even in consumer devices outside that. You know.. You have to be able get a consumer experience quickly when they turn a device on. And that's one of the things that people care about a lot when they are talking about "Embedded Linux"

Again, headless operation, we covered this didn't we, no X11

4:00 - 5:00:

So, sometimes no user interface at all when its just an embedded control device. A router. Something like that. Um..

But we care about.. when we do have a user interface.. we care about having clean user interfaces. User interfaces that work on a small screen. That work with a touch screen and that's a common theme as well.

And we care about solid state storage. i'm not going to go [pauses to rephrase] Excuse me. I'm not going to go into any more detail about solid state storage so don't worry

[Slide 6 - Embedded needs]

So these are our needs, but they are not specific to "Embedded Linux". None of them are specific to "Embedded Linux". Everybody else needs these. There is actually a huge cross over between what we need in the "embedded" space and what people doing big iron want as well.

So let's take a look at power management

5:00 - 6:00:

[Slide 7 - Power Management]

We care about it, for embedded, for battery life. We want to make our devices last as long as possible before they shutdown.

[Slide 8 - Power Management]

In a data centre with huge racks of machines running a server farm, what do you care about? You care about power management because you don't want to be running at full cost.. You know.. Running everything at full tilt.. Paying for the electricity to run.. To waste heat.. Paying for the electricity to run your air conditioning at full blast to get the heat out of the building again so you're paying for it twice. Um.. And so actually the power management requirement all the requirement for runtime power management on linux systems. This isn't an embedded requirement at all. This is something everybody cares about and nowdays they are starting to realise they do. So we are seeing a lot more.. Um.. Acceptance of this kind of.. Um.. requirement

6:00 - 7:00:

and we have seen some very good work going in to the linux kernel recently on power management where maybe ten years ago there was a lot more resistance from upstream with people saying "oh we don't care about this kind of thing.. Why would we care".

[Slide 9 - Tickless operation]

So tickless operation. We heard about tickless.. um.. kernel earlier. Um.. And yes, this is really useful.. It gives us big power savings when we are running on battery.. When we don't have to wake up the CPU every thousanth of a second, hundredth of a second and do stuff, just to realise we can go back to sleep again. So, that's great. That's an embedded requirement.

[Slide 10 - Tickless operation]

But, its not just an embedded requirement. If you think about big farms of virtual servers.. the big hosting farms running lots of virtual machines they absolutely do not want every virtual machine every one of thousands of virtual machines waking up every millisecond

7:00 - 8:00:

and doing all this gratuitous stuff just to go back to sleep again because that means that the hypervisor has then got to run every single virtual machine.. And the cost of that is insane you can actually have.. You know.. The physical hardware.. The hypervisor doing nothing but running the timer tick on thousands of virtual machines and having no time to do any real work.

So again we have tickless operation which is not just an embedded requirement.. Its something that we need everywhere.

[Slide 11 - Fast boot]

Fast boot, so i talked about this. We have requirements for mobile phones that you have to be able to make calls within a certain amount of time and its just a very important user experience requirement for many.. You know.. Tablet devices, ultrabooks etc.

We care deeply about fast boot.

There has been a lot of work going into Linux distributions to boot fast, it gets you to the login screen or to the working environment as quickly as possible.

8:00 - 9:00:

[Slide 12 - Fast boot]

But, you might guess where I'm going with this, Um.. big iron needs it too.

If you've got a server hosting some service and you've promised 99.999% availability, how many seconds a year are you allowed to be down? That's less than one reboot. You need to reboot. [speaker re-phrases] If you have to reboot, you're eating into your uptime guarantee very very quickly so you need your servers to come back quickly. Um..

And so fast boot is not just embedded. Its for everybody. Everybody cares.

AUDIENCE>> Even on a netbook there?

>> DAVID WOODHOUSE: Yeah, everybody. Absolutely. Yeah, I care if I get up to the front here, boot up my laptop and you're waiting for 5 minutes while it boots up and I can't do my presentation. Or if you're waiting for 5 minutes while we try to drive Open Office. Um..

9:00 - 10:00:

Yeah, absolutely, everybody cares.

[Slide 13 - User interfaces]

There's nothing that is solely an embedded concern.

And user interfaces as well. There are some specific requirements for handheld touchscreens but really these also cross the board.

[Slide 14 - User interfaces]

We've seen.. Um.. i don't know if they are even very popular but a lot of the concepts do actually make a lot of sense and we've seen that user interface development driven to a certain extent by requirements elsewhere so, yeah.. netbook in particular needs the uncluttered user environment.

[Slide 15 - Solid state storage]

And so, there is so much that is not a requirement for embedded. This is why i don't like talking about "Embedded Linux".

I like to think there is no such thing as "Embedded Linux". There is just Linux.

10:00 - 11:00:

So, flash storage. Um..

[Slide 16 - Solid state storage]

We have flash storage in embedded devices but we are increasingly seeing SSDs in laptops, even in servers. So again, despite me seeing earlier I'd like to see proper flash access in embedded devices. We do seem to be moving towards more and more embedded devices even new android devices having the fake disk: an SSD, an SD card and so, essentially, that's the same type of technology from the embedded device all the way up to the big iron.

And so we care about the same things.. You know.. Getting TRIM working, getting the device sleep working.. All the things that allow us to work around the problems of the "pretend to be a disk" approach. We care about that. Everywhere from..

11:00 - 12:00:

[inaudible question from the audience]

Sorry, What was the question?

[question from audience about whether linux kernel is used inside a SSD]

Inside an SSD? No.

It will be a microkernel of some kind. You know.. A tiny RTOS of some kind. Its very.. You could do, but its very unlikely to be a Linux kernel but there are as many answers to that question as there are different SSDs from different vendors of course. But basically, i don't know of any that would be using Linux

Linux is a good fit for many things but that isn't one of the things that linux is great at. In fact, I spent a lot of time saying that in the early days of "Embedded Linux"... "yes we love linux but don't try to use it for this" and that is one of those i think

12:00 - 13:00:

[Slide 17 - Others...]

So Execute In Place, um.. In the days of NOR flash which could be mapped as ROM and the CPU could access the data directly and run code from it directly it was very useful to have execute in place support so that rather than copying your code, your kernel, whatever, from flash into RAM and running it from there you could actually run it in place this meant that on a tiny device you could get away with a lot less RAM and a much faster startup time at a cost of the fact that running from flash will be slower... just because RAM goes faster.

So execute in place was really useful for that.

But in fact it turns out that execute in place and all the horrible tricks we have to play in the VM to map this page here, this physical thing that isn't RAM, map it in as a page cache, a page of a file etc.

13:00 - 14:00:

that's really useful for virtualisation as well so that support which actually eventually gave us execute in place in the kernel did not come from embedded developers it came from IBM working on S390

So that their virtual machines could share underlying disk devices and map. Basically, the host, the hypervisor, would load a page from the disk into RAM and that physical page would then be shared into all the virtual machines using the same execute in place methodology throughout the linux virtual memory system.

So again, its not just embedded.

[inaudible question from audience]

It has to be supported by the hardware, yes.

14:00 - 15:00:

but there is a software issue in that the virtual memory management system in linux has to be able to cope with the fact that we're not just managing standard pages of RAM that can be read and written to and made dirty and then written back to disk which is basically how all the memory management worked with flags on pages to mark them as dirty and needing writeback etc. You had to cope with the fact that actually what I'm doing is here, is when the userspace memory maps a bit of file, I am giving them physical access directly to the original copy which i must not write to. I cannot write to. Its ROM essentially. and so if they are going to try to write to it, we then have to copy it into a page in RAM and do copy on write etc. The complicated part of that was software. The hardware part, while necessary, from our point of view that was simple either its possible or its not.

If its possible then we have to do the software to cope, right? So the complicated part was the software, yeah.

15:00 - 16:00:

From the hardware point of view, in the embedded case of.. You know.. ROM, oh you know execute in place from flash the flash was there, there was a physical address for the flash which was not RAM and all you have to do is map it, but that was the complicated part.

And DMA APIs. So when you are doing.. when using a peripheral to DMA to or from memory.

A lot of the smaller systems had cache coherency issues so that you had to make sure you flushed the caches at the right time.

Basically um it was a big problem for a lot of drivers, a lot of network drivers in particular, wouldn't get it right. They'd just say right here is your address go and do it and would fail when you put that same device into an embedded system.

Um.. and they wouldn't use the DMA API properly.

16:00 - 17:00:

but we need them to because.. you know.. bigger systems have an I/O MMU now so if you just pick a physical address from the ether and give it to you device it won't work. So now we are seeing all of the drivers being fixed to use the DMA API properly so that they can work with.. you know.. current Intel/AMD systems Um.. with I/O MMU and this means those same drivers are fixed for the embedded use case where we need to use the same API for different reasons. So its just another case. I can't think of many things which really are embedded specific.

[Slide 18 - We are not so special!]

We are not that special. I don't like talking about "Embedded Linux".

I am one of the "Embedded Linux" maintainers for my sins but only under protest because Andrew Morton kind of insisted on there being somebody to do that. and you know most of what I do in that role which is very very little is to say this is not embedded this is.. You know.. This is generic.

17:00 - 18:00:

[Slide 19 - Community impressions]

So, to a certain extent as I said earlier people think of "Embedded Linux", the embedded linux um community, the mainstream kernel development/linux development thinks of Enterprise and Embedded in much the same way as just that stuff we don't want to have to think about.

They like to think of embedded/embedded developers as working with ancient code that nobody cares about anymore. Not working with upstream. Doing their own thing going off into the weeds not contributing their code back upstream. Doing stuff as a one off. You know.. Making some hack not attempting to get it upstream and just living with it and having a lot of pain when later they need to do a product refresh, a new product, update to a later kernel and none of the changes they've made are applicable anymore - they all have to be redone.

18:00 - 19:00:

But I don't like that perception and I'm pleased to see that it has starting to change. We no longer see.. You know.. many people who think of "Embedded Linux" as irrelevant to the general case. I think it is accepted to be a lot more relevant now.

[Slide 21 - Community impressions]

So.. You know.. We must prove them wrong.

I think we are doing a fairly good job of proving them wrong.

[Slide 22 - "Embedded" success stories]

Um.. We have some success stories. All this stuff works. Um.. and you know it shows the cross over between embedded development and what everybody else wants.

I've talked about some of those

Suspend to RAM is another important one. Started to become very important to people with laptops/Linux laptops only a few years ago. Before that desktop linux/laptop linux nobody ever cared or nobody ever at least had the expectation that it might work they just tolerated the fact that you could never suspend

19:00 - 20:00:

Um. And Squashfs which.. you know.. had a very.. um.. a lot of use in the embedded space where storage is tight but it also sees a lot of use on Linux distribution install CDs that kind of thing where again you need to pack a lot of data into a small space. Um.. Its just another example.

[Slide 23 - Working with the community]

So, I talked about.. You know.. We need to prove them wrong. We need to work better with the community. Um.. and these are some of the ways to do that you find the generic points of interest, look for those.. that cross over as i've been pointing out. Show why other people would be interested in what you are doing. Show that it is not just something esoteric that nobody but you would care about. Show your code show your working. Publish git trees. Um..

20:00 - 21:00:

sane ones that people can look at.. not just a git tree for your product with every one of the dozens of.. you know.. bits of kernel development that you might be working on but a separate tree for each driver perhaps, for each topic of development so that somebody who has an interest in what you are doing, somebody that you are trying to get engaged with what you are doing, to see how it is relevant to them make it so they can see it easily its not just in amongst a pile of crap that isn't relevant to them and if they are interested in one topic let them see that and send the patches out for review

And most importantly respond to the feedback that you get.. Some of it is a little unhelpful, there is the whitespace police who will um nitpick the most stupid things and ignore the blatant actual coding errors.. locking errors because they don't quite understand that they can understand the white space and they can tell you about that but.. yeah.. you have to take their feedback with a pinch of salt sometimes but respond to the feedback

21:00 - 22:00:

Um.. and the best way to shut the whitespace police up is not to get it wrong in the first place.. There is actually.. I'll talk about that in a little bit in a minute but.. yeah.. there are a lot of people who seem to post some patches disappear for 3 months, come back again for the next kernel release, want to get their patches merged and post the same patches again without having fixed any of the problems that were pointed out last time and that's a very quick way to get onto email blacklists to people or even just mental blacklists with people saying "i'm not going to read that patch again and i know its the same as last time". so, really, work with the maintainers be part of the community, don't go off and do stuff in isolation if at all possible. Um.. Staying close to upstream Um.. It has its pros and cons. Upstream moves fast and it is hard to be on that treadmill

22:00 - 23:00:

but it is in general worth it to stay close and to basically minimise your delta between what is in Linus' git tree at any moment and what you are shipping in your product. Every time.. you know.. I work on something my viewpoint is getting it to Linus as quickly as possible so that it is not my problem anymore. So that I can just download tomorrow's 3.7 release and it should work. Yes, I have to do the QA and I might not actually update but at least I know that when I do a product revision, when I do a new product, which is.. you know.. supercedes the product I am working on now I should be able to start from a mainline kernel with no patches I shouldn't have to spend weeks forward-porting stuff I did months ago that no longer applies, no longer works, because somebody else has made changes in that area.

23:00 - 24:00:

[Slide 24 - Staying close to upstream]

so, being close to upstream really does make it easier for new products and fixes so if somebody finds bugs in code that you're using but perhaps you haven't been dealing with it much. Those fixes are a whole lot easier. I appreciate there are some insane people who spend a lot of time backporting code to earlier kernels, but they still tell you should update if you can.

Right? Um.. It makes it a lot easier to take in external contributions. So, if somebody else is doing some work that you can use, they're likely to be working on a newer kernel as well so the closer you are the easier it is to use other peoples work, to collaborate with people and the more likely people are to review and test your code. It just makes life so much easier.

Yes. um. There is a cost inherent in writing code that can go upstream.

It is harder.

24:00 - 25:00:

It can, if you're not used to it especially, take more time. If you post code that doesn't pass muster in the first place and you have to review it and go through all of the feedback. Um.. then, yes that can be hard. And then the second point there [gestures to second Costs bullet] that it is a fast moving target and you're having to do these iterations of your own code perhaps the kernel has moved on by the time you've got your own house in order.

So, that's hard and you may have NDAs. You may have problems with information about your product leaking because you are working on certain kernel features, you might have marketing reasons essentially why you might not be able to release code sooner.

So, there are some costs. But I think the biggest one there is if you get it right the first time. It shouldn't take any longer. It should only be a benefit and if you get it right the first time, then you should be able to release it later.

25:00 - 26:00:

Now, Android is often a bad example of many things but they are getting better. They're doing.. They're getting there.. they are using current kernels they are relatively aggressive in updating to new kernels and getting their code merged.

There is still a fair amount of stuff outstanding but I think.. I think with a stock 3.6 kernel you can run Android user space. Am I right in that? I think so.

They have finally basically got everything at least in the staging tree and they are getting there.

[Slide 25 - Tips on contributing code]

So.. This is.. Yeah.. I've talked about this. Find parallel requirements.

Avoid hacking around problems. One of the things that a lot of people in the embedded space do, if they come from working on another operating system.. um.. where they don't have as much visibility into the core OS

26:00 - 27:00:

so the assumption is.. the first thing they think to do is to work around the problem in their own driver so rather than going fixing the infrastructure so that it supports your feature properly and generically you'll do some local hack. Don't do that. Um.. Its a really bad idea to do that kind of thing. It will get your code rejected. We saw a lot of this, when i was working on the OLPC with wireless mesh networks, and this had been done by Marvell as a wireless driver with some extra mesh support bolted onto the side with special IOCTls - which was the wrong way to do it. A lot of the work we had to do on that driver, aside from just fixing the code, was um defining a generic API for some of the mesh functionality and putting that into the core and then exposing it through the driver in a sane way rather than having it as a driver specific hack.

27:00 - 28:00:

Another thing people often get wrong is not caring about locking. Now this is perhaps less true these days where we have rt-preempt kernels but in the days where people would run embedded devices on a uniprocessor system where spinlocks were a complete no operation you would see a lot of locking bugs in drivers that um would be picked up in review and just mean you have to do more work to go and fix that if you didn't get it right in the first place.

The coding style, i talked about the white space police um.. There is some merit in what they do. If you imagine reading a book and every time you turn a page the new page is in a slightly different font with slightly different layout and borders, that's going to make it slower for you to read. Every time you turn a page your eyes, your brain is going to have to adjust to reading the new style and it makes it harder to understand.

28:00 - 29:00:

And that's why people care about stuff that might just seem like gratitous pedantry. Um.. I might suggest that perhaps they don't need about.. to care quite as much as they do but there is some merit

[Slide 34 - Questions?]

This wasn't supposed to be.. This isn't even the right presentation! Joy.

I had some more slides with details of some of the white space, coding style errors.. Um.. [pauses] Bear with me.. Sorry.

29:00 - 30:00:

Sorry. The perils from not presenting from my own laptop.

30:00 - 31:00:

[speaker looks for correct presentation on another laptop]

31:00 - 32:00:

Okay. I give up.. I was going to talk.. I was going to show, it doesn't work quite so well in voice, Um.. some of the problems that we get with erm um patch submissions.

There are a lot of things that um that you should do before sending a patch that are just cosmetic that you have absolutely no excuse for getting wrong. Um.. and i can show you off line an example but i was going to put it up on the screen and say can you see the ten errors in this slide and then i was going to show you the answers and they were such things as just white space in the wrong place but patches get mangled by your email client and introduce word wraps.

When you send a patch, first send it to yourself.. well actually before you send it to yourself, read it. Patches are human readable, you can look at it, you can understand it. You should understand it.

32:00 - 33:00:

But.. You know.. Unified diff patch is very easy to read. It has lines that start with a space, which are lines of context what is there both before and after your change. Lines that start with a minus sign, that are lines that get removed by the patch. Lines that start with a plus are lines which get added which are often very similar to the line that gets removed. Um.. You should be able to read that. You should be able to understand it. You should be able to look and see superfluous changes. Often you see patches with a whole bunch of completely gratuitous changes that should not be there so adding blank lines adding spaces or whatever in the middle of a line that hasn't otherwise changed, its a mess. It makes the patch hard to read. If you send a patch like that you have obviously not even read your own email before hitting send. Why would you expect people to take you seriously? So, first read your email before you send it

33:00 - 34:00:

Then send it to yourself. Save the resulting email to a file. Then attempt to apply that as a patch. If it doesn't work, go back to step 1 Work out why. If you are using Exchange and its converted all tabs to spaces. Work out if you are sending HTML. If its word wrapped, make it stop and then, there are a bunch of other things about just abiding by the coding style of the kernel so that people can understand and can read it naturally and it comes straight into their brain without them having to interpret what they see in front of their eyes because that's the reason for the coding style rules its to make stuff easy to read for the people who spend all day reading it and don't want to try to interpret different styles of code um so there are a few things along those lines i think i will try to share the rest of the slides off line if people are interested

So, um, do we have any questions?

34:00 - 35:00:

[inaudible question from audience]


[inaudible question from audience: continued]

Yes, very true. Use git send email that works nicely, but again make sure you send it to yourself first. um.. even with git send email its useful to read through the patches before you send them

so, yeah, that's a good trick. git send email

[inaudible question from audience]

No, I didn't.

35:00 - 36:00:

I haven't worked on many things like that thankfully. um. I know Karen Sandler of Gnome Foundation has been talking recently about her pacemaker and that's a very interesting presentation if you get a chance to see that. um, but no I've.. I haven't even heard particular horror stories of where JFFS2 is being used that would scare me. So, no, nothing interesting like that I'm afraid.

Ok. Thank you.