SmallSat 2023 Linux4Space birds-of-a-feather meeting

There was a "Linux in Space" birds-of-a-feather meeting held at the SmallSat conference in August of 2023.

Here is some information about that meeting: = Slides =
 * [[Media:SmallSat-linux-in-space-BOF.pdf|PDF]]

= Meeting Notes = Here are some notes from the Linux4space Birds-of-a-feather meeting

The meeting was held:

Wednesday, August 9, 11:00 am to 12:00, Room 336 in TSC, USU, Logan, Utah

Attendees:

 * 25 people signed a pass-around list to receive slides
 * Not sure it's appropriate to share names and/or e-mails without explicit permission
 * There was a good variety of organizations and individuals there.

Discussion Notes:
Increasing use of Linux in space missions seems inevitable.

Tim presented the status of Linux, talking about:
 * Linux cadence
 * realtime status
 * eBPF
 * io_uring
 * Rust support in the kernel
 * historical areas of focus for embedded Linux

See the presentation for slides on these topics

Discussion in meeting:

Do satellites need realtime? - lots of satellites have 30Hz (or lower) duty cycles - this doesn't require realtime - some satellites don't need realtime for actuators, but still care a lot about determinism (e.g. starlink) - Tim raised the point: valves require a millisecond to respond, so does Linux realtime need better latency response than that? - Yes, because there are things with tighter control loops, like motor controllers, opto-mechanical systems, that need quick control - (it sounds like stuff on payload computers may or may not need      realtime handling) - when interfacing with external hardware, determinism is highly desirable - realtime is often put onto microcontrollers off the main OBC, but those still need to be controlled with fine timing precision

Does Linux need small size for space? - yes - not for storage necessarily, but for updates. - is better is update image is small - satellites and space mission are often constrained on communications bandwidth - the size of an update package can be very critical - note: it might be worth investigating update deltas instead of full images

Does Linux need fast bootup time? - On Starlink, processors not in use are turned off to save power - being able to turn them on quickly is nice - this means boot time becomes an element of power management - if a satellite reboots, then mission is inoperative (or at least    non-communicative) during the computer off time - may lose 20 seconds of (either science or commercial) data while computer reboots - would be better to lose only 1 second of data (for science or mission) - (satellites are constantly moving in earth orbit, so they might        miss something if they reset at the wrong time) - lots of satellites prefer to warm boot, if possible, to save bootup time - nobody seems to be using snapshot booting for quick boot

What about drivers: - spacewire - there is proprietary spacewire hardware - but there are also open cores for spacewire - what busses are used (lots of different ones) - often, the control for a specialized piece of hardware has its own custom controller, bus or protocol

- people don't want to write Linux kernel drivers to control weird hardware - that's too hard - some drivers written in user-space

- drivers need to get support for FPGA instantiations of the hardware - need "dynamic function exchange"? (what's that?) - need dynamic device tree to handle dynamic hardware - there's lots of issues hooking of weird cameras and sensors, for which there either aren't Linux drivers, or the connection to the device is customized - sometimes develop drivers, but can't publish upstream due to NDA - this is a very common problem with embedded Linux

Things that need more work (or need to be easier to use): - virtio, rpmsg, openAMP - for heterogeneous systems - to communicate with stuff running on other processors and controllers in the system - there are lots of different boards and controllers on a satellite

- Linux recently broke spidev - ability to write a user-space SPI driver - need to check on this - some developers are using python to write SPI drivers

What about security: - physical access is not possible, but... - must have secure systems (communications, shell access, and updates) - for SpaceX, they require an architecture that has secure boot features - use crytographically signed components for secure boot - (that implies crytographically secure update bundles)

What about bootloaders: - U-boot now supports HTTPS - starlink has boards within a satellite that update from a central computer. Those currently use UDP, so HTTPS/TCP might be useful - what bootloader do people use? - everyone seems to be using u-boot - but u-boot is not quick at booting - it does stuff that Linux then re-does when Linux boots - there are some bare-metal bootloaders that boot Linux

- would be nice to have u-boot support features especially for cubesats - update, selection of other storage, selection of board or partition to boot (I think it does the last one already) - (would be nice to have boot decision based on external factors, like      flash data)

There is a lot of need to interface between FPGA cores and custom hardware. FPGA manager needs work.

Build systems: - not many people have heard of Yocto Project (a few have) - but they may be using it without knowing it  - People using YP are doing one of: 1) building it themselves using YP directly    2) use PetaLinux, which uses Yocto Project underneath 2) get their kernel image from a supplier, and don't mess with it - lots of projects are based on buildroot

Introduction to Linux4Space project - just started last year (Czech one) - There's a Czech one and a French one - plan to deliver Linux distribution customized for space - based on Yocto Project - are matching Linux distribution features to ESA requirements - question: is there enough overlap between mission requirements to  justify a shared distribution? - it's unclear - people use different flight software, different drivers, different software stacks, etc.

Interesting uses of Linux: - estimate that about 50% of cubesats use Linux in some way

- Starlink constellation is very successful - Starlink constellation now has 330,000 linux computers - 500,000 (half million) years of linux processing - starlink has almost no rad-hardened parts (not main Linux CPUs anyway) - 2 or 3 COTS parts is still less expensive than rad-hardened parts - and may also use less power, while still providing better performance

- Planet Labs constellation is successful

- Ingenuity helicopter - how is it still flying? - thermal swings really huge during Mars winter - lost inclinometer, but other hardware still working well - it was only designed for 30 days - not designed for Mars winter - amazing that it's 2 years in and still working - how big a factor is selection filtering for COTS parts? - it helps, but is hard to quantify - lots of use of Automotive parts, which have to handle vibration and some thermal extremes (but not like space) - have problems getting BW image at 30 Hz, when one is missed then timestamps get messed up     - flight 6 anomoly was related to this - ended up having to use convoluted GPIO solution to get accurate timestamp

Miscelaneous: - Tim asked if a Linux problem ever caused loss of mission - Ingenuity dropping camera frames might be a Linux issue

Radiation tolerance: - some projects have already been done to characterize SEU and reset rates in orbit - memory is a radiation sensor - thousands of satellites, with data sharing, might yield data useful to everyone - Southern Atlantic Anomaly definitely shows up in the data

Closing thoughts - would be nice to get white paper or hard data on realtime performance - would be nice to have a community where Linux experience is shared - coming up to speed on space AND Linux from scratch is hard for students - would be nice to create a mailing list for interested parties