New FDT format

Goals

 * Reduce size of FDT
 * On disk or other storage media
 * In memory


 * Reduce cpu overhead of accessing FDT
 * Unflattening
 * Direct access of FDT if not unflattening


 * Add functionality to overlays


 * Improve ability of de-compile to reflect original source


 * Improve phandle support

Non-goals

 * Remain compatible with previous version if possible
 * This version is expected to create a break in compatibility
 * Should attempt to identify all desired features that will break compatibility so next compatibility break will be far in the future

Previous Proposals
This topic has been under discussion for several years. This section is an attempt to document previous proposals as of April 2021.

people involved in the discussion
"Pantelis Antomarek.vasut@gmail.comniou"  Alan Tull  David Gibson  Geert Uytterhoeven  Grant Likely  Jon Loeliger  Kyle Evans  Michael Ellerman  Pantelis Antoniou  Rob Herring  Tom Rini  marek.vasut@gmail.com

"devicetree-spec@vger.kernel.org"  "devicetree@vger.kernel.org" , Devicetree Compiler 

plus Plumbers 2018 attendees

initial email
Initial email in the thread. This email asks for other people to suggest their needs that would require an FDT format change and provides some size data for the proposed new format. This email does not describe the new format, which will be done later in the thread. This proposal was aimed at the changes to better support overlays.

Summary:
 * new FDT version
 * backward compatible (except for overlays)
 * replace overlay nodes __symbols__, __fixups__, and __local_fixups__ with a symbol table
 * data fields that validation tools could optionally populate, and boot loaders and kernels could optionally require
 * significant size reduction of FDT that contains symbol information needed to apply overlays
 * on storage media
 * in memory
 * small increase in FDT that does not include symbol information for overlays (~ 100 bytes)
 * simplifies the implementation of the Linux kernel overlay apply to the live tree, post-boot
 * allows easy implementation of Linux kernel overlay application in early boot, in association with FDT unflattenning

Size reduction data was calculated for all of the .dts files in the Linux kernel source tree at arch/arm/boot/dts/ as of v4.15-rc7. The key column is "saved", which is how many bytes smaller the .dtb with symbols for the new format compared to the existing format.

proposed implementation
The proposed implementation was provided in a reply to the initial email.

For reference, the existing FDT version 17 format contains * header * mem_rsvmap block * dt_struct block * dt_strings block

The summary here is slightly different than the proposal in the email (removing some ideas that have not aged well).


 * new header field containing the offset of a new block "blocks"
 * this block would contain the offsets of all other blocks, including the blocks present in version 17
 * this would allow the existing block offsets to be removed from the current non-contiguous locations in the header, at the expense of breaking backward compatibility
 * this might allow adding additional blocks in future FDT versions without breaking backward compatibility
 * this adds slight code complexity to accessing the base location of each block (one added level of indirection)
 * REVIEW COMMENTS:
 * just append to header, maybe as an array
 * current scatter of offsets in header is ugly, but not bad enough to warrant the extra complexity of the "blocks" block
 * does not allow adding additional blocks in future FDT versions without breaking backward compatibility
 * new header field "chained" - probably a bad idea
 * REVIEW COMMENTS: negative comments
 * new header field "phandle_delta"
 * to handle a Linux kernel implementation detail, this would be better handled inside a kernel variable
 * new block "csums"
 * contains a separate checksum for each block
 * based on a suggestion by Grant Likely to be able to detect if a bootloader that did not understand the new version modified the FDT in a manner that corrupts the new data
 * probably overkill because the overall FDT checksum should detect the corruption
 * REVIEW COMMENTS:
 * no reason to have an extra block instead of fields in the "blocks" block or extra fields in the main header.
 * need to specify a checksum algorithm
 * three blocks to provide the overlay metadata, in a form that is more compact than the existing form, and easier to parse
 * "ext_phandle_use"
 * analagous to the data in the __fixups__ node
 * "int_phandle_use"
 * analagous to the data in the __local_fixups__ node
 * "symbols"
 * describes the phandle properties in the dt_strings block
 * REVIEW COMMENTS (overlay metadata):
 * need a more coherent description
 * metadata will need updating with any insertion or deletion in the tree, which is a bit of a pain.
 * (dgibson) Instead of adding new blocks, add new tags to the structure block
 * FDT_EXTERNAL_PHANDLE with a property offset and strings table offset would replace a __fixups__ entry
 * FDT_INTERNAL_PHANDLE with just a property offset would replace a __local_fixups__ entry.
 * They don't need an explicit property reference, because they would just apply to the immediately preceding property. **** That approach means we're back to local data, which can be shuffled around pretty easily for inserts and deletes. You'd have to adjust offsets in the fixups for one property when it was altered but not any further away than that.
 * Whenever you insert and delete (for reals, not with nops) you need to adjust all the offsets in the fixup blocks - and remove any fixups that reference something in a deleted chunk.
 * Nops don't require offset adjustments but you *still* must remove any fixups pointing into the nopped region.
 * new block "validate"
 * information that describes the results of any validation of the FDT and/or the devicetree source that the FDT was created from
 * REVIEW COMMENTS:
 * could go into the main header
 * pretty poorly defined
 * yes, it was presented as a concept that should be considered by people implementing the validation tools/process
 * add a field at the end of the FDT, "footer_magic"
 * a check to determine if a transformation of the FDT (eg by an external tool or by a bootloader) was started but not completed
 * this should not be needed - this situation should be detected by an invalid checksum
 * REVIEW COMMENTS:
 * like the idea of adding a footer, to detect truncated blobs, but poorly worded description in the proposal
 * (comment on a side effect of the "footer_magic" field) I'd be happy enough for the new version to say that totalsize must always be aligned to 32 (or even 64) bits, too
 * General REVIEW COMMENTS:
 * Any addition of blocks to the blob makes libfdt's job a lot harder for write operations. Juggling the 3 existing blocks is already pretty awkward.
 * paraphrase: given challenges to maintain compatibility, wondering how valuable backwards compatibility actually is.
 * If we drop the requirement for backwards compat, it beomces possible to encode the fixup information in a much more natural and easy to handle format. Instead of adding new blocks, we add new tags to the structure block.  So, say FDT_EXTERNAL_PHANDLE with a property offset and strings table offset would replace a __fixups__ entry, and an FDT_INTERNAL_PHANDLE with just a property offset would replace a __local_fixups__ entry.  They don't need an explicit property reference, because they'd just apply to the immediately preceding property.
 * That approach means we're back to local data, which can be shuffled around pretty easily for inserts and deletes. You'd have to adjust offsets in the fixups for one property when it was altered but not any further away than that.
 * It also extends easily to add path fixups as well.

Linux Plumbers 2018
video note: The video links include timestamps into the one single [video].


 * [video] DT memory (kernel), DT memory (bootloader), storage (FDT) size
 * [[Media:Device_tree_format_v18_181112_1926.pdf | [PDF]]] "Device tree format v18", Linux Plumbers Devicetree track November 2018, Simon Glass
 * [[Media:Proposed_new_DTB_format_181112_1926.pdf | [PDF]]] "Proposed new DTB format", Linux Plumbers Devicetree track November 2018, Simon Glass
 * [[Media:Size-kernel-fdt.pdf | [PDF]]] "Linux kernel memory size, FDT size", Linux Plumbers Devicetree track November 2018, Frank Rowand
 * [video] New FDT format & Overlays
 * [[Media:Fdt_format.pdf | [PDF]]] "FDT Format", Linux Plumbers Devicetree track November 2018, Frank Rowand

Linux Plumbers 2018 etherpad notes
https://etherpad.openstack.org/p/Device_Tree

local cache of etherpad notes