Guidance on Tools for Implementing Dynamic Plugin System in Firmware

Guidance on Tools for Implementing Dynamic Plugin System in Firmware

[Massimiliano Cialdi]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

Hello,

I am engaged in a firmware project with Cortex-M microcontrollers, focusing
on implementing dynamic plugin functionality. My setup includes FreeRTOS,
LittleFS, and gcc-arm-none-eabi.

I'm exploring creating a dynamic linker/loader for plugins. My queries are:

   1. *File Format and Loading:* Recommendations for file formats and
   loading strategies in a dynamic linker/loader system?
   2. *Function Entry Points:* Best practices for managing function entry
   points in non-PIC firmware for plugins?
   3. *Memory Management:* Managing different memory types (DTCM and SRAM)
   for plugin code and data.
   4. *Compilation and Linking:* Specific parameters or considerations for
   compiling and linking, including post-linking steps.
   5. *Security and Stability:* Ensuring safety and reliability when
   loading plugins dynamically.
   6. *Toolchain Adequacy:* Are the standard tools in the gcc-arm-none-eabi
   toolchain sufficient for this task, or is there a need for third-party
   tools or custom solutions?

Additionally, *I am seeking recommendations for any documentation,
tutorials, or learning resources* that could help deepen my understanding
of these topics, particularly in the context of firmware development for
ARM Cortex-M microcontrollers using FreeRTOS.

Any advice, experiences, or references to similar projects would be
immensely helpful. Thank you!
best regards

-- 
Et nunc, auxilium solis, vincam!
Oppugnatio solaris!
VIS!

Massimiliano Cialdi

Re: Guidance on Tools for Implementing Dynamic Plugin System in Firmware

[Nick Clifton]

Hi Massimiliano,

> I am engaged in a firmware project with Cortex-M microcontrollers, focusing on implementing dynamic plugin functionality. My setup includes FreeRTOS, LittleFS, and 
> gcc-arm-none-eabi.
> 
> I'm exploring creating a dynamic linker/loader for plugins. My queries are:

Most of your questions are going to be outside of the scope of the
binutils project so you might find asking on other mailing lists,
eg glibc and/or the kernel, to be helpful.

>  1. *File Format and Loading:* Recommendations for file formats

Whatever file format your OS supports.  This will probably be ELF.

 > and loading strategies in a dynamic linker/loader system?

K.I.S.S.!  Ie go with the simplest strategy that works.  Do not
bother with speed optimizations or anything like that to start
with.  Get your linking and loading working first.  Then if you
have time, work on improving them.


>  2. *Function Entry Points:* Best practices for managing function entry points in non-PIC firmware for plugins?

I am not sure what you mean here.  But my general advice for
implementing a new feature or system is to follow three simple
steps:

   1. Write the documentation.
   2. Write the testsuite.
   3. Write the code.

Writing the documentation and testsuite first will guide your
thinking and help you nail down exactly what you want to do and
how to do it.  Writing the code then becomes a lot easier.


>  3. *Memory Management:* Managing different memory types (DTCM and SRAM) for plugin code and data.

Meh - not really in the scope of the binutils.  You decide how the
memory should be used and then you create linker scripts and/or
loader configurations to implement those decisions.


>  4. *Compilation and Linking:* Specific parameters or considerations for compiling and linking, including post-linking steps.

Copy how this is done for already existing plugins.


>  5. *Security and Stability:* Ensuring safety and reliability when loading plugins dynamically.

No such thing.  Plugins are inherently insecure and unstable.


>  6. *Toolchain Adequacy:* Are the standard tools in the gcc-arm-none-eabi toolchain sufficient for this task, or is there a need for third-party tools or custom solutions?

Only your experience can answer that question.  But I can say
that  the GNU tools have been used in this area for a long time
now, so they are likely to do (most of) what you need.


> Additionally, *I am seeking recommendations for any documentation, tutorials, or learning resources* that could help deepen my understanding of these topics, particularly 
> in the context of firmware development for ARM Cortex-M microcontrollers using FreeRTOS.

You would have to ask the FreeRTOS community for that advice.

As far as the binutils go there is a reference work that covers
most of what linkers and loaders do:

   Linkers and Loaders by John R. Levine


Cheers
   Nick

Re: Guidance on Tools for Implementing Dynamic Plugin System in Firmware

[Massimiliano Cialdi]

On Mon, Nov 27, 2023 at 12:46 PM Nick Clifton <nickc@redhat.com> wrote:

> Most of your questions are going to be outside of the scope of the
> binutils project
I apologize if this is off-topic, but I think at least one question
may be appropriate for this mailing list.
For this reason, I will just put my use case in more context.

> so you might find asking on other mailing lists,
> eg glibc and/or the kernel, to be helpful.
You assume that I have a high-level OS. Although I use FreeRTOS (which
is little more than a scheduler), for the purposes of the answers I
seek consider my project as a bare-metal


> >  1. *File Format and Loading:* Recommendations for file formats
> Whatever file format your OS supports.  This will probably be ELF.
My OS does not support any format, so I was looking for one that is
"easy" to design and write, or already used in similar contexts.

> >  2. *Function Entry Points:* Best practices for managing function entry points in non-PIC firmware for plugins?
> I am not sure what you mean here.  But my general advice for
> implementing a new feature or system is to follow three simple
> steps:
In my plugin implementation scenario, unlike typical cases where
plugins solely provide functionalities to the host, my plugins will
also need to utilize APIs offered by the firmware. I've deduced that
the plugins should have two function pointer tables: one for the
functions they provide and another for the functions they use from the
firmware. Hence, I believe the linker/loader should populate this
second table. Additionally, the firmware must be designed to retrieve
function pointers at runtime for invoking functions provided by the
plugin. How can I effectively manage these function entry points and
ensure seamless interaction between the firmware and the plugins?

> >  4. *Compilation and Linking:* Specific parameters or considerations for compiling and linking, including post-linking steps.
>
> Copy how this is done for already existing plugins.
Unfortunately, the only two references I have found so far are of no help to me

Searching the Internet, I found a couple of projects:
- Linkable Loadable Extensions (LLEXT) in Zephyr OS
(https://docs.zephyrproject.org/latest/services/llext/index.html
Unfortunately, I am forced to use FreeRTOS, so I cannot use this
solution. Besides, it's still experimental and not yet stable or even
released
- Udynlink (https://github.com/bogdanm/udynlink the project is a bit
old, and of "pre-alpha" quality. Other than that it seems very rigid
and insufficiently flexible regarding the compilation process. There
are a number of forks that I should study better.

In light of the requirement for dual function pointer tables in my
plugins (one for functions they provide and another for functions they
use from the firmware) I am seeking guidance on the compilation
process. I anticipate needing to use flags like -fPIC or -fPIE and
link with options like --unresolved-symbols=ignore-in-object-files and
--emit-relocs. However, I would like to know what additional
parameters are necessary. Specifically, are the standard compiler and
linker sufficient to 'create' these tables, or is there a need for any
post-build steps? Should I anticipate the need to utilize existing
tools or develop custom solutions for this purpose?
In addition to understanding the compilation process for the plugins,
I also need guidance on how to modify the compilation of my firmware.
Specifically, how should I adjust the firmware's compilation process
to enable or facilitate the retrieval of function pointers that the
plugins require? Is there a particular approach or modification
necessary to ensure smooth interaction between the firmware and the
plugins, especially in the context of function pointer handling?

I hope this last question is enough to consider this post relevant to
the mailing list

regards
Max
-- 
Et nunc, auxilium solis, vincam!
Oppugnatio solaris!
VIS!

Massimiliano Cialdi

Re: Guidance on Tools for Implementing Dynamic Plugin System in Firmware

[Nick Clifton]

Hi Max,

> You assume that I have a high-level OS. Although I use FreeRTOS (which
> is little more than a scheduler), for the purposes of the answers I
> seek consider my project as a bare-metal

Ah!  Thanks for the clarification.

>>>   1. *File Format and Loading:* Recommendations for file formats
>> Whatever file format your OS supports.  This will probably be ELF.
> My OS does not support any format, so I was looking for one that is
> "easy" to design and write, or already used in similar contexts.

Well I would still suggest ELF, even though technically speaking it is
actually more complex than other file formats (eg COFF).  The reasons
for this are that the ELF format is very well documented and there are
lots of tools for inspecting and manipulating ELF binaries.

If however simplicity truly is your goal then maybe the a.out format
would be the best choice:  https://wiki.osdev.org/A.out

Also if you really are programming at the bare-metal level then you
may find that defining your own - very simple - file format will be
the best solution.  You could have just the features that you need
and nothing else, and keeping things simple always helps to make a
project easier to manage.

Do you anticipate the need for a file system ?  Ie are you going to
have these plugins exist as separate entities to be loaded on demand
or will there just be a single monolithic binary containing both
firmware and plugins ?

>>>   2. *Function Entry Points:* Best practices for managing function entry points in non-PIC firmware for plugins?
>> I am not sure what you mean here.  But my general advice for
>> implementing a new feature or system is to follow three simple
>> steps:
> In my plugin implementation scenario, unlike typical cases where
> plugins solely provide functionalities to the host, my plugins will
> also need to utilize APIs offered by the firmware. I've deduced that
> the plugins should have two function pointer tables: one for the
> functions they provide and another for the functions they use from the
> firmware.

In linker terminology I think that these would be called "exported" functions
for the API provided by the plugin and "imported" functions for the ones that
the plugin expects to be provided by the firmware's API.

> Hence, I believe the linker/loader should populate this
> second table. 

And the first one too.  Also in some file formats these two tables are
combined into one, with symbols having a type to indicate if they are
exported or imported.

> Additionally, the firmware must be designed to retrieve
> function pointers at runtime for invoking functions provided by the
> plugin.

Really ?  I would have expected it to be the other way around.  ie a plugin,
once it becomes active, should register with the firmware all of the function
pointers that it wants used.  So the firmware is passive and does not have to
retrieve anything.  Instead it is the plugins that must tell the firmware which
parts of its API they want to use.

> How can I effectively manage these function entry points and
> ensure seamless interaction between the firmware and the plugins?

Well this is obviously the job of the linker and loader combined.  Plus you
will need some way to annotate the sources for the plugins and the firmware
so that when they are compiled/assembled they can indicate which functions
they are exporting and which functions they are expected to be able to import.


>>>   4. *Compilation and Linking:* Specific parameters or considerations for compiling and linking, including post-linking steps.

> In light of the requirement for dual function pointer tables in my
> plugins (one for functions they provide and another for functions they
> use from the firmware) I am seeking guidance on the compilation
> process. I anticipate needing to use flags like -fPIC or -fPIE and
> link with options like --unresolved-symbols=ignore-in-object-files and
> --emit-relocs. However, I would like to know what additional
> parameters are necessary. Specifically, are the standard compiler and
> linker sufficient to 'create' these tables, or is there a need for any
> post-build steps?

In theory the standard compiler and linker should be sufficient (if you are
using one of the file formats mentioned above and not rolling your own).
The firmware would be represented as a library file, either shared or static
depending upon how you intend all of this to work.  This library should
provide symbols to satisfy all of the unresolved references found in the
plugin's code.  Meanwhile the plugin's API should be provided as a set of
globally visible symbols in its symbol table.  Handling libraries, resolving
symbols references and creating symbol tables are all things that the
linker should - in theory at least - be able to handle.

> Should I anticipate the need to utilize existing
> tools or develop custom solutions for this purpose?

Possibly.  It is hard to say without a specific design in place.


> I hope this last question is enough to consider this post relevant to
> the mailing list

Fair enough.

I have given my thoughts above.  Maybe others will also be willing to share
their insights.

Cheers
   Nick

Re: Guidance on Tools for Implementing Dynamic Plugin System in Firmware

[Massimiliano Cialdi]

On Mon, Dec 4, 2023 at 1:25 PM Nick Clifton <nickc@redhat.com> wrote:
> Well I would still suggest ELF, even though technically speaking it is
> actually more complex than other file formats (eg COFF).  The reasons
> for this are that the ELF format is very well documented and there are
> lots of tools for inspecting and manipulating ELF binaries.
>
> If however simplicity truly is your goal then maybe the a.out format
> would be the best choice:  https://wiki.osdev.org/A.out
it would be great to have an elf interpreter inside my FW bare metal.
Do you know of one that would fit? Keep in mind that in the bare metal
context I don't have any posix calls or system calls, and even the
malloc has to be redirected to the one I have available.
Alternatively do you know of any a.out interpreters?

> Also if you really are programming at the bare-metal level then you
> may find that defining your own - very simple - file format will be
> the best solution.  You could have just the features that you need
> and nothing else, and keeping things simple always helps to make a
> project easier to manage.
for the time being I would prefer a format that is ready-made, and
used by a few dozen people in the world ;-) gives me more guarantees.
Then at a later time, if there is a need, I will see if and how to
optimize.

> Do you anticipate the need for a file system ?  Ie are you going to
> have these plugins exist as separate entities to be loaded on demand
> or will there just be a single monolithic binary containing both
> firmware and plugins ?
Yes I will have a filesystem (littleFS) and mass storage of a few
megabytes (between 2 and 16). The plugins will be autonomous separate
entities. All the ones on FS (in a certain path) will be loaded at FW
startup, at least in the scenario I have in mind at the moment. Maybe
in the future I might think of a more dynamic use case still.

> > Additionally, the firmware must be designed to retrieve
> > function pointers at runtime for invoking functions provided by the
> > plugin.
> Really ?
Yes, I was "inspired" by dlopen(), dlsym(), etc.
I thought: I load a plugin, try to retrieve, for example, the
functions "init" and "compute," if I find them I can call them,
otherwise I unload the plugin and report the error.

> > How can I effectively manage these function entry points and
> > ensure seamless interaction between the firmware and the plugins?
>
> Well this is obviously the job of the linker and loader combined.  Plus you
> will need some way to annotate the sources for the plugins and the firmware
> so that when they are compiled/assembled they can indicate which functions
> they are exporting and which functions they are expected to be able to import.
for the plugin, I don't think there is any problem to "annotate" the
symbols I want to export. For my FW it might be much more difficult,
because it may happen to use third-party lbraries distributed only as
static libraries (so I don't have the source files).

> In theory the standard compiler and linker should be sufficient (if you are
> using one of the file formats mentioned above and not rolling your own).
> The firmware would be represented as a library file, either shared or static
> depending upon how you intend all of this to work.  This library should
> provide symbols to satisfy all of the unresolved references found in the
> plugin's code.
was what I thought too

> Meanwhile the plugin's API should be provided as a set of
> globally visible symbols in its symbol table.  Handling libraries, resolving
> symbols references and creating symbol tables are all things that the
> linker should - in theory at least - be able to handle.
maybe I should write an example, at first even without filesystems,
just to start working on something concrete. Is that a fair line in
your opinion?

best regards
Max


-- 
Et nunc, auxilium solis, vincam!
Oppugnatio solaris!
VIS!

Massimiliano Cialdi