Fluent Interop 1.0

I am quite pleased to announce that I have finally built a working version of the Fluent Interop library proposed here. The goal of this library is to provide a composable and friendly way to write speed-critical routines such as twiddling output bits in cases where the .NET side cannot be made to run fast enough.

While it is certainly possible to compile any custom code that one might want into the firmware, there are many disadvantages to this:

• lengthy compile-run-test cycle
• requires knowledge of C++ and toolchain
• not shareable with others unless they too are willing to reflash their firmware

The fluent interop system requires a firmware change as well, but the difference is that the change need only be deployed once. What is required is an entry point in the firmware that makes the processor jump to the base address of an array of code in RAM. Once that change is in place (one can even imagine it becoming a standard part of the firmware in some future release), then any custom code can be run, with no reflashing, simply by cooking up the right opcodes in an array on the C# side.

To show just how zippy native code can be, here is a video that compares a simple algorithm running on the .NET side to the same algorithm running on the native side:

http://www.youtube.com/watch?v=9EcEUbtgO2I

And here are some statistics, cut from the output of the program:

• CSharp code ran in 0 minutes, 33 seconds, and 886 millseconds
• Compiling fluent code ran in 0 minutes, 22 seconds, and 414 millseconds
• Run Native code 1X ran in 0 minutes, 0 seconds, and 410 millseconds

From these stats we can conclude that the native code for this algorithm runs 82 times faster than the C# code, which is pretty sweet, and is a promising sign for potential real-world uses like the BitBanger. (I would do more direct measurements if I could, but Santa didn't bring me my Saleae logic analyzer like he was supposed to).

It's kind of a bummer that the compiler took 22 seconds to run. It's slower than I had hoped. Maybe some improvements are possible there; I'm not sure. In any case, one usage model is to perfect your code offline (perhaps running the compiler under the NETMF emulator) and then when you're done, paste the resulting opcodes into your target program.

Now to provide a little more background. The fluent interop system comes in three parts:

• A new entry point in the firmware which allows the processor to jump to an array of opcodes stored in RAM. These opcodes could come from anywhere (so long as they are valid Thumb opcodes conforming to the proper calling convention).
• A C# interface to that library
• The Fluent Interop "compiler", a library which transforms programs that are written in a funky meta-language into Thumb opcodes.

The first two parts are very small. The change I made to the firmware is trivial, and so is the C# interface to it. This means that you can embed Thumb opcodes (whether generated by my freaky system or obtained elsewhere) into your program in a very low-overhead manner. The key benefit here is that you can run those opcodes out of RAM, which means that you can change them at will without needing to reflash your firmware all the time.

The third part is the bulk of the system. It is my fluent compiler, provided for people who don't find it convenient to hand-assemble Thumb codes. I wrote a proof-of-concept in the other post, but now I have a real, non-vaporware version.

To make these concepts more concrete, here are the moving parts involved in the YouTube video.

The demo algorithm is straightforward; the thing I did which may not be obvious is to use XOR in order to identify which bits changed between iterations. This was intended to eliminate needless calls to OutputPort.Write() and hopefully make the C# code run as fast as possible.

    private static void TestCSharp(int limit) {
      var outputPorts=AllocateOutputPorts();

      for(var numberToDisplay=0; numberToDisplay<limit; ++numberToDisplay) {
        var priorNumber=numberToDisplay-1;
        var bitsThatAreDifferent=numberToDisplay^priorNumber;

        var loopMask=numberToDisplay;
        for(var bitIndex=0; bitIndex<outputPorts.Length; ++bitIndex) {
          if((bitsThatAreDifferent&1)!=0) {
            var valueToDisplay=(loopMask&1)!=0;
            outputPorts[bitIndex].Write(valueToDisplay);
          }
          bitsThatAreDifferent>>=1;
          loopMask>>=1;
        }
      }

      DeallocateOutputPorts(outputPorts);
    }

The native version looks like this:

    private static readonly short[] compiledCode=unchecked(new[] {
        (short)0xB5F0, (short)0xB086, (short)0x9005, (short)0x9104, (short)0x9E13, (short)0x9603, (short)0x9E17, (short)0x9602,
        (short)0x2600, (short)0x9601, (short)0xE027, (short)0x9801, (short)0x1E40, (short)0x9E01, (short)0x4046, (short)0x9600,
        (short)0x9901, (short)0x2200, (short)0xE019, (short)0x9E00, (short)0x2701, (short)0x1C33, (short)0x403B, (short)0x2B00,
        (short)0xD00E, (short)0x2701, (short)0x1C0B, (short)0x403B, (short)0x9E02, (short)0x6874, (short)0xB44F, (short)0x9802,
        (short)0x0080, (short)0x9E08, (short)0x5830, (short)0x9903, (short)0xF000, (short)0xF815, (short)0x1C07, (short)0xBC4F,
        (short)0x9E00, (short)0x1076, (short)0x9600, (short)0x1049, (short)0x1C52, (short)0x9F05, (short)0x42BA, (short)0xDBE2,
        (short)0x9E01, (short)0x1C76, (short)0x9601, (short)0x9E01, (short)0x9F04, (short)0x42BE, (short)0xDBD3, (short)0xB006,
        (short)0xBCF0, (short)0xBC02, (short)0x4708, (short)0x4720
      });

    private static void TestFluent(short[] code, int limit) {
      var outputPorts=AllocateOutputPorts();

      //unfortunately the fluent code cannot take an array of enums.
      //So I copy it to an array of ints
      var pinIds=new int[allPins.Length];
      for(var i=0; i<allPins.Length; ++i) {
        pinIds[i]=(int)allPins[i];
      }

      code.Invoke(i0: pinIds.Length, i1: limit, ia0: pinIds);

      DeallocateOutputPorts(outputPorts);
    }

And that's basically the meat of the program running on the YouTube video. (The entire program is available in the attachment in the solution Demo_Precompiled\CountToN\CountToN.sln)

If you happen to be conversant in the Thumb instruction set, this would take you pretty far. For everyone else, I've cooked up this freaky meta-language which I feel is the best way to embed native constructs in C# code. What follows is the source code which generated the above opcodes.

    private static short[] CompileFluent() {
      var code=CodeGenerator.Compile((g, numPins, limit, ង, ច, ឆ, ជ, ឋ, ឌ, ព, ផ, ត, ថ, pins, ធ, ម, វ) => {
        g.For(numberToDisplay => numberToDisplay.AssignFrom(0),
          numberToDisplay => numberToDisplay<limit,
          numberToDisplay => numberToDisplay.AssignFrom(numberToDisplay+1),
          numberToDisplay => {
            var priorNumber=new IntVariable("priorNumber", numberToDisplay-1);
            var bitsThatAreDifferent=new IntVariable("bitsThatAreDifferent", numberToDisplay^priorNumber);

            var loopMask=new IntVariable("loopMask", numberToDisplay);

            g.For(i => i.AssignFrom(0), i => i<numPins, i => i.AssignFrom(i+1), i => {
              g.If((bitsThatAreDifferent&1)!=0,
                () => {
                  var valueToDisplay=new IntVariable("valueToDisplay", loopMask&1);
                  g.SetPinState(pins[i], valueToDisplay);
                });
              bitsThatAreDifferent.AssignFrom(bitsThatAreDifferent.ShiftRight(1));
              loopMask.AssignFrom(loopMask.ShiftRight(1));
            });
          });
      });
      return code;
    }

If you compare it to the C# version, you can see strong similarities in structure, though the syntax is totally different. (By the way, those funky little characters (ង, ច, ឆ, ជ, etc) are my cute way of using tiny Unicode characters to denote unused arguments to the function. If you recall from the last post, every function generated in my system needs to have a fixed number of parameters; so this is a cute way of keeping the unused ones from being visually distracting).

The routine above returns a short[] array, ready to invoke immediately. (You could also print out the values in that array and embed the results in another program. That is what I did for Demo_Precompiled).

The source code for the above is in the attachment in the solution called DemoFluent\CountToN\CountToN.sln

To run any of these, you will need to reflash your tinybooter (because I built the firmware with gcc 4.4.1) and also your firmware. The files you need to do so are in the directories TinyBooter and Firmware respectively. This is a relatively straightforward process, but you should only do it if you are comfortable reflashing your Netduino. Do this at your own risk! I disclaim all responsibility!

The attached zip file has four subdirectories:

• TinyBooter - for reflashing your bootloader with SAM-BA
• Firmware - for reflashing your firmware with MFDeploy
• Demo_Precompiled - a lightweight project, which is the code behind the YouTube video
• Demo_Fluent - a copy of the above, but this time also including the big-ass compiler library

In a separate post, I'll provide the source code for the firmware changes. Learning how to rebuild the firmware from source was such a nightmare for me that I would not wish it on anyone else.

I will also try to follow up with more information, assuming anyone is interested in this crazy little project. There are lots of moving parts, so it's not possible to explain the whole thing in one post. This system is quirky and Corey-specific enough that I'm sure it looks rather baffling on first glance. If it turns out that anyone does care, one thing I can post is my test suite, which has lots of little fluent programs in order of increasing complexity. There's some good stuff in there towards the end, like a fluent version of factorial and of quicksort (recursive!!!)

ALERT Scroll down to a later message for a newer version of the library

Edited by Corey Kosak, 11 January 2011 - 03:52 AM.

Wow Corey. This is pretty impressive! I'm flying back from CES right now, but look forward to testing this on my Netduino after I get back home. Chris

Wow, Corey, that's amazing. So many potential uses!

You must have something already, but in case it somehow helps you even a little bit with making an 'offline' compiler, I wrote a simple function (wrapped in a console app for testing) that takes in a short[] and outputs text that can be easily cut and pasted into VS. It was fun for me to write, regardless.

using System;
using System.Text;

namespace ArrayOfShortsToText
{
    class Program
    {
        public static string AOStoText(short[] aos, string name = "compiledCode")
        {
            string nl = Environment.NewLine;
            string result = @"private static readonly short[] " + name + "=unchecked(new[] {" + nl;
            int count = 0;
            foreach (short i in aos)
            {
                result += @"(short)0x" + i.ToString("X4");
                count++;
                if (count < aos.Length)
                {
                    result += ", ";
                    if ((count % 5) == 0)
                    {
                        result += nl;
                    }
                }

            }

            result += nl + @"});" + nl;
            return result;
        }

        static void Main(string[] args)
        {

            short[] compiledCode = unchecked(new[] {
                (short)0xB5F0, (short)0xB086, (short)0x9005, (short)0x9104, (short)0x9E13, (short)0x9603, (short)0x9E17, (short)0x9602,
                (short)0x2600, (short)0x9601, (short)0xE027, (short)0x9801, (short)0x1E40, (short)0x9E01, (short)0x4046, (short)0x9600,
                (short)0x9901, (short)0x2200, (short)0xE019, (short)0x9E00, (short)0x2701, (short)0x1C33, (short)0x403B, (short)0x2B00,
                (short)0xD00E, (short)0x2701, (short)0x1C0B, (short)0x403B, (short)0x9E02, (short)0x6874, (short)0xB44F, (short)0x9802,
                (short)0x0080, (short)0x9E08, (short)0x5830, (short)0x9903, (short)0xF000, (short)0xF815, (short)0x1C07, (short)0xBC4F,
                (short)0x9E00, (short)0x1076, (short)0x9600, (short)0x1049, (short)0x1C52, (short)0x9F05, (short)0x42BA, (short)0xDBE2,
                (short)0x9E01, (short)0x1C76, (short)0x9601, (short)0x9E01, (short)0x9F04, (short)0x42BE, (short)0xDBD3, (short)0xB006,
                (short)0xBCF0, (short)0xBC02, (short)0x4708, (short)0x4720
              });
            Console.WriteLine();
            Console.Write(AOStoText(compiledCode, "TEST"));
        }
    }
}

Wow, Corey, that's amazing. So many potential uses!

You must have something already, but in case it somehow helps...

Thanks Bill! I do have a "ToCSharpCode()" method already in my FluentInterop\coding\MiscExtensionMethods.cs file, but I didn't realize I could use ToString("X4") and so I had needlessly written all the ToHex code myself. So your suggestion can help make this behemoth a little smaller in size. Thanks!

Damn!!! You've actually written a C# (ish) -> ARM Thump compiler??? Very impressive!

I have a few questions:

Would it possible to change the code.Invoke function to this signature:

public object Invoke(object obj, object[] parameters);

This is the real Method invoke as you know. My question is, wouldn't it be possible to give an unknown number of arguments to the invoked Thump code? This is fairly standard/easy in ANSI C/low level enviroments I think. (I'm no low level expert though.) It would make the concept look much less like a "hack". Much nicer. So nice, that it would be a potiential permanent extension for the Secret Labs fw, I think. (If you pass a wrong number of arguments or types, the program will crash and burn. This is acceptable.)

Another thing. Does your meta-language have arbitrary access to the rest of the netmf? Access to class member variables even? (Your sample code seems to do just that. But it's really rather unbeliveable.)

Also I'm not really that fond of your "meta"-language. It has a feeling of "hack" as well. I like it as an optional way to create Thump code. (And I'd might even use it at times, if it were available.) But I'd prefere something more seperate for my pro projects. Like a seperate ANSI C file. (The language wouldn't really matter. So long as it's something wellknown and existing. And it should be compiled offline.) Do you have a way (tutorial perhaps) to use an exiting compiler (your Gcc 4.4.1) for creating the Thump code? Directly useable by your Invoke function ofc?

I do realize btw, that ANSI C functions are easy obtainable through "compiling the whole TinyCLR". But your arguments still apply:

- lengthy compile-run-test cycle
- requires knowledge of C++ and toolchain
- not shareable with others unless they too are willing to reflash their firmware

I presume it would be possible to use Thump code generated by gcc, together with the RVDS fw?

Thanks everyone for the kind responses. I'll try to respond to the various issues brought up, probably in separate replies.

But first, the attached file contains "Fluent Interop 1.1", which fixes several disastrous bugs in 1.0 and also contains my "test suite". I can't guarantee that there are no more bugs, but I can guarantee that this version passes all 23 of my little testcases. If you really want to delve into my madness, the last testcase is an implementation of QuickSort, which both calls itself recursively, and also calls out to another Fluently-compiled method, to perform its array element swap operation. For amusement value, I'll do some timing comparisons on very large arrays if I get around to it.

The testsuite is available in TestSuite\TestSuite.sln

I didn't realize I could use ToString("X4") and so I had needlessly written all the ToHex code myself.

Ah, maybe the problem is with my reading comprehension. I didn't realize until now that you had written your program for the "real" .NET rather than the micro framework. Indeed, the micro framework does not support ToString("X4"). Anyway, thanks :-)

Would it possible to change the code.Invoke function to this signature:

public object Invoke(object obj, object[] parameters);

It may be possible (I would have to learn a lot more about the C#-to-native marshaling process), but it would also certainly complicate the interface to the firmware, which is very simple right now. I also wanted to make sure you were aware that, on the C# side, the sixteen arguments to my little lambda have different types (there are four ints, four byte arrays, four short arrays, and four word arrays): this means that fluent utterances like "x[4]" evaluate to different things depending on whether x is a byte, short, or int array; and won't even compile if x is an int. So my users are getting a degree of type safety that they wouldn't get in a variable number of arguments scenario.

On the invocation side, I thought I would have alleviated most complaints about long argument lists due to my use of named arguments. For example, if you wanted to pass just the first argument (i0) and the very-last argument (ia3), leaving defaults for the rest, you can simply say:

code.Invoke(i0:xxx, ia3:yyy);

which I thought was compact enough. What do you think?

Another thing. Does your meta-language have arbitrary access to the rest of the netmf? Access to class member variables even? (Your sample code seems to do just that. But it's really rather unbeliveable.)

Well, the answer is "it depends on what you mean". The compiled code is certainly not calling back to .NET, nor (in the current implementation) does it have access to anything other than the arguments passed to it. But what you may be referring to is the weaving of C# and Fluent constructs together, which provides some real power to the fluent approach (despite its "hacky" appearance which I know you dislike). For those of you who are fans of the lambda calculus, this can be thought of as a form of "currying".

Here is a simple example which illustrates what I mean:

using FluentInterop.CodeGeneration;
using FluentInterop.Fluent;
using Kosak.SimpleInterop;
using Microsoft.SPOT;

namespace Tests {
  public static class Curry {
    public delegate int AddFunc(int item);

    public static AddFunc MakeAddNFunc(int n) {
      var code=CodeGenerator.Compile((g, x, គ, ង, ច, ឆ, ជ, ឋ, ឌ, ព, ផ, ត, ថ, ទ, ធ, ម, វ) => g.Return(x+n));
      return item => code.Invoke(item);
    }

    public static void Main() {
      var f=MakeAddNFunc(12);  //line A

      Debug.Print("f(50)="+f(50)); //line B
      Debug.Print("f(10)="+f(10)); //line C
    }
  }
}

The MakeAddNFunc function is a function that builds other functions (again, a step closer to the robot holocaust).

MakeAddNFunc is a function
* which takes an integer argument N
* and which returns a function
** which takes an integer argument X
** and returns N+X

So you can think of line A as building a one-argument "Add 12" machine. If you look at the compiled code you can see that it baked the number 12 deep into the assembly (at line 0004)

      0000: B4C0  PUSH {regs=11000000}                      1011|0|10|0|11000000     (Format 14)
      0002: 1C01  ADD R1,R0,#0                              00011|1|0|000|000|001    (Format 2)
      0004: 310C  ADD R1,#12                                001|10|001|00001100      (Format 3)
      0006: 1C08  ADD R0,R1,#0                              00011|1|0|000|001|000    (Format 2)
0`top.0`regs.0`params.17`body.0`return::sourceLabel:
0`top::leave:
      0008: BCC0  POP {regs=11000000}                       1011|1|10|0|11000000     (Format 14)
      000A: 4770  BX R14                                    010001|11|0|1|110|000    (Format 5)

That's an awful lot of drama for a simple add instruction (I need to work on my register allocator), but anyway, the output of this program is of course.

f(50)=62
f(10)=22

Here I was sneaky enough to implement the inner machine in my fluent framework so that people can get a really fast add instruction. A more practical place that this might be useful is a customizer for BitBanger. One of the arguments to BitBanger is whether you want little or big endian; that code has to test that flag many times through the loop while it is running. With a fluent approach, C# can look at the bigendian flag at code generation time, and emit two different versions of the machine code depending on whether bigendian is true or false. In this approach, there is zero runtime overhead (after the compilation is done) of testing that bigendian flag. This shows that the fluent approach has the potential to beat even libraries precompiled into the firmware.

I'm not really that fond of your "meta"-language. It has a feeling of "hack" as well. I like it as an optional way to create Thump code. (And I'd might even use it at times, if it were available.) But I'd prefere something more seperate for my pro projects.

Well, you don't have a lot of choice when you're embedding one language inside another. You don't have any control over syntax for example. It's a bit of a miracle that I was able to get as far as I did.

You're certainly welcome to compile code with gcc, and figure out how to extract the opcodes from ELF or ECOFF or whatever format it is using. That should interoperate with my system just fine. This would be the reasonable approach for larger programs. I haven't had the time to figure out how to do that myself.

By the way, I keep referring to Thumb opcodes all the time because I've got Thumb on the brain. I should make it clear that our hardy little processor is quite happy to execute Thumb or ARM opcodes, so take your pick. I chose to generate thumb because they are so cute and I thought they would generally be the more dense representation of a given program.

And by the way, I still owe the forum a package of my firmware patches (what is the best way to supply these by the way?). Just to demystify what I did on the firmware side, here is the meat of my change:

#include "SimpleInterop.h"
#include "SimpleInterop_Kosak_SimpleInterop_NativeInterface.h"

using namespace Kosak::SimpleInterop;

static void *otherFunctions[]={
  (void*)&::CPU_GPIO_GetPinState,
  (void*)&::CPU_GPIO_SetPinState,
};

INT32 NativeInterface::Execute( CLR_RT_TypedArray_INT16 param0, INT32 param1, INT32 param2,
INT32 param3, INT32 param4, CLR_RT_TypedArray_UINT8 param5, CLR_RT_TypedArray_UINT8 param6,
CLR_RT_TypedArray_UINT8 param7, CLR_RT_TypedArray_UINT8 param8, CLR_RT_TypedArray_INT16 param9,
CLR_RT_TypedArray_INT16 param10, CLR_RT_TypedArray_INT16 param11, CLR_RT_TypedArray_INT16 param12,
CLR_RT_TypedArray_INT32 param13, CLR_RT_TypedArray_INT32 param14, CLR_RT_TypedArray_INT32 param15,
CLR_RT_TypedArray_INT32 param16, HRESULT &hr )
{
  typedef int (*sig_t)(int, int, int, int,
                       UINT8*, UINT8*, UINT8*, UINT8*,
                       INT16*, INT16*, INT16*, INT16*,
                       INT32*, INT32*, INT32*, INT32*,
                       void*);

  INT32 codeAddress=(INT32)param0.GetBuffer();
  sig_t fp=(sig_t)(codeAddress+1); //+1 to call the routine in thumb mode
  return (*fp)(param1, param2, param3, param4,
               param5.GetBuffer(), param6.GetBuffer(), param7.GetBuffer(), param8.GetBuffer(),
               param9.GetBuffer(), param10.GetBuffer(), param11.GetBuffer(), param12.GetBuffer(),
               param13.GetBuffer(), param14.GetBuffer(), param15.GetBuffer(), param16.GetBuffer(),
               otherFunctions);
}

And by the way, I still owe the forum a package of my firmware patches (what is the best way to supply these by the way?).

For now, I'd zip up the files that you've changed from the core firmware and attach them to the first post in the thread (along with the HEX files). Pretty soon, you should be able to create a branch of the Netduino firmware at CodePlex and integrate your changes there...which people will be easily able to download and compile.

Chris

... on the C# side, the sixteen arguments to my little lambda have different types (there are four ints, four byte arrays, four short arrays, and four word arrays): this means that fluent utterances like "x[4]" evaluate to different things depending on whether x is a byte, short, or int array; and won't even compile if x is an int. So my users are getting a degree of type safety that they wouldn't get in a variable number of arguments scenario.

Type safety is nice, I agree. But suppose I wanted to transfer 2 floats and have it return a double. Or perhaps I want to transfer my own custom structure. You cannot predict all types and combinations of input. Your function signature will always be only half way there. Generic functionality is however expected to contain all scenarios in all situations. Hence the "object[]" or the "void*". .NET 2.0 greatly improved this by implementing the Generics. But most lower functions are still working in "void*" with a Generic interface above it. Until netmf gets (if ever) Generics, "object" is the sad reality.

Well, you don't have a lot of choice when you're embedding one language inside another. You don't have any control over syntax for example. It's a bit of a miracle that I was able to get as far as I did.

It is a bit miraculus, yes. But embedding one language inside another is not a must I think. As .NET programmer we often work with many languages in each project. C#, C++, ANSI C, VB is a common mix. But they each have their own file, as a minimum. (Which is what I would also like in this case.)

You're certainly welcome to compile code with gcc, and figure out how to extract the opcodes from ELF or ECOFF or whatever format it is using. That should interoperate with my system just fine. This would be the reasonable approach for larger programs. I haven't had the time to figure out how to do that myself.

I will dig into this then. ^^ It must be possible to compile to Thump code, somehow.

You cannot predict all types and combinations of input.

I disagree with this pretty strongly. The point of this library is to do simple algorithms (and, in particular, pin I/O) as quickly as possible on the native side. Anything involving nice manipulations of C# data structures ought to stay on the .NET side where it belongs.

But suppose I wanted to transfer 2 floats and have it return a double.

Utterly pointless on this CPU, which has no hardware floating point support.

Generic functionality is however expected to contain all scenarios in all situations.

But I am not offering generic functionality. This is an extremely resource-limited device. I am offering an escape hatch from the .NET world for certain highly constrained scenarios such as fast pin I/O. Marshalling data back and forth across the boundary kills the whole point.

But embedding one language inside another is not a must I think.

I have personally enjoyed the advantages offered by dynamic compilation. Since this is open source, you are welcome to do separate compilation or even hand-assembly if that is your preference.

I will dig into this then. ^^ It must be possible to compile to Thump code, somehow.

Thumb, not thump. The gcc command is "gcc -mthumb". But as I mentioned previously, the firmware could also easily be made to call ARM code rather than thumb if that was preferred for some reason.

Fair enough

I hope my response didn't come off as grouchy. I didn't mean it that way. If I had to say it again, I would say that I've been fighting with this implementation for a couple of months and I finally reached one of my milestones, which was to generate correct code for nontrivial examples for the synchronous output case. (My next milestone is to try to figure out a sensible way to do asynchronous input code like interrupt handlers.) There are certain issues that one just doesn't appreciate until one faces them directly. One example: in your external compilation approach, how is your C code going to call an external function defined in the firmware? After all, you don't have a linker. Another example is your reference to floating point. The moment your C code does a floating point op (or even an integer divide!!!), it's going to call out to the firmware, and again, you don't have a linker. So, it really changes one's perspective once you start to work on it. So, I'm being sincere when I say you should try out your separate compilation ideas. If you can get something to actually work, everybody wins. :-)

I hadn't thought about the linking. I've seen my VisualDSP do this kind of linking. (You define where which libs are located, in a memory layout file.) It's not very realistic in this kind of solution though.

I've just realized, I know of another piece of firmware that has solved this kind of linking issue, many years ago. The Linux Kernel. When you develop linux drivers, you need access to the source in order to compile them. (Like us and the netmf) You then have the choice of linking the driver directly into the kernel and deploy the whole thing, or you can create a "module". The "module" is actually just the object file which you'd use for the linking. (It's named .ko instead of .o btw.) Modules are linked into the kernel at runtime though. And they can be unlinked as well. This greatly improves the development and deployment process. Only problem with this approach is that you'd have to use the exact same compiler as the kernel. I admit though, that I might be trying to solve something different than you. I seek an easy way to deploy arbitrary native code. This will address several issues: - The performance issue. - Easy development of native drivers. - Easy sharing of native drivers. - The growing TinyCLR. In Netduino regi this might be the most important. If we had an easy way to deploy native code to the exiting TinyCLR, we'd be able to remove drivers such as the huge ethernet, PWM, I2C, SD etc. from the main compile. This would give us much more free mem. And we'd be able to extend the toolbox with many more drivers and libs without being worried of the "size".

...modules...

Crazy idea! I love it. But it's so far out of my comfort zone that I don't think I have an intelligent opinion about it. One question: are you talking about loading the various modules into precious RAM or are you talking about dynamically flashing the Flash?

The runtime linked modules are "equal" to the rest of the kernel, so that would mean "dynamically flashing the Flash" in the Netduino case, I imagine. You'd might save a few lines of code, if you'd make "reboot" a requirement. (Reboot after flashing new modules.) Still I don't rightly know how Linux accomplish this piece of magic. One could always start reading about "insmod" though ...

This seems like a fair article on runtime linking: http://www.linuxjour...om/article/6463 One ought to be able to create an executable object file through the ARM tools. That is, an object file that has already been relocated and had its symbols redefined. (The linker might be able to create these.) Part of what you've created is akin to a standard Loader. It should be fairly easy to make it load elf instead of thumb opcodes. (I think) And if the elf was already relocated and prepared ... Only problem with this, is that you'd only be able to create 1 executable object file. (Extra object files would have wrong start addresses.)