12 replies to this topic

[MJ43]

Hi First post and first question if I may. Completely new to micro controllers, .net and C#, only programming experience I have is a little bit of Fortran, so as everything is a little alien to me at the moment please bear with me if the question is a trifle simple. I am hoping to use the Netduino+ as a datalogger on my motor bike, I race solo and sidecar. Aim is to hook up a GPS (I have a 10Hz unit), and monitor position, speed, suspension travel (using Sharp IR sensors), Throttle position, Exhaust Gas Temp, Fuel air ration with a Lambda sensor, pitch and roll gyros, X,Y,Z accelerometers, rpm, engine temps etc. Initial bit of read and write to the SD card sorted (amazingly easy), though I have to do a bit more testing to see what is the most efficient way of writing to the card without slowing the data logging. At the moment there is a noticeable pause whilst data is written - possibly to the amount of data and buffer size/ Anyway on to the question. At the moment I am logging data on the bench at the end of each test I stop extract the SD card, plug into the computer to see if data has written correctly. This seems a rather clumsy way of accessing what is on the memory card. Is there an easy way of accessing the SD card with it left in situ, ideally via the USB port? Thanks in advance Mark PS spent a couple of days searching forums and code stores but not found anything yet.

[Mario Vernari]

Hello Mark and welcome to join us. Your idea is really exciting. You'll have to choose the sensors carefully, but -if nothing going wrong- you may consider a small display as well. Not a complex graphic/TFT, because it requires too resources to manage it, but rather a simple char-matrix LCD, monochromatic. Quick and dirty... About the SD download via USB, I'm not sure you can do it. Of sure you may do it via Ethernet, though. If that fits your goal anyway. Hope it helps. Cheers

[mekros]

I'm helping Mark co-develop the code. We've made very slow progress. So far we've set up the codeplex site, got some temporary code up to preserve the project, and separate development. I'm hampered by the lack of hardware (starting literally with no electronics gear at home) and we're both been out of the coding game for over 10 years. It looks like I'll be angling some of my sensors onto I2C to enable to me to get N+ to read all the sensors, and I'll try to set up the webserver to allow for the file to be downloaded while on the fly.

[MJ43]

As mekros has said progress is slow. Testing interrupt speed for an RPM counter, basically call an interrupt when the ignition triggers, check the number of ticks and work out RPM. Sending tick output to debug.print I can work at 500Hz which equates to 30000 rpm which is ok in my case as I only need 28000 rpm (2 pulses per revolution so 28000 measured is actually 14000 rpm). However, if I send tick output to the SD card I am down to 80Hz which is only 4800 rpm way too slow. Is there a quick way of writing data to the SD card. Code I am using is below. Sorry if there is an easy answer C# and all this oops stuff is alien to me - I have only ever really used Fortran so on a steep learning curve. Cheers Mark /* Deal with an interrupt from the ignition * We record the time in ticks then continue * */ static void rpm_OnInterrupt(uint data1, uint data2, DateTime time) { //Get Machine time currentTime = GetCurrentTimeInTicks(); rpm.DisableInterrupt(); sw.Write(currentTime); sw.Write("\r\n"); Debug.Print(currentTime.ToString()); rpm.EnableInterrupt(); } // Get machine time static public long GetCurrentTimeInTicks() { /* Possibly 10000 ticks per millisecond. Use this to calculate timing * by calculating number of ticks between events * http://stackoverflow...oard-for-dimmin */ return Microsoft.SPOT.Hardware.Utility.GetMachineTime().Ticks; }

[NeonMika / Markus VV.]

Anyway on to the question. At the moment I am logging data on the bench at the end of each test I stop extract the SD card, plug into the computer to see if data has written correctly. This seems a rather clumsy way of accessing what is on the memory card. Is there an easy way of accessing the SD card with it left in situ, ideally via the USB port?

I have seen a project that could be of interest for you. I don't know the exact topic, but it was about simulating a keyboard (PC is the receiver and Netduino acts as a keyboard).
You could attach a little button etc. on you netduino to start the transmit. Then you read the file on the SD card and simulate key for key to send the whole file key for key to the PC, where for example word or the editor is opened.

Greets Markus

[mekros]

Thinking out aloud Mark (and hoping someone more experienced can answer ), is there a way you get setup a big buffer (like a matrix/table) and call a write to the SD once the buffer is full? It will reduce the amount of write cycles and increase speed a little.

[MJ43]

Rob I have to do some writing tests and if they don't prove successful (I suspect they will be too slow) I was thinking along the lines of using a buffer. To prevent losing valuable data I will only write to the SD card when the bike is upright and has been so for a few seconds, i.e. only on the straights. That means hopefully the buffer will be clear when braking, cornering and accelerating out of a corner the bits that are of real interest. Loss of 1/10 of a second of data in a straight line I don't see as crucial.

[mekros]

That makes sense, but you may come undone because EGT is critical towards the latter half of a straight and you don't want to lose data there. You'll also have to make the buffer big enough for the biggest sweep bend that would be encountered.

[dougY]

have you tried creating an array of data points?

the way i have been handling this sort of thing is to write to an array or a queue.

then you either have another thread sleeping, waking up, and doing the writes to the SD
or the other option is to test the length as you write to it and if the queue length hits a certain limit you do the write.

so you end up doing this:

1. the interrupt fires calling the sample logger function which writes the data to the queue or array
2. the write to disk thread is spinning in the background. every few seconds or 300ms it wakes up, checks the length of the queue and if its long enough it writes to disk and removes the items. you will need to vary your thread.sleep(xxx) calls accordingly. you will run out of memory quick if you are storing a data point with several parameters.

here is a snippet of code that i have been using to store the data in xml.
i created a class(deviceResult) that can return its data as an xml snippet (string) via the call to generateXMLnode()
that function takes the current time as a parameter cause it costs less memory to store the ticks as ints than it does to store the time in every result i enqueue. a quick calculation was worth it as a trade against speed cause i was running into memory limit problems doing the other way by storing the dateTime which is more bytes.
for the same reason it takes the GUID its assigned so that isnt being stored in every object in the queue.

in my case i am not making as many samples as you seem to be so i am getting away with generating the data to write within the time the fileStream is open. you may want to do that before entering the "using(FileStream..." section of code to increase throughput.

you may also want to play around with the last parameter of the new FileStream call cause that's your buffer size.
it may affect your write speed. i chose 128 cause i care more about using less memory than i do about speed since i am not sampling as often as you seem to be.

deviceResult dev = null;
String       res = "";
using (FileStream fs = new FileStream(resultsFileName, FileMode.Append, FileAccess.Write, FileShare.None, 128 ))
{    
    while ((dev = (deviceResult) resultsQueue.Dequeue()) != null)
    {
        res = dev.generateXMLnode( deviceGUID, startUpDateTime );

        byte[] xmlBytes = Encoding.UTF8.GetBytes( res );

        fs.Write( xmlBytes, 0, (int) xmlBytes.Length );
    }
    fs.Close();
}

[dougY]

i think someone mentioned this above:
have the rate of data logging controlled by either the lean of the bike or by the change in direction. it seems you care more about the telemetry data surrounding a turn than you do in the straight away.

so maybe the GPS or a gyro sensor could give feedback to a monitoring thread and that thread could in turn modify a variable that the logging thread uses to decide how fast to "fill" the results queue.
i wouldnt go for an interrupt approach on that.
i would use polling from that thread.
something like the following where "settings" is the reference to the global object holding the sampling parameters and is passed in when you create the object which has the method executed in your ThreadStart() call from main()

//Program.cs code
 public static void Main()
 {
      settings = new appSettings();  //initialize from SD card

      Debug.Print( "memory before obj instantiation: " + Debug.GC( false ) );

      readSensor   sen = new readSensor( settings );
      checkForLean chk = new checkForLean( settings );
      logToSD      log = new logToSD( settings );

      Debug.Print( "memory after obj instantiation: " + Debug.GC( false ) );


      Thread senThread = new Thread( new ThreadStart( sen.Run ) );
      Thread chkThread = new Thread( new ThreadStart( chk.Run ) );
      Thread logThread = new Thread( new ThreadStart( log.Run ) );  // calling it Run like its JAVA ;)

      Debug.Print( "memory after threads created: " + Debug.GC( false ) );

      senThread.Start();
      chkThread.Start();
      logThread.Start();
  
      Debug.Print( "memory after threads started: " + Debug.GC( false ) );

      //sen.Join(); // changed this recently to poll for crashed threads
      while (true)
    {
        if (!senThread.IsAlive)          
        {
            senThread = new Thread( new ThreadStart( sen.Run ) );
            senThread.Start();  
            //Thread.Sleep( 12000 );  //i needed this, you probably dont
        }
        if (!chkThread.IsAlive)   
        {
            chkThread = new Thread( new ThreadStart( chk.Run ) );
            chkThread.Start(); 
            //Thread.Sleep( 12000 );
        }
        if (!logThread.IsAlive) 
        {
            logThread = new Thread( new ThreadStart( log.Run ) );
            logThread.Start(); 
        }
        Thread.Sleep(1000 * 180);  // once every 3 minutes, we check if threads are still alive and restart them if necessary
    }
}

so then you need to create your readSensor, checkForLean and logToSD classes and give them a constructor that takes an appSettings object and a method called Run() that is void Run(void) in signature

// check For Lean
void Run()
{
float currentDirection = -1;
float tempDirection    = -1;
while(true)
{
    tempDirection = miracleOfGPS_unit.getDirection();
    if (currentDirection varies way more than tempDirection)
    {
         settings.sampleRate = 20;
    }
    if (currentDirection varies a little more than tempDirection)
    {
         settings.sampleRate = 10;
    }
    if (currentDirection varies not much compared to tempDirection)
    {
         settings.sampleRate = 4;
    }
    // the above section can probably have more math and whatnot if you are so inclined to 
    // calculate the derivatives of the change.  after all you could put the sample queue in the 
    // appSettings object and have access to it here.  just remember you are multi-threading
    //  and you may need to be thread safe  or at least think about what can happen if its being modded as you read it

    currentDirection = tempDirection;
    Thread.Sleep(50);  // gonna have to play around here and see what works 50 may be too fast or slow
                       // should also be stored in settings like settings.checkForLeanSampleRate and loaded from SD card
                       //  so in the field you can just edit the settings.xml file on the SD card and not worry
                       // about a recompile
}
}

 //sample thread
while(true)
{
    // check samples and log to queue or array then sleep

    Thread.Sleep(1 / settings.sampleRate * 1000);  //settings.sampleRate is in samples per second
}

guess if you still need the interrupt code you could check the time elapsed against 1 / settings.sampleRate * 1000 * 10000 (or however you get from samples a second to ticks) and only log if enough time has passed.

yet another idea and should probably be classified as a "duct tape" solution:
if your runs on the track are close enough (by a few seconds) you may be able to more simply teach the course to the code.
for instance.

1. for 15 secs in straightaway so dont log.
2. for 8 secs in chicane - log heavily to memory
3 ....

success of that depends on the complexity of the course layout and the variance of your lap performances. i dont race so i have no idea if thats impossible.

i guess it was watching that Ayrton Senna documentary last night that is inspiring me to post on your thread. i agree with the other posters that its an awesome project.

[bobk]

you can use two net talking i2c. one can listen and write to SD card, one can be the master reading wensors and sending data to log.

[miker00lz]

You might want to consider incorporating a very simple web server into your Netduino project for accessing the SD card data. It is obscenely easy to do.

[mekros]

I'm not a fan of using 2xN+, I'd like to keep the costs down and apply the KISS principal as much as possible. (Not discarding the thought, just shelving it for now)

I like what @dougY has proposed, I'll try and implement that in a future code iteration.

To keep everyone out of suspenders, here is where we are keeping our code:

Codeplex Datalogger project

I've got a really old version of code there for MJ43, I just put my code up based on using the I2C implementation and forgot to add the SD card writing to it, so that'll get added in the next iteration.

I'm glad to see that this has got the thoughts and intrigue from other coders