Netduino home hardware projects downloads community

Jump to content


The Netduino forums have been replaced by new forums at community.wildernesslabs.co. This site has been preserved for archival purposes only and the ability to make new accounts or posts has been turned off.
Photo

Multithreading and writing to SD card

multithreading SD datalogging

  • Please log in to reply
6 replies to this topic

#1 Verdris

Verdris

    Advanced Member

  • Members
  • PipPipPip
  • 128 posts
  • LocationReno, NV

Posted 19 January 2013 - 12:12 AM

I have a project which is currently forced to use two separate microcontrollers, one to log GPS data and the other to govern an array of sensors. They're separate since the GPS logging comes in so fast that there's just too much data, while the sensors only need log every two seconds. Since I can't time things precisely, I've decided to go for the multithreading Netduino solution.

 

So, is it possible for two distinct threads to log their data to the same SD card simultaneously? Not to the same file, obviously.

 

The hardware that each thread is using will be distinct, one uses a COM port to chat with the GPS receiver, and the other used SPI and I2C to run the sensors. No overlap there, but I'm concerned about the datalogging.



#2 AxelG

AxelG

    Advanced Member

  • Members
  • PipPipPip
  • 52 posts

Posted 19 January 2013 - 01:16 AM

I have written a couple applications that have multiple threads accessing the SD card at the same time; it works but you need to be careful with a couple things.  You hit on the first one; one thread per file (reading and writing).  If you need two threads, simple locking mechanism can ensure read/seek/write transactions remain pure.  Since each thread keeps their own file structures (Assuming you are not sharing them!); mostly you are OK.

 

I have also found that dealing with the SD card is quite resource intensive; especially when mixed with time-sensitive sensor reading.  It is easy to create bottlenecks in the code that create data flow problems. Multi-threading on the netduino is not parallel processing; it just allows you to branch execution into separate places in your code - processor time is shared between the threads so only one instruction is running at a time.  The right locking scheme can make all of the difference.

 

SD cards also use SPI.  This is another area to make sure you code is aware there are multiple SPI devices.



#3 Verdris

Verdris

    Advanced Member

  • Members
  • PipPipPip
  • 128 posts
  • LocationReno, NV

Posted 21 January 2013 - 03:13 AM

I have written a couple applications that have multiple threads accessing the SD card at the same time; it works but you need to be careful with a couple things.  You hit on the first one; one thread per file (reading and writing).  If you need two threads, simple locking mechanism can ensure read/seek/write transactions remain pure.  Since each thread keeps their own file structures (Assuming you are not sharing them!); mostly you are OK.

 

I have also found that dealing with the SD card is quite resource intensive; especially when mixed with time-sensitive sensor reading.  It is easy to create bottlenecks in the code that create data flow problems. Multi-threading on the netduino is not parallel processing; it just allows you to branch execution into separate places in your code - processor time is shared between the threads so only one instruction is running at a time.  The right locking scheme can make all of the difference.

 

SD cards also use SPI.  This is another area to make sure you code is aware there are multiple SPI devices.

Thanks for this reply. Could you explain what is meant by a "locking scheme"?



#4 ericcox

ericcox

    Member

  • Members
  • PipPip
  • 19 posts

Posted 22 January 2013 - 08:14 AM

Here's how I usually implement this problem on the full version of .NET on the PC: use one thread for the GPS data, one for the sensor data, and a third thread to send the data to the SD card.  You could use a Queue object to manage the data flow.  Your worker threads would stuff the data into the queue, using a locking object to eliminate collisions. I haven't used this kind of thing on the NetMF however - there may be threading idiosyncrasies on NetMF for which I haven't accounted.

 

 

object lockObject = new Object();Queue queue = new queue();AutoResetEvent dataIsAvailable = new AutoResetEvent(false);public static void Main(){  ... Create Logging Thread   (Make sure to start this one first)  ... Create GPS Thread  ... Create Sensor Thread  Thread.Sleep(Timeout.Infinite);}public void GPSThreadStart(){   while (true)   {     .... raw GPS data received     ... create data object from raw GPS data     	 lock (lockObject)     {         queue.Enqueue(data);     }	 dataIsAvailable.Set();   }}(Sensor thread is the same basic idea)public void LoggingThreadStart(){   while ( true )   {	  dataIsAvailable.WaitOne();	  while (queue.Count > 0 ) 	  {		 object data;         lock (lockObject)         {           data = queue.DeQueue();         }		 ... Write data to SD card	  }   }}

 

The AutoResetEvent allows the threads to wait until data is available without using any CPU cycles.  The lock() statements make sure the Dequeue/Enqueue calls are completed before a thread context switch is allowed, avoiding collisions.  Using this scheme, you can enqueue sensor data inside interrupt handlers, and even write the logging data to the same file if you like.  When I do that, I usually use a base class to enforce a common record format, then derived classes for each different type of record.

 

Hope that helps,

Eric



#5 NeonMika / Markus VV.

NeonMika / Markus VV.

    Advanced Member

  • Members
  • PipPipPip
  • 209 posts
  • LocationUpper Austria

Posted 22 January 2013 - 10:24 AM

Pretty nice piece of code Eric, I think I will give this a try sometime.

Do you know  if the AutoResetEvent is in the .Net MF too? This multi-threading-mechanism is pretty cool.

 

Greets, Markus


NeonMika.Webserver
> Control your N+ and write webservice methods easyily
> Receive data from you N+ (in XML or JSON)
> Browse the SD on your N+ directly in the browser and d
own - and upload files

 

If you need help with NeonMika.Webserver, please just leave a note in the thread and/or contact me via Skype :)

 

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
--- Mistakes teach you important lessons. Every time you make one, you are one step closer to your goal. ----
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------


#6 ericcox

ericcox

    Member

  • Members
  • PipPip
  • 19 posts

Posted 22 January 2013 - 07:27 PM

Pretty nice piece of code Eric, I think I will give this a try sometime.

Do you know  if the AutoResetEvent is in the .Net MF too? This multi-threading-mechanism is pretty cool.

 

Greets, Markus

 

Thanks Markus.

 

Yep, AutoResetEvent and ManuResetEvent are in the NetMF.  IIRC, in the full framework it's a single class called EventWaitHandle with the ability to set auto or manual at will.

 

Couple more things I thought of:

  1. try to keep the code within your lock statements as small as possible.  Rule of thumb in the full framework is 5 IL instructions, that's usually 1 or 2 lines of code.

  2. If your system is designed to run forever, then this code should work, but if you build in the concept of a shutdown/restart or start/stop kind of thing, then you will have to manage the shutdown properly to avoid losing any data. Basically you send a signal to all threads to shutdown, then kick the dataIsAvailable handle (tricking the logger thread into emptying the queue), Then you have to wait for the logger thread to send a signal that it is done before shutting down.  I usually just use "volatile bool" variables for all these signals.  Of course, if a few lost sensor readings or GPS don't matter, then the point is moot  :D )



#7 AxelG

AxelG

    Advanced Member

  • Members
  • PipPipPip
  • 52 posts

Posted 23 January 2013 - 06:40 AM

Thanks for this reply. Could you explain what is meant by a "locking scheme"?

Sure:

In my code I open the Filestream once and keep it open.  It would be a bad thing if multiple threads interacted with the the stream at "the same time".  I therefore create an object to lock it (private object LockMe = new object();) and any time I am reading or writing to the filestream I make sure the code holds a lock on this object (lock(LockMe) { //Code here } )

 

This way, you can ensure only one thread is interacting with the filestream at any given time.  Just make sure the scope of LockMe is shared between all threads.

 

I hope this helps.  You can google "C# lock()"  for more reference.







Also tagged with one or more of these keywords: multithreading, SD, datalogging

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

home    hardware    projects    downloads    community    where to buy    contact Copyright © 2016 Wilderness Labs Inc.  |  Legal   |   CC BY-SA
This webpage is licensed under a Creative Commons Attribution-ShareAlike License.