Netduino home hardware projects downloads community

Jump to content


The Netduino forums have been replaced by new forums at community.wildernesslabs.co. This site has been preserved for archival purposes only and the ability to make new accounts or posts has been turned off.
Photo

Serial to SD -- Need Fast Writing

SD SD Card Serial Serial Port Write to SD Datalogger GPS Logger

  • Please log in to reply
8 replies to this topic

#1 Mike B

Mike B

    New Member

  • Members
  • Pip
  • 2 posts

Posted 21 May 2013 - 01:37 AM

I have been trying to find ways to write to the SD card from a serial stream as quickly as possible, but I have ran into problems and have been unable to find a solution. I have seen many posts that talk about this topic in a general way, but have seen nothing specific that has helped.

 

My code looks like this:

 

public static void Main()

{

serial = new SerialPort(SerialPorts.COM1, 115200, Parity.None, 8, StopBits.One);

serial.Open();

serial.DataReceived += new SerialDataReceivedEventHandler(serial_DataReceived);

Thread.Sleep(Timeout.Infinite);

}

 

static void serial_DataReceived(object sender, SerialDataReceivedEventArgs e)

{

do

{

readLen = serial.Read(buffer, bufferPos, ProtocolLength - bufferPos);

bufferPos += readLen;

if (bufferPos > 2000)

{

bufferPos = 0;

using (StreamWriter sw = new StreamWriter(new FileStream(@"SDtest.txt", FileMode.Append)))

{

led.Write(true);

sw.Write(new string(Toolbox.NETMF.Tools.Bytes2Chars(buffer)));

//sw.Close();

//sw.Dispose();

led.Write(false);

}

buffer = new byte[ProtocolLength];

}

}

while (readLen > 0);

}

 

Basically I am sending data to a 2k buffer that I then write to an SD card once the buffer comes close to filling up. I am receiving GPS data at intervals of 1 second and have my serial connection setup to receive as fast as I am able to. The incoming data varies between 600 bytes and around 2k bytes. What is happening is that when the program goes to write the data to the SD card data is lost is the process.

 

Any advice or suggestions would be greatly appreciated.

 

Things I have tried include looking at the SD card speed which is around 9MB/sec and removing "sw.Close()" and "sw.Dispose()" to try and gain any extra speed...I have even tried "Thread.Sleep()" mainly out of desperation. All have made no long term difference.



#2 Bi Qin

Bi Qin

    Member

  • Members
  • PipPip
  • 23 posts

Posted 21 May 2013 - 05:43 AM

Hi Mike,are you sure that data lost happened during the writing process?As the SPI bus speed is much more faster than uart,2KB/s should not be a problem.

I've met file lost issue too,but that happened when I power off the device or remove the card,and I'm still working on that.



#3 hanzibal

hanzibal

    Advanced Member

  • Members
  • PipPipPip
  • 1287 posts
  • LocationSweden

Posted 21 May 2013 - 09:11 AM

Managing FAT32 adds a lot of overhead. You could try using the SD card in raw mode without a file system. 

 

Of course, you can't use the regular NETMF functions for this, instead you have to treat the SD card like the SPI slave device that it is. You'd also have to use a special PC app to extract the data from the SD.

 

I know Nwazet has a data acquisition module called DAQ and I think they use their own FAT32 implementation for that. Perhaps you could look into the underlaying implementation of that and find some code for doing raw SD card writing:

http://nwazet.com/co...Q/10-FileSystem

 

If not, there ought to be some Arduino code for this that you could port.



#4 ziggurat29

ziggurat29

    Advanced Member

  • Members
  • PipPipPip
  • 244 posts

Posted 21 May 2013 - 12:56 PM

@mikeb

personally, in my serial handler, I do little other than consume the data and push it into a queue where other threads handle it.

 

also, I should mention that in my case I had problems with getting SerialDataReceivedEventHandler to work reliably, so I took a different approach (ditching the event handler and instead using a worker thread to monitor and read received data, but otherwise functionally similar to the event handler), but I seem to recall when I was trying to use the event handler that doing fileio there made things 'worse'.

 

Anyway, you might consider a queue to push in your received records, and a separate thread to pull them out more leisurely.

 

oh, ps, and welcome to the forums

 

@hanzibal:  thanks so much for the link; I desperately need to read/write to a raw partition; hopefully there's some scoops therein I can use, though I don't know how netmf will feel about it, or even if the spi the sd is on will be reachable....



#5 hanzibal

hanzibal

    Advanced Member

  • Members
  • PipPipPip
  • 1287 posts
  • LocationSweden

Posted 21 May 2013 - 01:39 PM

.... though I don't know how netmf will feel about it, or even if the spi the sd is on will be reachable....

 

Oops, didn't think of that, might be a problem.

 

With the mini I always use a simple SD card holder breakout on the regular SPI port.



#6 ziggurat29

ziggurat29

    Advanced Member

  • Members
  • PipPipPip
  • 244 posts

Posted 21 May 2013 - 02:19 PM

...

With the mini I always use a simple SD card holder breakout on the regular SPI port.

maybe I will have to do the same.  dataloss is a problem for me (filesystem corruption even worse) since my app, when deployed, is not particularly easy to service.  but I could easily write the code to manage a couple blocks for config, and the others for logs and data, if I did but have that access.

one fine day if I can ever get back to firmware modding I may build that in...



#7 Mike B

Mike B

    New Member

  • Members
  • Pip
  • 2 posts

Posted 22 May 2013 - 06:55 PM

Thanks all for the advice. I agree with both Bi Qin and hanzibal in that the FAT32 process should be fast enough to handle the data rate but the overhead takes some amount of time. I would generally not think that the amount of time would be so great as to cause me problems. The only thing that I have tried that seems to have helped, without going the DAQ route, is to reformat the file as a FAT32 but with an allocation of 2048. I figured that since I would not be writing more than 2048 bytes, setting the allocation size to this might streamline the system a bit better. This seems to have improved things but I am not really sure why...generally I would not think that this would have much of an effect.

 

Any thoughts???



#8 hanzibal

hanzibal

    Advanced Member

  • Members
  • PipPipPip
  • 1287 posts
  • LocationSweden

Posted 22 May 2013 - 07:32 PM

I don't recall what the standard blocksize is but if it normally is less than what you typically write, then the speed increase would make sense as you basically would only have to make one hop through the linked list or whatever FAT32 uses to keep track of the multiple parts that files consist of. If I'm not mistaken, FAT uses an initial table of contents with file names etc and a pointer to the first block of each file which in turn points to the next and so forth like a linked list. Jumping around the blocks would obviously take longer than just writing to or reading from a single byte stream. I argued that the FS itself introduces a significant overhead that raw writes wouldn't. I've seen an Arduino play a music video on a Nokia 5110 LCD and that was only made possible using raw reads from the SD and a streamlined home brew file format where bits of video and music was interleaved.



#9 ziggurat29

ziggurat29

    Advanced Member

  • Members
  • PipPipPip
  • 244 posts

Posted 23 May 2013 - 11:38 PM

truthfully, for the logging I described, I would forgo the filesystem altogether, and just have a fixed allocation of blocks, each containing a header with a generation number, and use the ancient 'ping pong' approach on the last block; i.e. using two blocks, and alternating which one for each write (incrementing the generation number).  You'll never lose more than your last record due to power outage.

 

For the databas-y stuff, the logic in, say, sqlite, already presumes a linear array of blocks for filesystem and transaction log, so it should be easy(ish!) to port to some non-file-but-nonetheless-sane-api storage.

 

As it seems to have been a trend today in my posts, I can't find time for firmware modding, but when I do I will!  At this rate, practically, it may be August2013...







Also tagged with one or more of these keywords: SD, SD Card, Serial, Serial Port, Write to SD, Datalogger, GPS Logger

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

home    hardware    projects    downloads    community    where to buy    contact Copyright © 2016 Wilderness Labs Inc.  |  Legal   |   CC BY-SA
This webpage is licensed under a Creative Commons Attribution-ShareAlike License.