ND3 SD Filestream.Read() seems to become broken - Netduino 3 - Netduino Forums
   
Netduino home hardware projects downloads community

Jump to content


The Netduino forums have been replaced by new forums at community.wildernesslabs.co. This site has been preserved for archival purposes only and the ability to make new accounts or posts has been turned off.
Photo

ND3 SD Filestream.Read() seems to become broken

SD

  • Please log in to reply
3 replies to this topic

#1 ukkiwisurfer

ukkiwisurfer

    Advanced Member

  • Members
  • PipPipPip
  • 32 posts
  • LocationLondon

Posted 08 November 2015 - 11:24 PM

Hi,

 

I'm seeing some odd behaviour sometimes when reading off an SD card. I have a dedicated thread that reads data to be sent, and reads it off the SD and into a MemoryStream, before being transmitted via an AMQP connection.

 

The MSDN documentation implies that the Read operation will leave the underlying stream at the position that the previous FileStream.Read() left it at. However I am seeing situations where it appears to be going backwards, but apparently only on certain files.

 

The FileStream has been opened only for reading, with no share allowed for any other process. Attached below is a snapshot of the logging file. The structure is:   Timestamp | Thread Id | Logical component name | Log message

 

2015-11-08 23:00:31.128 | 16 | DataTransferService | Started processing file - FileSize: '20151108225110431.data'

2015-11-08 23:00:31.185 | 16 | DataTransferService | File is within allowable size. FileSize: '554'

2015-11-08 23:00:31.274 | 16 | DataTransferService | Reading filestream. Loop: 1, BytesRead: 400, Position: 400

2015-11-08 23:00:31.358 | 16 | DataTransferService | Reading filestream. Loop: 2, BytesRead: 400, Position: 246

2015-11-08 23:00:31.440 | 16 | DataTransferService | Reading filestream. Loop: 3, BytesRead: 400, Position: 92

2015-11-08 23:00:31.526 | 16 | DataTransferService | Reading filestream. Loop: 4, BytesRead: 400, Position: 400

2015-11-08 23:00:31.582 | 16 | DataTransferService | Reading filestream. Loop: 5, BytesRead: 400, Position: 246

2015-11-08 23:00:31.667 | 16 | DataTransferService | Reading filestream. Loop: 6, BytesRead: 400, Position: 92

 

The relevant code where this is being detected is as follows (where m_BufferSize defaults to 512):

 

 ...

 byte[] buffer = new byte[m_BufferSize];

 ...

 

 using (var stream = m_FileHelper.OpenStreamForRead(m_Configuration.TargetPath, fileName, m_BufferSize))

 {

   int index = 1;

   int offset = 0;

   int bytesRead = 0;

 

   while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)

   {

     LogDebug("Reading filestream. Loop: {0}, BytesRead: {1}, Position: {2}", index, bytesRead, stream.Position);

     memoryStream.Write(buffer, 0, bytesRead);

 

     index++;

    }

  }

 

Anyone else come across this odd behaviour?



#2 ukkiwisurfer

ukkiwisurfer

    Advanced Member

  • Members
  • PipPipPip
  • 32 posts
  • LocationLondon

Posted 08 November 2015 - 11:34 PM

My only solution at the moment is to manually seek in the FileStream such that the previous code now looks like:

 

 using (var stream = m_FileHelper.OpenStreamForRead(m_Configuration.TargetPath, fileName, m_BufferSize))

 {

   int index = 1;

   int offset = 0;

   int bytesRead = 0;

 

   while (offset < fileSize)

   {

     LogDebug("Seeking position in filestream. Position: {0}", offset);

 

     stream.Seek(offset, SeekOrigin.Begin);

     bytesRead = stream.Read(buffer, 0, buffer.Length);

     offset += bytesRead;

 

     LogDebug("Reading filestream. Loop: {0}, BytesRead: {1}, Position: {2}", index, bytesRead, stream.Position);

     memoryStream.Write(buffer, 0, bytesRead);

 

     index++;

    }

 

This implies that somehow that (in the original approach) the file stream's position after a Read() is either being corrupted or altered by something else. If it is the latter, how does this correlate with a FileShare.None flag for the file stream?



#3 KiwiDev

KiwiDev

    Advanced Member

  • Members
  • PipPipPip
  • 100 posts
  • LocationNew Zealand

Posted 14 November 2015 - 05:28 AM

Hi UKKiwiSurfer

 

From the sound of it I am working on a "similar" project, I have inbound data from nRF24L01 equipped devices uploading to an Azure EventHub using AMQPNetLite, with queuing on the MicroSD?

 

My application is not very "thready" and the message payloads are small so the scope for problems is most probably less.

 

How big are your payloads?

 

try

{

   ...

 

   Message message = new Message()
   {
      BodySection = new Data()
      {
         Binary = File.ReadAllBytes(eventFile),
      },
      ApplicationProperties = new Amqp.Framing.ApplicationProperties(),
   };
 
   FileInfo fileInfo = new FileInfo(eventFile);
 
   message.ApplicationProperties["AcquiredAtUtc"] = fileInfo.CreationTimeUtc;
   message.ApplicationProperties["UploadedAtUtc"] = DateTime.UtcNow;
   message.ApplicationProperties["GatewayId"] = gatewayId;
   message.ApplicationProperties["DeviceId"] = deviceId;
   message.ApplicationProperties["EventId"] = Guid.NewGuid();
 
   sender.Send(message);
 
   File.Delete(eventFile);
 
   new Microsoft.SPOT.IO.VolumeInfo("\\sd").FlushAll();
...
}
catch (Exception ex)
{
...
 
Had you considered ReadAllBytes or similar?
 
 

@KiwiBryn

blog.devmobile.co.nz



#4 ukkiwisurfer

ukkiwisurfer

    Advanced Member

  • Members
  • PipPipPip
  • 32 posts
  • LocationLondon

Posted 14 November 2015 - 11:29 PM

Hi KiwiBryn,

 

I hadn't actually considered File.ReadAllBytes(). I'll give it a try. With regards to payload size its consistently about 556 bytes, so quite small but I get about 1 message every 5 seconds (thereabouts). 

 

It does sound very similar. The only difference is I'm publishing to a RabbitMQ instance running locally for further processing before being pushed into the cloud. I've followed your other posts with interest. :-)

 

Your insight with regards to VolumeInfo.FlushAll() will probably help mitigate the corruptions I sometimes see on the SD. ie. https://github.com/N...eter/issues/235. So thanks for this as well.

 

Thanks,

Jason.







Also tagged with one or more of these keywords: SD

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

home    hardware    projects    downloads    community    where to buy    contact Copyright © 2016 Wilderness Labs Inc.  |  Legal   |   CC BY-SA
This webpage is licensed under a Creative Commons Attribution-ShareAlike License.