ND3 SD Filestream.Read() seems to become broken

Hi,

I'm seeing some odd behaviour sometimes when reading off an SD card. I have a dedicated thread that reads data to be sent, and reads it off the SD and into a MemoryStream, before being transmitted via an AMQP connection.

The MSDN documentation implies that the Read operation will leave the underlying stream at the position that the previous FileStream.Read() left it at. However I am seeing situations where it appears to be going backwards, but apparently only on certain files.

The FileStream has been opened only for reading, with no share allowed for any other process. Attached below is a snapshot of the logging file. The structure is:   Timestamp | Thread Id | Logical component name | Log message

2015-11-08 23:00:31.128 | 16 | DataTransferService | Started processing file - FileSize: '20151108225110431.data'

2015-11-08 23:00:31.185 | 16 | DataTransferService | File is within allowable size. FileSize: '554'

2015-11-08 23:00:31.274 | 16 | DataTransferService | Reading filestream. Loop: 1, BytesRead: 400, Position: 400

2015-11-08 23:00:31.358 | 16 | DataTransferService | Reading filestream. Loop: 2, BytesRead: 400, Position: 246

2015-11-08 23:00:31.440 | 16 | DataTransferService | Reading filestream. Loop: 3, BytesRead: 400, Position: 92

2015-11-08 23:00:31.526 | 16 | DataTransferService | Reading filestream. Loop: 4, BytesRead: 400, Position: 400

2015-11-08 23:00:31.582 | 16 | DataTransferService | Reading filestream. Loop: 5, BytesRead: 400, Position: 246

2015-11-08 23:00:31.667 | 16 | DataTransferService | Reading filestream. Loop: 6, BytesRead: 400, Position: 92

The relevant code where this is being detected is as follows (where m_BufferSize defaults to 512):

 ...

 byte[] buffer = new byte[m_BufferSize];

 ...

 using (var stream = m_FileHelper.OpenStreamForRead(m_Configuration.TargetPath, fileName, m_BufferSize))

 {

   int index = 1;

   int offset = 0;

   int bytesRead = 0;

   while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)

   {

     LogDebug("Reading filestream. Loop: {0}, BytesRead: {1}, Position: {2}", index, bytesRead, stream.Position);

     memoryStream.Write(buffer, 0, bytesRead);

     index++;

    }

  }

Anyone else come across this odd behaviour?

My only solution at the moment is to manually seek in the FileStream such that the previous code now looks like:

 using (var stream = m_FileHelper.OpenStreamForRead(m_Configuration.TargetPath, fileName, m_BufferSize))

 {

   int index = 1;

   int offset = 0;

   int bytesRead = 0;

   while (offset < fileSize)

   {

     LogDebug("Seeking position in filestream. Position: {0}", offset);

     stream.Seek(offset, SeekOrigin.Begin);

     bytesRead = stream.Read(buffer, 0, buffer.Length);

     offset += bytesRead;

     LogDebug("Reading filestream. Loop: {0}, BytesRead: {1}, Position: {2}", index, bytesRead, stream.Position);

     memoryStream.Write(buffer, 0, bytesRead);

     index++;

    }

This implies that somehow that (in the original approach) the file stream's position after a Read() is either being corrupted or altered by something else. If it is the latter, how does this correlate with a FileShare.None flag for the file stream?

Hi UKKiwiSurfer

From the sound of it I am working on a "similar" project, I have inbound data from nRF24L01 equipped devices uploading to an Azure EventHub using AMQPNetLite, with queuing on the MicroSD?

My application is not very "thready" and the message payloads are small so the scope for problems is most probably less.

How big are your payloads?

try

{

   ...

@KiwiBryn

blog.devmobile.co.nz

Hi KiwiBryn,

I hadn't actually considered File.ReadAllBytes(). I'll give it a try. With regards to payload size its consistently about 556 bytes, so quite small but I get about 1 message every 5 seconds (thereabouts). 

It does sound very similar. The only difference is I'm publishing to a RabbitMQ instance running locally for further processing before being pushed into the cloud. I've followed your other posts with interest. :-)

Your insight with regards to VolumeInfo.FlushAll() will probably help mitigate the corruptions I sometimes see on the SD. ie. https://github.com/N...eter/issues/235. So thanks for this as well.

Thanks,

Jason.