How to use the SD card and StreamWriter

I'm trying to get the SD card working but get an error when closing the StreamWriter.

Here is the code:

var fs = new FileStream(fileName, FileMode.Append, FileAccess.Write, FileShare.None, 512);
var sw = new StreamWriter(fs);
sw.WriteLine("Hello Netduino");
sw.Close();

and this is the error:

    #### Exception System.IO.IOException - CLR_E_FILE_IO (4) ####
    #### Message: 
    #### Microsoft.SPOT.IO.NativeFileStream::Write [IP: 0000] ####
    #### System.IO.FileStream::Write [IP: 002a] ####
    #### System.IO.StreamWriter::Flush [IP: 0021] ####
    #### System.IO.StreamWriter::Dispose [IP: 0016] ####
    #### System.IO.TextWriter::Dispose [IP: 0005] ####
    #### System.IO.StreamWriter::Close [IP: 0004] ####
    #### RoomMonitor.Application::DataLogUpdate [IP: 0090] ####
A first chance exception of type 'System.IO.IOException' occurred in Microsoft.SPOT.IO.dll
    #### Exception System.IO.IOException - 0x00000000 (4) ####
    #### Message: StreamWriter Flush. 
    #### System.IO.StreamWriter::Flush [IP: 002b] ####
    #### System.IO.StreamWriter::Dispose [IP: 0016] ####
    #### System.IO.TextWriter::Dispose [IP: 0005] ####
    #### System.IO.StreamWriter::Close [IP: 0004] ####
    #### RoomMonitor.Application::DataLogUpdate [IP: 0090] ####
A first chance exception of type 'System.IO.IOException' occurred in System.IO.dll

Szymon, does this also happen on a regular Netduino with the v4.1.1 alpha firmware? I know that at least one user reported an issue with the order in which objects were closed/disposed. Also, what happens if you don't do Append--but do open or create instead? Chris

Szymon, does this also happen on a regular Netduino with the v4.1.1 alpha firmware? I know that at least one user reported an issue with the order in which objects were closed/disposed.

Also, what happens if you don't do Append--but do open or create instead?

Chris

Chris,
I tried to close FileStream before StreamWriter and it didn't crash, but nothing was written to the file.
So then I tried to call Flush before closing and this crashes with similar exception.

I also changed FileMode to Open and OpenAndCrate and both don't work too.

But this is part of larger project so maybe there is some other dependency. Tomorrow I will try to test it with simple program.

Chris,
Today I tried to run a very simple example. And following code works fine if I put it at the start of the program:

using (var fs = new FileStream(fileName, FileMode.OpenOrCreate, FileAccess.Write, FileShare.None, 512))
using (var sw = new StreamWriter(fs))
{
    sw.WriteLine("hello from netduino");
    sw.WriteLine("done.");
}

However if I try to iterate all files and display their content I immediately get an OutOfMemoryException. I use the same pattern to dispose FileStream and StreamReader in each iteration but this doesn't help. I only have 5 files with only few bytes of content so I think this is behavior is strange.

string[] files = Directory.GetFiles(@"\SD");
for (int i = 0; i < files.Length; i++)
{
    Debug.Print("filename: " + files[i]);

    using (var fs = new FileStream(files[i], FileMode.Open, FileAccess.Read, FileShare.None, 512))
    using (var sr = new StreamReader(fs))
    {
        Debug.Print("contents: ");
        Debug.Print(sr.ReadLine());
    }
}

Here is the exception:

Failed allocation for 685 blocks, 8220 bytes

Failed allocation for 685 blocks, 8220 bytes

    #### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (1) ####
    #### Message: 
    #### System.IO.StreamReader::ReadLine [IP: 000a] ####
    #### FileSystemTest.Program::Main [IP: 012b] ####
A first chance exception of type 'System.OutOfMemoryException' occurred in System.IO.dll
An unhandled exception of type 'System.OutOfMemoryException' occurred in System.IO.dll

Maybe the FileStream buffers dont get released properly ?

Can anyone else try to run this code and see if it works fine on their Netduino Plus?

I'm trying to get the SD card working but get an error when closing the StreamWriter.

Szymon,

I just ran your code from the first post on my Netduino Plus and I can't get it to give me any errors. Could you please attach your project as a ZIP file? [If you could attach a few ZIP files--one per project where you're having troubles with file access--that would be really great.]

We're working on a firmware update and I'd like to try to address this if possible.

Thanks, Szymon.

Chris

P.S. I am able to reproduce the behavior from your second post. Interesting. It does appear like buffers aren't getting cleaned up properly. We'll dig into the .NET MF SD SPI and System.IO code and see what's going on... It appears to be the ReadLine() function that's causing the issue, interestingly enough... Take that out, and all is well. Curious. Chris

P.S.

I am able to reproduce the behavior from your second post. Interesting. It does appear like buffers aren't getting cleaned up properly. We'll dig into the .NET MF SD SPI and System.IO code and see what's going on...

It appears to be the ReadLine() function that's causing the issue, interestingly enough... Take that out, and all is well. Curious.

Chris

Chris,
Thanks for looking into this. If you change to use ReadToEnd() the behavior is the same. So this might be memory leak inside the StreamReader itself.

Here's an odd thing. I was unable to write a file to the MicroSD card using FileStream/Stream/etc. I decided to check the card by writing a file to it on the PC. That worked fine, so I stuck the card back into the netduino to see if it could read the directory. It read the directory just fine and was able to write files without any issues from then on! I'll keep playing with the file system and reporting my results. Stacy

Chris,
Thanks for looking into this. If you change to use ReadToEnd() the behavior is the same. So this might be memory leak inside the StreamReader itself.

I'm getting exactly the same error as Szymon when I try to read a file from the SD card. I actually wrote my sample before I saw this thread, but my code is almost the same.

Here's the sample code where I use a StreamReader object to get the first line of a text file:

using System;
using System.IO;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;
using SecretLabs.NETMF.Hardware;
using SecretLabs.NETMF.Hardware.Netduino;

namespace SDCardSample
{
    public class Program
    {
        public static void Main()
        {
            string currentDir = Directory.GetCurrentDirectory();
            Debug.Print("Current directory: " + currentDir);

            string[] dirs = Directory.GetDirectories(currentDir);
            foreach (string d in dirs)
            {
                Debug.Print("dir: " + d);
                string[] files = Directory.GetFiles(d);
                foreach (string f in files)
                {
                    Debug.Print("file: " + f);
                    StreamReader reader = new StreamReader(f);
                    Debug.Print(reader.ReadLine());
                    reader.Close();
                }
            }
        }
    }
}

And the exception that gets thrown:

Current directory: \
dir: \SD
file: \SD\FOO.TXT
Failed allocation for 685 blocks, 8220 bytes

Failed allocation for 685 blocks, 8220 bytes

    #### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (1) ####
    #### Message: 
    #### System.IO.StreamReader::ReadLine [IP: 000a] ####
    #### SDCardSample.Program::Main [IP: 005e] ####
A first chance exception of type 'System.OutOfMemoryException' occurred in System.IO.dll
An unhandled exception of type 'System.OutOfMemoryException' occurred in System.IO.dll

Note that this happens for both ANSI and UTF-8 encoded files (I created the files on the PC).

We've fixed the "trying to mount an SD card when none is present locks up .NET MF" issue on the Netduino v4.1.1 firmware...and are now working on this issue... Chris

On the memory issues: 1. We haven't found any memory leaks...this is good. 2. The FAT file system was creating up to about 8 2KB caches when data was read. We're tuning this down quite a bit. The memory was reclaimed when the SD card was unmounted--but a total cache size of 2KB is probably more appropriate for this type of device. 3. The StreamReader uses 4KB buffers by default; we're tuning these down to 512 bytes. ReadToEnd() and ReadLine() read data in chunks of 4KB -- and had to create new buffers of at least 8KB whenever they needed to read more data. We'll post a beta update for the Netduino Plus firmware (v4.1.0.4 beta 1) with these changes by the end of the week. Chris

On the memory issues:

1. We haven't found any memory leaks...this is good.
2. The FAT file system was creating up to about 8 2KB caches when data was read. We're tuning this down quite a bit. The memory was reclaimed when the SD card was unmounted--but a total cache size of 2KB is probably more appropriate for this type of device.
3. The StreamReader uses 4KB buffers by default; we're tuning these down to 512 bytes. ReadToEnd() and ReadLine() read data in chunks of 4KB -- and had to create new buffers of at least 8KB whenever they needed to read more data.

We'll post a beta update for the Netduino Plus firmware (v4.1.0.4 beta 1) with these changes by the end of the week.

Chris

Chris,
This is great news. Thanks for fixing this so quickly.

You guys are great... something is reported and it's fixed in a week or so.

Quick question about the SD card reader, would you be able to write a code to execute a larger one stored on the SD card?

Quick question about the SD card reader, would you be able to write a code to execute a larger one stored on the SD card?

You should be able to use AppDomains to load assemblies dynamically off of the MicroSD card. We haven't tested this on the Netduino Plus...but we've done it on lab hardware before.

Chris

P.S. Regarding the "failed allocation" message... That is actually a Garbage Collector debug message which tells you that it couldn't find enough contiguous memory--and so it's going to compact memory and try again. If memory allocation actually fails, you'll get an OutOfMemoryException. To make this clearer, we've changed the "failed allocation" message so that it now says something along the lines of "could not allocate...compacting". And by default, this GC message won't appear unless you enable GC messages generally. I just ran a test app on a Netduino with the updated firmware and: 1. If you try to mount an SD card when none exists, you will now get an exception instead of locking up .NET MF. 2. Memory usage is much, much, much smaller. 2KB cache for the FAT file system and <=1KB buffer for FileStreams. 3. Garbage collection messages are now clearer (and no longer appear in the Output window when GC messages are turned off) We'll post the 4.1.0.4 beta within a day or two (for Netduino and Netduino Plus). We will also post an updated v4.1.1 alpha (for Netduino) by sometime next week. Chris

On the memory issues:

1. We haven't found any memory leaks...this is good.
2. The FAT file system was creating up to about 8 2KB caches when data was read. We're tuning this down quite a bit. The memory was reclaimed when the SD card was unmounted--but a total cache size of 2KB is probably more appropriate for this type of device.
3. The StreamReader uses 4KB buffers by default; we're tuning these down to 512 bytes. ReadToEnd() and ReadLine() read data in chunks of 4KB -- and had to create new buffers of at least 8KB whenever they needed to read more data.

We'll post a beta update for the Netduino Plus firmware (v4.1.0.4 beta 1) with these changes by the end of the week.

Chris

If we get SDHC support the buffers may need to be at least 1024 since that is the sector size of the HC device.

We have posted updated firmware to address the SD card mounting and memory allocation issues. http://forums.netdui...patch-4-beta-1/ SD card-related updates in the firmware update: * Mounting invalid MicroSD cards now throws an exception * Reduced memory usage when accessing SD cards Please let me know if this new firmware fixes things for you... Chris

We have posted updated firmware to address the SD card mounting and memory allocation issues.
http://forums.netdui...patch-4-beta-1/

SD card-related updates in the firmware update:
* Mounting invalid MicroSD cards now throws an exception
* Reduced memory usage when accessing SD cards

Please let me know if this new firmware fixes things for you...

Chris

Cool - thanks for getting the new firmware out so quickly!

This seems to fix the problem I was having with the StreamReader.ReadLine() method yesterday:

Current directory: \
dir: \SD
file: \SD\FOO.TXT
The quick brown fox jumps over the lazy dog.

Now I need to figure out what to do with 1.83 GB of storage.

Hello,

I am running dab's code to list all directories and files. I get following debug output:

Current directory: \
dir: \SD
#### Exception System.NotSupportedException - CLR_E_NOT_SUPPORTED (4) ####
#### Message:
#### Microsoft.SPOT.IO.NativeFindFile::.ctor [IP: 0000] ####
#### System.IO.Directory::GetChildren [IP: 0054] ####
#### System.IO.Directory::GetFiles [IP: 0008] ####
#### NetduinoPlusListener01.LogWriter::PrintDirectories [IP: 0034] ####
#### NetduinoPlusListener01.LogWriter::RemovableMedia_Insert [IP: 0004] ####
#### Microsoft.SPOT.IO.RemovableMedia::MessageHandler [IP: 0041] ####
A first chance exception of type 'System.NotSupportedException' occurred in Microsoft.SPOT.IO.dll
An unhandled exception of type 'System.NotSupportedException' occurred in Microsoft.SPOT.IO.dll

As you can see, an exception is thrown when invoking Directory.GetFiles() method.
What does 'System.NotSupportedException' mean? Is my SD card not compatible or it is something else?
I have latest version of N+ firmware.

Regards,
Myszor