7 replies to this topic

[Novice Alex]

Hi Guys, Currently I have developed a near to completed application on the N+ using Netduino firmware (to get the extra RAM). But only today, I realise N+ has a problem reading text file (with around 500 records) of size around 12Kb. I need to read back for historical reporting on daily basis. I am using the streamreader.readtoend method, but it will throw SystemOutofMemory. I also tried using the pre-define size buffer(1024) to read block by block, but still meet the same exception. I also tried using the readline method together with stringbuilder library to join each readline returns. But still hit the same exception. Then I finally give up. I have to reduce my sampling interval to ensure the daily file size is around 6K instead, then everything goes well. Just like to check anyone out there has a better idea how to handle large text file data log files? Thanks in advance. Regards, Novice Alex

[Stefan]

The file size shouldn't be a problem, I've read files of a few MBs. But you shouldn't try to load the -full- file in memory, you'll never succeed. I normally read a file until EOL (or any other end of record marker), parse it, continue reading.

[Novice Alex]

Hi Stefan, If I don't do the readtoend method, the process will be very slow and it will cause exception from my interrupt (8 pulse inputs). Do you have any codes on how to do file seek? Maybe, I can use your method, but need to call the read file many times using the seek method to remember where I left off on the previous call.

[Stefan]

If I don't do the readtoend method, the process will be very slow and it will cause exception from my interrupt (8 pulse inputs).

Do you have any codes on how to do file seek? Maybe, I can use your method, but need to call the read file many times using the seek method to remember where I left off on the previous call.

It all depends on the file format. Since you're writing daily stats? You could do several things:
- Make records an X amount of bytes large
- Create a file format that works with data blocks
- Create an index

I have too little information right now to really be able to help you out.

[Nevyn]

Just like to check anyone out there has a better idea how to handle large text file data log files?

What are you doing with the file once you have read it?

Regards,
Mark

[Novice Alex]

What are you doing with the file once you have read it?

Regards,
Mark

Hi Mark,

These data files (daily) are records of the engine consumption that sample every 1 mins.
Currently the record format is as follows: [mins since today's 00:00] | [Double variable with 3 decimal places]

Regards,
Alex

[Novice Alex]

It all depends on the file format. Since you're writing daily stats? You could do several things:
- Make records an X amount of bytes large
- Create a file format that works with data blocks
- Create an index

I have too little information right now to really be able to help you out.

Hi Stefan,

That's sound like a good idea. Since each records is fit by the format [mins since 00:00 of the day] | [Double value with 3 decimal place]. I can fix string format for both. And use ReadBlock to do it. However, need to do some testing to see whether it improve in processing.

in fact, currently when I read back the full content, I will then pass the full file content (as string) back to the caller for further parsing and then determine the require range (report time start to end) and accumulate the total value in the field (2) - Double.

Regards,
Alex

[Stefan]

in fact, currently when I read back the full content, I will then pass the full file content (as string) back to the caller for further parsing and then determine the require range (report time start to end) and accumulate the total value in the field (2) - Double.

Maybe you should pass the filestream resource instead.

I know it's tempting to read the full file and process that. It's the simplest way most of the time. But it's asking for memory issues. On a PC with 8GB RAM it's not such a big deal with text files, but imagine doing the same with a 1080p video file. Luckely video containers also work with blocks of data, for this particular reason.

When you know every day it writes 100 bytes, you know that day 4 starts after 300 bytes. Reading 100 bytes from 300 is very fast. I would go for that approach.