7 replies to this topic

[ukkiwisurfer]

Hi,

 

I'm getting a 'Buffer OVFLW' message being logged to the console window. I have a general try/catch block at the application level, but it appears that this error is not being caught by the try/catch block. Can someone explain how this could happen? 

 

 try

 {

    var serviceManager = new ServiceManager();

    serviceManager.Run();

  }

  catch (Exception ex)

  {

     Debug.Print(ex.StackTrace);

  } 

 

And from the logging output (console window) I see it as:

 

2015-11-11 17:32:02.288 | 14 | FileSystemLogger | File matches pattern. File: '\SD\data\tosend\20151111173150715.data'.

Buffer OVFLW

Buffer OVFLW

 

It appears that it isn't being caught as I'd expect - I wanted to see what the stack trace was to work out what was causing the software to fail.

[dankline]

Set a breakpoint on the debug line.  Check to size and content of the ex.Stack.Trace Variable.  Make sure that you are not overflowing the stack with recursion or iteration.  I'd need to see more of the code to guess better.  The debugger should help you sort it out.

[ukkiwisurfer]

Hi Dankline,

 

Thanks for your reply. Unfortunately I don't know where its crashing, hence my attempt to use an application level try/catch block. I have the process running as I type with the breakpoint set to the debug.print statement within the catch block.

 

Its a multi-threaded application with separate data capture, data storage and transmission threads, accessing the ND3's inbuilt SD card.

 

My suspicion is that because I'm pushing the SD card in terms of volume of writes and reads, the SD card is getting corrupt. The error seems to appear around the point where it enumerates over folders to get the data files.

 

However its not consistent in terms of when it happens. Sometimes its within 10 minutes, other times its after a couple of hours consistent running.

 

I have noticed that if I try running it debug from within VS2013 it usually takes much longer to happen than if I just run the board and connect to it via USB using the .NET micro framework deployment tool (and view the output via the console listview). Which makes me wonder whether the debugging is actually masking the issue because of the additional latency it introduces.

 

In the normal .NET world you can use WinDbg.exe to analyse the crash dump files. Is there anything equivalent for .NET MF?

[ukkiwisurfer]

I would add that this error message isn't being recorded through my logging system, and when it happens it takes the whole board down. Only a restart will recover it. No .NET exception is being raised either which makes me highly suspicious.

 

1. Has anyone encountered this type of error before?

2. Any suggestions as to how to capture more error information around what is actually happening? If its not raising a .NET exception, are their any type of crash dump files that get generated? 

 

Thanks,

Jason.

[KiwiDev]

Hi Jason,

 

I have had this sort of problem before when my serial port code has been a bit flakey and other code has been hogging the processor.

 

I can see Buffer OVFLW in the netduino\netmf github repository

 

netmf / DeviceCode / pal / COM / usart / usart.cpp 

 

Could you tell us a bit more about your application so we know where to look?

 

@KiwiBryn

blog.devmobile.co.nz

[ukkiwisurfer]

Hi KiwiBryn,

Sure, I can shed some details on the project. I'm building out a data capture platform, initially as a PoC for within the home.

The data is captured via a UDP broadcast that is happening elsewhere on the network. These packets are then saved as files on the SD card.

A separate thread runs, looking for files in a specific subdirectory, to then load each, one at a time into memory, before using amqp lite to transmit the data to a gateway (rabbitMQ).

Due to the lack of real analysis tools for the net mf platform that the desktop versions of .net support I've also had to write a generalised logging framework. To work out what is happening.

I wonder how you approach the same w.r.t logging and tracing on a constrained device?

Thanks,
Jason.

[KiwiDev]

Hi Jason,

 

I have found these sort of problems are usually in timer callbacks, eventhandlers(yours or NetMF) or background threads,

 

In my most recent issue the exceptions were coming from inside the serial port driver code.

 

I'm always careful to have a try-catch loop as a last resort to stop any uncaught exceptions nuking the "thread" in eventhandlers, timer callbacks and background threads.

 

I user Reflector to drill into the NetMF to see what is going on a lot. 

 

 

Does your UDP code stitch buffers together or just push each individual payload to SD card using 

 

public static void WriteAllBytes(string path, byte[] bytes)

 

or similar?

 

For logging I have a pretty "thin" wrapper around Debug.Print (just prefixes with current date & time). 

 

Sometimes I have had to resort to toggling LEDs to get diagnostics for heisenbugs.  

 

Cheers

 

@KiwiBryn

blog.devmobile.co.nz

[ukkiwisurfer]

Hi KiwiBryn,

 

I too have try/catch blocks (normally) around critical code sections. I realised over the weekend that even though I have an application level try/catch block, as the UDP capture code is running in its own thread, I was missing the associated try/catch block from the relevant thread looping code. Since making this change I'm not getting buffer overflow messages any more. 

 

The UDP thread captures whatever is in the buffer and then adds it to an internal (memory) queue. Another thread writes these blocks to the SD card (think micro-service architecture) using a StreamWriter in order to add additional metadata around the payload body. A third thread(service) reads these files off the SD and publishes to AMQP.

 

My next set of challenges is around an Unhandled Exception message being thrown from within the framework code (not a .NET exception) and a memory leak.

 

I have no idea where the un-handled exception is being thrown from but it seems to be happening around the point that my memory leak forces memory levels to a critical level. So my first plan of attack is to find the memory leak. To do this I'm going to disable all but one service and see if I can identify which service is leaking. Any other suggestions on approach? 

 

I've introduced a Debug.GC(true) into the code where I persist the file data via AMQP, which has helped considerably and I print out the free memory for reference. However over a couple of hours I can see the memory level gradually dropping. But obviously Debug.GC() just sticking plaster over a much more fundamental issue.

As I've introduced extensive logging statements into the code that also muddies the waters with regards to memory allocations for strings.

 

It would be nice if the tooling around .NETMF development moved into the late 20th century and not require print statements to identify leaks and the like. In cases like this I miss my crash dumps and WinDbg (however arcane the commands are) it definitely takes the guess work out of where objects are being pinned in memory.

 

Just imagine how more productive we would be.... :-)

 

Thanks,

Jason.