Ethernet stop working after around 20-60 minutes running

I wrote a simple TCP server that accepts client connections and sends them change information on the digital input pins, and I noticed that after around an hour or two (varies), Ethernet connectivity on the Netduino plus stops working. Until I power cycle or reset the Netduino my application won't accept new connections (all existing connections are dropped), and more interestingly I'm no longer able to ping the Netduino from my PC. In general, the rest of the application seems to be running normally when it's in this state; changes in input pins trigger the onboard LED to turn on/off as expected. When I pause the debugger in Visual Studio I see the code is waiting on a socket.Accept(), call as if everything is working normally. For testing purposes I tried to build the simplest TCP socket server application I could (attached), but even with this I have the same problem. I even tried changing the code around to bypass the entire TCP socket server by instead sending UDP messages to a hard coded server on my network, but no luck. I'd dismiss it as a hardware issue if it didn't work at all, but since it works for a while after it power's up I'm not so sure. Anyone else seen something like this, or maybe have any ideas? One last thing: after going through the steps above using the latest release firmware I went ahead and updated to the 4.2 beta 1 bits to see if the issue would magically go away, however it made no difference.

Hi Derek, Can you try something really quick for me? Build an app which requests a web page from one of your own servers. Have it request a page every 30 seconds. Then let it run for a few hours. Does that work? If so, it's probably a glitch in the lwIP networking stack. If not, it's either a router issue or a really really strange hardware issue. [Next steps would be to plug the Netduino directly into your PC to do a long-term connectivity test, or to try it on another Netduino Plus, etc.] Welcome to the Netduino community, Chris

Hi Chris, I tried running a simple webpage polling app as you suggested, and I had the same result. I also tried putting a different network switch down right by the Netduino, but it didn't make any difference. I was able to get it running reliably by keeping it off of our network and connecting it directly to a PC using a crossover cable. I'd need to do some more testing to be positive, but it really seems like the ambient broadcast traffic on our network causes the network stack to fall down after running for a relatively short period of time. I have another Netduino Plus on order, and I'll see if I can use it to get some more definitive troubleshooting information. I'd like to get it working reliably on our network for some other projects I have in mind. Thanks for the help, Chris.

Hi Chris,

I tried running a simple webpage polling app as you suggested, and I had the same result. I also tried putting a different network switch down right by the Netduino, but it didn't make any difference. I was able to get it running reliably by keeping it off of our network and connecting it directly to a PC using a crossover cable.

I'd need to do some more testing to be positive, but it really seems like the ambient broadcast traffic on our network causes the network stack to fall down after running for a relatively short period of time. I have another Netduino Plus on order, and I'll see if I can use it to get some more definitive troubleshooting information. I'd like to get it working reliably on our network for some other projects I have in mind.

Thanks for the help, Chris.

I can confirm this bug.

Netduino will stay up for days on a quiet, local LAN. If I take it and plug it into our university's LAN, it freezes within minutes. The board remains responsive to input pin events, but the network connectivity no longer works until you press the reset button.

It seems the Netduino's stack can't handle tons of broadcast traffic created by management services on large lans. (Cisco proprietary, or STP, etc.)

A potential work around would be to put a cheap nat router between the netduino and lan, and port forward 80 on the wan side ip to the netduino. That should block all of the broadcast noise, but let in the http requests.

I can confirm this bug.

Netduino will stay up for days on a quiet, local LAN. If I take it and plug it into our university's LAN, it freezes within minutes. The board remains responsive to input pin events, but the network connectivity no longer works until you press the reset button.

It seems the Netduino's stack can't handle tons of broadcast traffic created by management services on large lans. (Cisco proprietary, or STP, etc.)

Exact same behavior in my university, after few seconds of our typical 30-45KB/s background broadcast noise, while it worked fine for weeks at home with low traffic.

A potentials work around would be to put a cheap nat router between the netduino and lan, and port forward 80 on the wan side ip to the netduino. That should block all of the broadcast noise, but let in the http requests.

But this kill the beauty of a tiny electronic to connect something to internet. If an additional router was requited, then is better and far cheaper to buy an OpenWRT compliant router such as Linksys 54G for 60€ and program it in C

I would be just happy if it would be a way to reset the eth interface when N+ detect lose of network connectivity

I would be just happy if it would be a way to reset the eth interface when N+ detect lose of network connectivity

I used an external watchdog timer chip to overcome the occasional lockups by the Netduino Plus. On a quiet LAN, it will still lockup, just not as often. In my most recent test, it ran for 2 days before locking up.

Could you please post this as a bug report at netmf.codeplex.com? Perhaps there are network settings (buffers, sockets, etc.) that could be changed in the .NET MF PK to be able to discard extra traffic? Generally with microcontrollers running software network stacks, broadcasting a lot of messages at them can cause a denial-of-service type of issue...as they simply can't keep up with processing it all. Chris

Could you please post this as a bug report at netmf.codeplex.com? Perhaps there are network settings (buffers, sockets, etc.) that could be changed in the .NET MF PK to be able to discard extra traffic?

Generally with microcontrollers running software network stacks, broadcasting a lot of messages at them can cause a denial-of-service type of issue...as they simply can't keep up with processing it all.

Chris

I don't have enough details yet. I'll try to find the time to setup a hub in one of our campus labs and get some wireshark data.

I used an external watchdog timer chip to overcome the occasional lockups by the Netduino Plus. On a quiet LAN, it will still lockup, just not as often. In my most recent test, it ran for 2 days before locking up.

Hi. Can you elaborate? I don't know how a watchdog chip could monitor just ethernet response issues? The board's logic, input pins, etc, remains responsive.

Hi, I have the very same problem with my NETduino Plus. It seems that Ethernet link establishment negotiation with a Switch/Router (I have 100Mbs) does not succeed for some reason and NETduino tries and tries it repeatedly. That is why IP over Ethernet does not work either. It is strange that flashing between different firmware versions did not help. I have also updated the bootloader with SAM-BA but Ethernet connectivity remains dead. Best Regards, Seppo

Hi Seppo,

I have the very same problem with my NETduino Plus. It seems that Ethernet link establishment negotiation with a Switch/Router (I have 100Mbs) does not succeed for some reason and NETduino tries and tries it repeatedly. That is why IP over Ethernet does not work either. It is strange that flashing between different firmware versions did not help. I have also updated the bootloader with SAM-BA but Ethernet connectivity remains dead.

Do you have Wireshark? Can you capture this negotation attempt and then post it as a bug report at netmf.codeplex.com (and copy the link here so we can all take a look at it)?

Netduino uses the industry-standard lwIP networking stack build into NETMF and used by many products...but part of open source is finding these types of glitches and then contributing to the core libraries. I know that lwIP has been used quite widely in the real world, but I'm sure that there will be issues to work out due to the large number of IP implementations. Let's see if we can get this one sorted out.

Chris

Hi Chris, Thanks for your quick response. Luckily we do not need any deeper studies with Wireshark. I restarted my Ethernet Switch and after that my NETduino Plus had its Ethernet link up and running :-) Strange that my PC which was connected to another port in the same switch worked all the time as usual. I studied the NETduino HW a bit and found that DM9161A chip performs autonegotiation for Ethernet link establishment autonomously. That is why I was about to suspect a HW failure in that chip. Anyway, everything seems to be OK now and at least I got some more experience with this tiny work horse. Best Regards, Seppo

I studied the NETduino HW a bit and found that DM9161A chip performs autonegotiation for Ethernet link establishment autonomously.

I'm wondering if maybe you've come across something that has always plagued systems. I know in the windows world auto negotiation has always been problem some for devices and I have worked with network technicians who have recommended that it be turned off.

Maybe we should look at adding the capability to set the port speed on the Netduino Plus. If the network folks are having people turn it off, maybe we need to provide similar capabilities to allow the device to coexist with all of the other devices on the network.

Maybe we should look at adding the capability to set the port speed on the Netduino Plus. If the network folks are having people turn it off, maybe we need to provide similar capabilities to allow the device to coexist with all of the other devices on the network.

Yes, I agree and think that a capability for "manual" Ethernet PHY settings would add the overall reliability in the critical applications in general.
In my opinion the selection between using DHCP or fixed IP addresses/network settings tackles something similar but residing on the upper layers of the protocol stack.

Best Regards,
Seppo

This looks like the same issue: http://netmf.codeple...m/workitem/1336 There's a solution posted below.