The Netduino forums have been replaced by new forums at community.wildernesslabs.co.
This site has been preserved for archival purposes only
and the ability to make new accounts or posts has been turned off.
I often get socket error #10055 (WSAENOBUFS). I read this happens when the send-buffer from the socket is too full already when executing a Send() call.
I looked but this doesn't have anything to do with the available memory on the Netduino, so appairently there's a seperate part of memory for the socket send buffer.
I wrote two work-arounds for this, but I'm looking for a more decent solution.
Workaround 1:
bool Retry = true;
while (Retry)
{
Retry = false;
try
{
this._Sock.Send(Data);
}
catch (SocketException e)
{
if (e.ErrorCode == 10055) // WSAENOBUFS
{
// Appairently the buffer is full. Lets wait .1 sec and try again
Thread.Sleep(100);
Retry = true;
}
else
throw;
}
}
Solution:
When the Send() call fails with socket error #10055, I'll wait for a tenth of a second, and I just try it again.
Downsides:
1. I don't like solving problems by just catching it. It just doesn't feel right.
2. When the debugger isn't active, for some reason that try/catch fails and my program just crashes and halts.
So this workaround is totally useless in a production environment.
Solution:
Every byte gets 0.01 second to be transferred.
Downsides:
1. The number 10 came up by trial/error in one specific test case. I'm not sure if this delay will always work.
2. Code execution gets way slower then it should be.
I tested this with a few Netduino Pluses with two firmware versions:
SolutionReleaseInfo.solutionVersion: 4.1.0.6
SolutionReleaseInfo.solutionVendorInfo: Netduino Plus by Secret Labs LLC
SoftwareVersion.BuildDate: Feb 14 2011
and:
SolutionReleaseInfo.solutionVersion: 4.2.0.0
SolutionReleaseInfo.solutionVendorInfo: Netduino Plus (v4.2.0.0 RC5) by Secret Labs LLC
SoftwareVersion.BuildDate: Apr 8 2012
Now I would imagine a nice solution would be checking how full the Send() buffer is, and just wait until the number reaches 0, until I call another Send() call. But I can't find a way to do so.
So, if anyone can help me with a decent solution, I would be very grateful!
Now I would imagine a nice solution would be checking how full the Send() buffer is, and just wait until the number reaches 0, until I call another Send() call. But I can't find a way to do so.
So, if anyone can help me with a decent solution, I would be very grateful!
In reviewing the firmware source I didn't see a way to pull this off either without building a custom firmware. Even considering a custom firmware I ran into a dead end on the source tree to actually locate the buffer implementation. I got as far as NativeSocket, which appears to just be a definition for some linked externals. I haven't had a chance to investigate much further yet. Curious to see if someone finds a better solution to this, and also any pointer to determine where the source is for the referenced calls in NativeSocket.
I got as far as NativeSocket, which appears to just be a definition for some linked externals.... any pointer to determine where the source is for the referenced calls in NativeSocket.
NativeSocket is a namespace alias for Microsoft.SPOT.Net.SocketNative, the managed wrappers call native counterparts implemented in CLR\Libraries\SPOT_Net\spot_net_native_Microsoft_SPOT_Net_SocketNative.cpp, which in turn call socket functions from DeviceCode\Pal\LWIP.
I would imagine a nice solution would be checking how full the Send() buffer is, and just wait until the number reaches 0, until I call another Send() call.
But isn't Send() method call blocking? If it is supposed to block until all of the bytes in the buffer are sent (or timeout expires), how do you plan to retrieve the number of bytes that have or have not yet been transmitted? From where such code will be called?
Do you access the socket concurrently, i.e. from multiple threads? Isn't it possible there is a previous operation in progress (or another, e.g. receive) while the code attempts to call Send() method?
Hi Stefan
A few things spring to mind, the most obvious is...
Does the same thing happen if you try and send the socket async, as it seems from your description the code is trying to send before the previous transmission has finished. By moving to the async model a separate thread will handle the actual transmission with a call back to the main body of code, so rather than arbitarily sleeping the thread you can block until the call back has responded ensuring that all data is transmitted fully.
Generally i have had all sorts of problems trying to deal with synchronous sockets in the desktop world, where as the added complexity of the async model (although if you have ever used jquery callbacks should be a familiar concept) has paid off for it self through the solving of a lot of weird network issues i have encountered plus once you have gotten your head around the model it works as a pretty nice standardised pattern.
That said i find it quite troubling that your try catch solution doesnt work without the debuggger attached, surely a core c# language construct should work without a debugger being attached, if thats the case then that is a major flaw in netmf/netduino implementation.
Despite its inelegance i would say that it is an adequet solution, i have lost the link but there was a good blog post i read years ago basically complaining about how certain events have been given exception status, despite the fact that they are actually expected thus forcing you to over load your catch blocks to explicitly ignore them and stop the error being thrown, because the original developers missued the exception handling routines for messaging.
Nak.
But isn't Send() method call blocking? If it is supposed to block until all of the bytes in the buffer are sent (or timeout expires), how do you plan to retrieve the number of bytes that have or have not yet been transmitted? From where such code will be called?
Appairently it's not blocking, otherwise that error shouldn't show up, I believe.
Does the same thing happen if you try and send the socket async, as it seems from your description the code is trying to send before the previous transmission has finished.
It could happen in theory in the app I wrote, so I simplified a repro, listed below, showing that that's not the cause of the problem.
That said i find it quite troubling that your try catch solution doesnt work without the debuggger attached, surely a core c# language construct should work without a debugger being attached, if thats the case then that is a major flaw in netmf/netduino implementation.
That's absolutely true. Now I have a small repro, I'm going to hit in a few bugs I think.
Okay, I made an easy repro code:
using System;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Net.NetworkInformation;
using Microsoft.SPOT.Hardware;
using SecretLabs.NETMF.Hardware;
using SecretLabs.NETMF.Hardware.NetduinoPlus;
namespace NetduinoPlusApplication1
{
public class Program
{
public static void Main()
{
// Prints out all IP addresses
NetworkInterface[] Ips = NetworkInterface.GetAllNetworkInterfaces();
for (int i = 0; i < Ips.Length; ++i)
Debug.Print(Ips[i].IPAddress.ToString());
// Listens on the port 23
Socket Listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
Listener.Bind(new IPEndPoint(IPAddress.Any, 23));
Listener.Listen(1);
// Accepts the first connection
Socket Connection = Listener.Accept();
// Stops further listening
Listener.Close();
// ANSI modes: Normal, Forecolor: White, Backcolor: Black ( as String(): "\x1b[00;37;40m" )
byte[] AnsiData = { 27, 91, 48, 48, 59, 51, 55, 59, 52, 48, 109 };
// Text ( as String(): "Hello World!\r\n" )
byte[] TextData = { 72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33, 13, 10 };
while (true)
{
Connection.Send(AnsiData);
Connection.Send(TextData);
}
}
}
}
As you can see, no weird, unexpected stuff. I just send in data, very quickly. Could someone else try this on his netduino?
To reproduce, use a terminal client to port 23 on the Netduino.
I have managed to repro the issue on a netduino here.
Having played with the demo code it does appear that the Send method is returning before the ND+ has actually finished transmitting to the endpoint.
Adding a Thread.Sleep(100) between each send seemed to fix the issue for me, so i hypothesise that some sort of delay is required to allow the ND+ to shift the data out. Given that this is a telnet server i wouldn't expect this to be much of a problem due to the relatively slow response times between messages...
That said i think a bit of a change in architecture would make a more useful as you could add reading to the network comms thread as well, then just lock and sync a pair of queue's between the main and network comms thread.
using System;
using System.Collections;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Net.NetworkInformation;
namespace NetworkTest {
public class Program {
#region Private Member Variables
private static readonly object QueueSyncObject = new object();
private static readonly Queue ResponseQueue = new Queue();
private static Socket _connection;
private static Thread NetworkCommsThread = new Thread(NetworkCommsProcessor);
private static byte[] AnsiData = { 27, 91, 48, 48, 59, 51, 55, 59, 52, 48, 109 };
private static byte[] TextData = { 72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33, 13, 10 };
#endregion
#region Application Methods
public static void Main() {
// Prints out all IP addresses
NetworkInterface[] Ips = NetworkInterface.GetAllNetworkInterfaces();
for(int i = 0; i < Ips.Length; ++i) {
Debug.Print(Ips[i].IPAddress);
}
// Listens on the port 23
Socket Listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
Listener.Bind(new IPEndPoint(IPAddress.Any, 23));
Listener.Listen(1);
// Accepts the first connection
_connection = Listener.Accept();
NetworkCommsThread.Start();
// Stops further listening
Listener.Close();
var toggle = true;
while(true){
lock(QueueSyncObject){
ResponseQueue.Enqueue((toggle) ? AnsiData : TextData);
}
toggle = !toggle;
}
}
private static void NetworkCommsProcessor() {
while(true) {
lock(QueueSyncObject) {
if(ResponseQueue.Count <= 0) {//No messages to send, so skip the loop
continue;
}
var bytesToSend = (byte[])ResponseQueue.Dequeue();
try {
_connection.Send(bytesToSend);
}
catch(SocketException ex) {
switch(ex.ErrorCode) {
case 10054://Safe to ignore client has just disconnected
//Add additional safe to ignore expected exceptions
break;
default:
throw;
}
}
catch(Exception ex) {
//Something really weird has happened here... :-/
throw;
}
//Although there still is a thread sleep its not in the main thread so
//shouldnt stop additional work happening :-)
Thread.Sleep(100);
}
}
}
#endregion
}
}
That said i think a bit of a change in architecture would make a more useful as you could add reading to the network comms thread as well, then just lock and sync a pair of queue's between the main and network comms thread.
Oh yes I know that, my repro code doesn't look like the stuff I'm using. The good part about repro-code: short, easy to read, no excess code
I'm glad you can reproduce the issue though! Now I know it's not just me, or my network.
Given that this is a telnet server
Well, for me it really makes a difference if I add a sleep in between all Send-commands. A lot of data is happening, also behind the scenes. If I type "HELP" and get a list of 10 lines, it would take quite some time for that to show up, which feels very slow.
Having played with the demo code it does appear that the Send method is returning before the ND+ has actually finished transmitting to the endpoint.
Indeed, and that's not very useful. I want a confirmation my data is sent before I send more data. Or query the actual buffer state.
Oh yes I know that, my repro code doesn't look like the stuff I'm using. The good part about repro-code: short, easy to read, no excess code
I'm glad you can reproduce the issue though! Now I know it's not just me, or my network.
Well, for me it really makes a difference if I add a sleep in between all Send-commands. A lot of data is happening, also behind the scenes. If I type "HELP" and get a list of 10 lines, it would take quite some time for that to show up, which feels very slow.
I dont think the amount of data matters, but the delay between transmissions does i say this as i have just added a line break to the end of the hello world string (13,10) and repeated hello world 130 times, so as long as you could concatenate the response and send as one big chunk it would appear to be ok....
Removed snippet as it didnt wrap and frankly a 9000 odd char repeating snippet adds very little value
I dont think the amount of data matters, but the delay between transmissions does i say this as i have just added a line break to the end of the hello world string (13,10) and repeated hello world 130 times, so as long as you could concatenate the response and send as one big chunk it would appear to be ok....
True, but my approach should work just fine in my humble opinion. You're suggesting a work-around #3: creating my own sending buffer and sending all data at once. I'm just wondering if there's a decent solution. Normally, Socket.Send() should just work fine.
I can look into combining as much data as possible and send it all at once, but I prefer not to. It would effectively consume more memory.
True
I think the only real fix not involving a workaround will involve a firmware fix, to implement what you originally suggested of a mechanism to determine the send buffer state.
Interestingly it if you try and get or set the send buffer size you receive a 10042 (bad protocol option) which suggests it hasn't been implemented, if the send buffer size could be determined you could at least dynamically calculate a suitable delay but again its not an elegant solution either. It would be interesting to know what the throughput from managed to code to network interface is. I suspect given the results i have gained from sending a 9k byte array the requirement for a delay occurs following the transmission of the data. I suspect that their is some costly process involved with ending the transmission as the 100mS delay was needed for both the original hello world string and the subsequent buffered message queue.
With regards to your concerns about consuming more memory by using a buffer I personally wouldn't be to concerned due to the fact that the standard screen buffer size (under windows anyway) for the command shell is 80 chars by 25 lines so in other words you would need a single 2k buffer to represent a full screen of data (assuming you only deal with ASCII text). Which could be represented as a static readonly byte array which should not incur any GC overhead... But without knowing the memory requirements of your app i cant say whether that would be acceptable or not...
Nak.
I think the only real fix not involving a workaround will involve a firmware fix, to implement what you originally suggested of a mechanism to determine the send buffer state.
Indeed. I think I'm going to file a bug report for this. But first I wanted to make sure if it's a bug.
Interestingly it if you try and get or set the send buffer size you receive a 10042 (bad protocol option)
Indeed, I also tried that
With regards to your concerns about consuming more memory by using a buffer I personally wouldn't be to concerned due to the fact that the standard screen buffer size (under windows anyway) for the command shell is 80 chars by 25 lines so in other words you would need a single 2k buffer to represent a full screen of data (assuming you only deal with ASCII text).
True for a part, but with an interactive console, a lot of protocol data will be sent over the line. I already found that using a full colored shell got quite slow if I added sleeps.
Maybe (I did not test this) it helps to check first if the socket it ready to send data:
if (Connection.Poll(-1, SelectMode.SelectWrite)) {
Connection.Send( yourData );
}
Looks interesting, will try it tonight. That sounds like a fast and efficient workaround!
Maybe (I did not test this) it helps to check first if the socket it ready to send data:
if (Connection.Poll(-1, SelectMode.SelectWrite)) {
Connection.Send( yourData );
}
Doesn't seem to make any difference for me I'm afraid
However the plot thickens...
I do not need the thread.sleep at all if and only if the message i am sending is > 21 bytes in length.... Quite why such an arbitrary length works is any ones guess, which leads to workaround #4
If the line is less than 21chars long pad with spaces to make it 21 chars long
Here is the code i got going, as you can see i simply added spaces to the end of the string and tested the ANSI codes work with a padded string by setting the background to red and it seems to work ok, admittedly I have done nothing about the formatting but that was never the issue here
using System.Net;
using System.Net.Sockets;
using Microsoft.SPOT;
using Microsoft.SPOT.Net.NetworkInformation;
namespace NetworkTest {
public class Program {
public static void Main() {
// Prints out all IP addresses
NetworkInterface[] Ips = NetworkInterface.GetAllNetworkInterfaces();
for(int i = 0; i < Ips.Length; ++i)
Debug.Print(Ips[i].IPAddress.ToString());
// Listens on the port 23
Socket Listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
Listener.Bind(new IPEndPoint(IPAddress.Any, 23));
Listener.Listen(1);
// Accepts the first connection
Socket Connection = Listener.Accept();
// Stops further listening
Listener.Close();
// ANSI modes: Normal, Forecolor: White, Backcolor: Red ( as String(): "\x1b[00;37;41m" )
byte[] AnsiData = { 27, 91, 48, 48, 59, 51, 55, 59, 52, 49, 109, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32 };
// Text ( as String(): "Hello World!\r\n" )
byte[] TextData = { 72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33, 32, 32, 32, 32, 32, 32, 32, 13, 10 };
while(true) {
Connection.Send(AnsiData);
Connection.Send(TextData);
}
}
}
}
Forgot to copy code to notepad first so lost all formatting and got a load of RTF color coding instead
Doesnt seem to make any difference for me i'm afraid
However the plot thickens...
I do not need the thread.sleep at all if and only if the message i am sending is > 21 bytes in length.... Quite why such an arbitary length works is any ones guess, which leads to workaround #4
If the line is less than 21chars long pad with spaces to make it 21 chars long
Oh interesting! I'm going to see if I can do something with that! For telnet, I could add dummy data. For other protocols it would be different though... Have to think it over well
I can only wonder how this "min message size" issue has gone under the radar?
I can only assume that people have been sending larger chunks of data, but i would have thought hello world telnet would have been done before this!
Nak.
I can only wonder how this "min message size" issue has gone under the radar?
I can only assume that people have been sending larger chunks of data, but i would have thought hello world telnet would have been done before this!
Well, it only occures when -repeatedly- sending small messages, in a short amount of time. That's not something that would be done with a Hello World sample
I'm going to test it tonight, and let you all know!
With a lot of help from nakchak I found a -kinda proper- workaround.
The issue is still there, but for now I used:
this._Sock.Send(Data);
// I implemented this sleep here for an issue I encountered.
// For more details, read http://forums.netduino.com/index.php?/topic/4555-socket-error-10055-wsaenobufs/
if (Data.Length < 32) Thread.Sleep(Data.Length * 10);
Also, I tried to avoid sending smaller chunks of data since the problem only occures when sending small chunks.
Its open source now, just fix it yourself isnt that what open source is all about? broken by default™ is what I always say.
But otherwise, it is a tiny processor. You can not keep it 100% busy and expect it to also handle hardware level tasks, like interrupts from an ethernet port. I do remember seeing something like 21 bytes being the difference between trying to immediately send vs buffering data. Actually I think I remember 21 being discussed as a mistake, it was suppose to be 20 but the code was off by 1. Then again that may have been something similar but entirely elsewhere. But the more important, code that choses immediate vs delayed, especially in small embedded systems, is quite common.
