Netduino home hardware projects downloads community

Jump to content


The Netduino forums have been replaced by new forums at community.wildernesslabs.co. This site has been preserved for archival purposes only and the ability to make new accounts or posts has been turned off.

Wim Roeling's Content

There have been 11 items by Wim Roeling (Search limited from 11-April 19)


By content type

See this member's

Sort by                Order  

#42433 "Chunked" webserver to send large amounts of data

Posted by Wim Roeling on 30 December 2012 - 01:18 PM in General Discussion

            req.SendFinalChunkedResponse();

Based on work done by others I rewrote the webserver project to enable it to send responses that are larger than Netduino's internal memory. To do this I modified the Request object to support chunked transfer encoding (see http://en.wikipedia....ansfer_encoding ).

In order to send a chunked response, the data should be split in "chunks" of a reasonable size. In almost all cases this is very ease: if you want to send an image that is far bigger than Netduino's memory, just readit in memory from the SD card in chunks of, say, 1024 bytes and send each chunk separately. Thus, Netduino's memory will not use more than 1024 bytes at the time while the complete immage is send.

Before the first chunk a "header"should be send (this will also initialize chunked transfer) and after the last chunk the response had to be finalized.

 

Here is my code of the enhanced Request:

using System;using System.Net.Sockets;using System.Text;using System.Net;using System.Threading;namespace WebSpace{    /// <summary>    /// Holds information about a web request and is able to send chuncked responses    /// </summary>    public class Request : IDisposable    {        private string method;        private string url;        private Socket client;        static char[] hexDigits = { '0', '1', '2', '3', '4', '5', '6', '7','8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};        //convert a byte to a hex string        public static string ToHexString(int i)        {            int b1 = i & 0x0F;            int b2 = (i >> 4) & 0x0F ;            int b3 = (i >> 8) & 0x0F;            int b4 = (i >> 12) & 0x0F;            if (b4 > 0) return "" + hexDigits[b4] + hexDigits[b3] + hexDigits[b2] + hexDigits[b1];            else if (b3 > 0) return "" + hexDigits[b3] + hexDigits[b2] + hexDigits[b1];            else if (b2 > 0) return "" + hexDigits[b2] + hexDigits[b1];            else return "" + hexDigits[b1];        }        internal Request(Socket Client, char[] Data)        {            client = Client;            ProcessRequest(Data);        }        // Request method        public string Method        {            get { return method; }        }        // URL of request        public string URL        {            get { return url; }        }        // Client IP address        public IPAddress Client        {            get            {                IPEndPoint ip = client.RemoteEndPoint as IPEndPoint;                if (ip != null) return ip.Address;                return null;            }        }        // Send a response back to the client        public void SendResponse(string response, string type = "text/html")        {            if (client != null)            {                string header = "HTTP/1.1 200 OKrnContent-Type: " + type + "; charset=utf-8rnContent-Length: " + response.Length.ToString() + "rnConnection: closernrn";                client.Send(Encoding.UTF8.GetBytes(header), header.Length, SocketFlags.None);                client.Send(Encoding.UTF8.GetBytes(response), response.Length, SocketFlags.None);            }        }        public void SendHeaderChunkedResponse(string type = "text/html")        {            if (client != null)            {                string header = "HTTP/1.1 200 OKrnContent-Type: " + type + "; charset=utf-8rnTransfer-Encoding: chunkedrnConnection: closernrn";                client.Send(Encoding.UTF8.GetBytes(header), header.Length, SocketFlags.None);                Thread.Sleep(10) ;            }        }        public void SendNextChunkedResponse(string response)        {            if (client != null)            {                string hex = ToHexString(response.Length);                client.Send(Encoding.UTF8.GetBytes(hex + "rn"), hex.Length+2, SocketFlags.None);                client.Send(Encoding.UTF8.GetBytes(response + "rn"), response.Length+2, SocketFlags.None);                Thread.Sleep(10) ;            }        }        public void SendNextChunkedResponse(byte[] response, int length)        {            if (client != null)            {                string hex = ToHexString(length);                client.Send(Encoding.UTF8.GetBytes(hex + "rn"), hex.Length + 2, SocketFlags.None);                client.Send(response, length, SocketFlags.None);                client.Send(Encoding.UTF8.GetBytes("rn"), 2, SocketFlags.None);                Thread.Sleep(10);            }        }        public void SendFinalChunkedResponse()        {            if (client != null)            {                client.Send(Encoding.UTF8.GetBytes("0rnrn"), 5, SocketFlags.None);                Thread.Sleep(10);            }        }        public void Send404()        {            string header = "HTTP/1.1 404 Not FoundrnContent-Length: 0rnConnection: closernrn";            if (client != null)            {                client.Send(Encoding.UTF8.GetBytes(header), header.Length, SocketFlags.None);            }        }        private void ProcessRequest(char[] data)        {            string content = new string(data);            string firstLine = content.Substring(0, content.IndexOf('n'));            // Parse the first line of the request: "GET /path/ HTTP/1.1"            string[] words = firstLine.Split(' ');            method = words[0];            url = words[1];            // Could look for any further headers in other lines of the request if required (e.g. User-Agent, Cookie)        }        #region IDisposable Members        public void Dispose()        {            if (client != null)            {                client.Close();                client = null;            }        }        #endregion    }}

 

Here is the code of the Listener:

using System;using System.Net.Sockets;using System.Net;using System.Text;using System.Threading;namespace WebSpace{    public delegate void RequestReceivedDelegate(Request request);    public class Listener : IDisposable    {        const int maxRequestSize = 1024;        readonly int portNumber = 80;        private Socket listeningSocket = null;        private RequestReceivedDelegate requestReceived;        public Listener(RequestReceivedDelegate RequestReceived)            : this(RequestReceived, 80) { }        public Listener(RequestReceivedDelegate RequestReceived, int PortNumber)        {            portNumber = PortNumber;            requestReceived = RequestReceived;            listeningSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);            listeningSocket.Bind(new IPEndPoint(IPAddress.Any, portNumber));            listeningSocket.Listen(10);            new Thread(StartListening).Start();        }        ~Listener()        {            Dispose();        }        public void StartListening()        {            while (true)            {                try                {                    using (Socket clientSocket = listeningSocket.Accept())                    {                        IPEndPoint clientIP = clientSocket.RemoteEndPoint as IPEndPoint;                        // Debug.Print("Received request from " + clientIP.ToString());                        var x = clientSocket.RemoteEndPoint;                        int availableBytes = clientSocket.Available;                        // Debug.Print(DateTime.Now.ToString() + " " + availableBytes.ToString() + " request bytes available");                        int bytesReceived = (availableBytes > maxRequestSize ? maxRequestSize : availableBytes);                        if (bytesReceived > 0)                        {                            byte[] buffer = new byte[bytesReceived]; // Buffer probably should be larger than this.                            int readByteCount = clientSocket.Receive(buffer, bytesReceived, SocketFlags.None);                            using (Request r = new Request(clientSocket, Encoding.UTF8.GetChars(buffer)))                            {                                // Debug.Print(DateTime.Now.ToString() + " " + r.URL);                                if (requestReceived != null) requestReceived(r);                            }                        }                    }                }                catch (Exception e)                {                    Thread.Sleep(10);                    Program.Log("!!! ERROR !!! " + e.Message + "n" + e.StackTrace);                }            }        }        #region IDisposable Members        public void Dispose()        {            if (listeningSocket != null) listeningSocket.Close();        }        #endregion    }}

And finaly, here is some example code to send a large XML stream. Note the following three calls:

            req.SendHeaderChunkedResponse("text/xml");
                        req.SendNextChunkedResponse(chunk, length);
            req.SendFinalChunkedResponse();

An example to send the contents of a large files in simple XML as <data> ..... </data>

        private const int CHUNK_SIZE = 4096;        private static void List(Request req, string file)        {            byte[] chunk = new byte[CHUNK_SIZE + 256];            int len = 0;            req.SendHeaderChunkedResponse("text/xml");            len = add(chunk, len, "<?xml version='1.0'?>n<data>n");            using (var sr = new StreamReader(file))            {                String line;                while ((line = sr.ReadLine()) != null)                {                    if (len > CHUNK_SIZE)                    {                        req.SendNextChunkedResponse(chunk, len);                        len = 0;                    }                    if (sr.EndOfStream) break;                }            }            len = add(chunk, len, "</data>n");            req.SendNextChunkedResponse(chunk, len);            len = 0;            req.SendFinalChunkedResponse();        }        private static int add(byte[] chunk, int len, string msg)        {            for (int i = 0; i < msg.Length; i++)            {                chunk[ptr++] = (byte)msg.ToCharArray(i, 1)[0];            }            return len;        }

 




#30524 Socket error #10055 (WSAENOBUFS)

Posted by Wim Roeling on 11 June 2012 - 01:31 PM in Netduino Plus 2 (and Netduino Plus 1)

Maybe (I did not test this) it helps to check first if the socket it ready to send data:

if (Connection.Poll(-1, SelectMode.SelectWrite)) {
   Connection.Send( yourData );
}



#29428 First chance Socket Exception on socket.Accept()

Posted by Wim Roeling on 20 May 2012 - 08:56 AM in Netduino Plus 2 (and Netduino Plus 1)

I see this piece of code very often (it's in every Google hit on webservers with Netduino). It is easy to solve the problem that you all have by interting a try / catch block within the while (true) loop.
What happens is that for some reason an error may occur. This will stop the listener, it will never recover from it. I had the same problem: sometimes the webserver worked for a day and than it suddenly stopped. Whit the modification above it has worked continuously for many months.

Pattern:
public void listen()
{
   while (true)
   {
        try
        {
             // your listener code here ...
             // .....
             // .....
        }
        catch (Exception)
        {
             // ignore
        }
    }
}
Kind regards,
Wim Roeling
The Netherlands



#25601 Firmware update

Posted by Wim Roeling on 16 March 2012 - 09:06 AM in Netduino Plus 2 (and Netduino Plus 1)

Wim, if it may help, do you live nearby Breda? You're from the NL right? If so, I would love to help out?

Yes, I am from the NL, Vlaardingen.
And as you can see I finally did a successful update to 4.2 after all.



#25600 Firmware update

Posted by Wim Roeling on 16 March 2012 - 09:05 AM in Netduino Plus 2 (and Netduino Plus 1)

Hi Wim,

Very strange. There's a lot of new code in .NET MF 4.2 beta, but we're having pretty good luck around here with RC4.

If you can boil down a repro case to a few dozen lines of code...I'd love to test it and help get to the root of the issue.

Chris

Chris,

I goto it to work yesterday, as you can see from my previous reply. Thanks anyway!



#25577 Firmware update

Posted by Wim Roeling on 15 March 2012 - 10:15 PM in Netduino Plus 2 (and Netduino Plus 1)

Hi Wim,

Remove that assembly from your project references, and then add this one in instead:
C:\Program Files (x86)\Secret Labs\Netduino SDK\Assemblies\v4.2\SecretLabs.NETMF.IO.dll

Chris

I didn't need that assembly after all, so I organized my usings and deleted it.
But still the 4.2 Netduino is refusing to work properly. Very small projects (like a blinking LED and that sort of stuff) work fine, but more complex projects all fail. Even stranger: A complex project that I almost completely stripped while leaving the unused methods in the project also fails. (Size of assemblies 16752 bytes). It doesn't even get into the first instruction in the debugger. I'm really puzzled now. The network seems to be ok (I can ping the device) but that's all the life I can see. My LCD display does nothing, etc..
My other Netduino with firmware 4.1 runs all those same projects OK.

------------------------------------------------------------------------------
FORGET ALL THE ABOVE - It's working fine now!!! I forgot to deploy a sub project to the 4.2 target framework. The compiler does not give a warning for this, like it does for the main project. Thanks for your patience anyway!



#25573 Firmware update

Posted by Wim Roeling on 15 March 2012 - 08:08 PM in Netduino Plus 2 (and Netduino Plus 1)

Hi Wim,

If you remove 4.1 and install 4.2, you can still develop projects in 4.0, 4.1 and 4.2. It has downwards compatibility.

Hopefully that'll answer your question! Please let us know if you need any help.

I removed 4.1 and reinstalled 4.2. The test program (blinking LED) is working properly now on the updated Netduino, but all other apps that I made (needing network support and an LCD display) fail to run.
MFDeploy device info shows the info below now. I'm worried about "SecretLabs.NETMF.IO,4.1.0.0". Still version 4.1 apparently.

DeviceInfo:
HAL build info: 4.2.0.0, Netduino Plus (v4.2.0.0 RC4) by Secret Labs LLC
OEM Product codes (vendor, model, SKU): 255, 255, 65535
Serial Numbers (module, system):
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
Solution Build Info: 4.2.0.0, Netduino Plus (v4.2.0.0 RC4) by Secret Labs LLC
AppDomains:
default, id=1
Assemblies:
mscorlib,4.2.0.0
Microsoft.SPOT.Native,4.2.0.0
Microsoft.SPOT.Hardware,4.2.0.0
Microsoft.SPOT.Net,4.2.0.0
System,4.2.0.0
Microsoft.SPOT.Hardware.SerialPort,4.2.0.0
Microsoft.SPOT.IO,4.2.0.0
System.IO,4.2.0.0
Microsoft.SPOT.Hardware.PWM,4.2.0.0
Microsoft.SPOT.Hardware.Usb,4.2.0.0
SecretLabs.NETMF.Hardware,4.2.0.0
SecretLabs.NETMF.Diagnostics,4.2.0.0
SecretLabs.NETMF.IO,4.1.0.0
Kobush.NETMF.Hardware.LCD,1.0.0.0
SecretLabs.NETMF.Hardware.NetduinoPlus,4.2.0.0
NetduinoPlusApplication4,1.0.0.0



#25545 Firmware update

Posted by Wim Roeling on 15 March 2012 - 01:51 PM in Netduino Plus 2 (and Netduino Plus 1)

Hi Wim,

It looks like you're still deploying some 4.1 assemblies with your application. Did you change the target framework from .NET MF 4.1 to .NET MF 4.2 in the project properties?

You might want to manually replace the few assemblies in your project references which are targeting the old version. That shouldn't happen--but it appears that there's a mismatch somehow.

Did you uninstall the .NET MF 4.1 and Netduino 4.1 SDKs...and install the .NET MF 4.2 QFE1 SDK and Netduino 4.2 Beta SDK?

Chris


I changed the target framework in the project to MF 4.2, since a warning message tells you to do so if it is configured wrongly.

I did not uninstall MF 4.1, nor Netduino 4.1 SDK, because I thought they can live next to each other.
I understand it's better to remove them first and just install 4.2 for both? Can I still switch to 4.1 for the second Netduino I own? It's not realy important, because as soon as 4.2 works I will upgrade the other board aswell.

Thnx in advance,
Wim Roeling



#25523 Firmware update

Posted by Wim Roeling on 14 March 2012 - 09:26 PM in Netduino Plus 2 (and Netduino Plus 1)

Today I updated my Netduino Plus to firmware 4.2.0.0 RC4. First I erased the Netduino and loaded the new bootloader with the SAM-BA 2.11 CDC tool. After that I updated the firmware with MFDeploy. All worked well, as you can see from the device info below. I also updated the MF SDK to version 4.2 and finally I deployed a tiny program to blink the onboard LED. But whatever I try: it just doesn't do anything. I can't even step through the program with the debugger. It just doesn't seem to react on anything. I happen to have two Netduino's (the other one still running version 4.1) and that one run's the same tiny program as expected. What can be wrong with the one that I updated to 4.2 ?? Can anybody help me? DeviceInfo: HAL build info: 4.2.0.0, Netduino Plus (v4.2.0.0 RC4) by Secret Labs LLC OEM Product codes (vendor, model, SKU): 255, 255, 65535 Serial Numbers (module, system): FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF Solution Build Info: 4.2.0.0, Netduino Plus (v4.2.0.0 RC4) by Secret Labs LLC AppDomains: default, id=1 Assemblies: mscorlib,4.2.0.0 Microsoft.SPOT.Native,4.2.0.0 Microsoft.SPOT.Hardware,4.2.0.0 Microsoft.SPOT.Net,4.2.0.0 System,4.2.0.0 Microsoft.SPOT.Hardware.SerialPort,4.2.0.0 Microsoft.SPOT.IO,4.2.0.0 System.IO,4.2.0.0 Microsoft.SPOT.Hardware.PWM,4.2.0.0 Microsoft.SPOT.Hardware.Usb,4.2.0.0 SecretLabs.NETMF.Hardware,4.2.0.0 SecretLabs.NETMF.Diagnostics,4.2.0.0 NetduinoPlusApplication3,1.0.0.0 SecretLabs.NETMF.Hardware.NetduinoPlus,4.1.0.0 SecretLabs.NETMF.Hardware,4.1.0.0



#24739 Slow webserver

Posted by Wim Roeling on 27 February 2012 - 04:42 PM in Netduino Plus 2 (and Netduino Plus 1)

Hi Wim and welcome to the Netduino community.

A micro platform will always have more limitations then PCs. To be honest, I would never even think about making a Netduino a server. My approach for communicating with a Netduino and the web is set up differently:

I've got a webserver, with Linux, Apache, MySQL, PHP. On this server I run a small socket listener with MySQL support as well.
My Netduino connects to the socket and has an open connection all the time with the webserver. All requests are translated to a protocol that only contains data. No markup, no image files, etc. They're all stored on the webserver itself.

That method is actually very fast and makes communication with the netduino instant, without losing much resources on the Netduino.

I know this doesn't answer your question: "how to speed up the webserver", but it can help you with a faster approach.


Currently my socket listener is very beta (if not alpha), so I'm not going to share code in it's current state. But it's a project I'm working on.


Hi Stefan,

I'm not running a complete webserver on a Netduino. It's just a tiny web application to setup an intelligent device. Like you will find a lot today in network printers, routers, etc.. So a LAMP stack (Linux, Apache, MySQL, PHP) is out of the question. The largest amount of data it will ever have to send is between 10 and 20k. That's not a lot, but it takes 10 to 20 seconds. And that's way to long, I think, for a 32Mhz processor. I used to have an 8Mhz Archimedes computer (whith the first ARM processor) in the early eighties which was way faster. I just can't understand where the cycles go to.

Anyway, I'm considering a Raspberry Pi now to do the job. An amazing $25 creditcard sized Linux computer with a 700Mhz ARM and 256MB of RAM.

Regards,

Wim Roeling



#24729 Slow webserver

Posted by Wim Roeling on 27 February 2012 - 11:04 AM in Netduino Plus 2 (and Netduino Plus 1)

Hi, I experience slow download speed from the webserver (I used the small webserver as posted bij Greg Zimmers) when the response length is larger than a few KB. (Example: a 6K response takes about 8 seconds). In earlier posts I saw that a firmware update might solve this problem. Yesterday I upgraded to 4.1.1-B1, but the webserver is still slow. I'm the happy owner of more than one Netduino, and they're all having the same problem. So it can not be a hardware problem. Yesterday I rewrote the webserver so that it now supports chunked transfer. I was hoping that keeping the chunk size low would solve the problem, but it didn't. (The chunked transfer was needed anyway, in order to be able to send data larger than the Netduino's memory size). Can anyone help me to improve the download speed? Wim Roeling




home    hardware    projects    downloads    community    where to buy    contact Copyright © 2016 Wilderness Labs Inc.  |  Legal   |   CC BY-SA
This webpage is licensed under a Creative Commons Attribution-ShareAlike License.