Netduino home hardware projects downloads community

Jump to content


The Netduino forums have been replaced by new forums at community.wildernesslabs.co. This site has been preserved for archival purposes only and the ability to make new accounts or posts has been turned off.

Wim Roeling

Member Since 24 Feb 2011
Offline Last Active Oct 26 2020 11:00 AM
-----

Topics I've Started

"Chunked" webserver to send large amounts of data

30 December 2012 - 01:18 PM

            req.SendFinalChunkedResponse();

Based on work done by others I rewrote the webserver project to enable it to send responses that are larger than Netduino's internal memory. To do this I modified the Request object to support chunked transfer encoding (see http://en.wikipedia....ansfer_encoding ).

In order to send a chunked response, the data should be split in "chunks" of a reasonable size. In almost all cases this is very ease: if you want to send an image that is far bigger than Netduino's memory, just readit in memory from the SD card in chunks of, say, 1024 bytes and send each chunk separately. Thus, Netduino's memory will not use more than 1024 bytes at the time while the complete immage is send.

Before the first chunk a "header"should be send (this will also initialize chunked transfer) and after the last chunk the response had to be finalized.

 

Here is my code of the enhanced Request:

using System;using System.Net.Sockets;using System.Text;using System.Net;using System.Threading;namespace WebSpace{    /// <summary>    /// Holds information about a web request and is able to send chuncked responses    /// </summary>    public class Request : IDisposable    {        private string method;        private string url;        private Socket client;        static char[] hexDigits = { '0', '1', '2', '3', '4', '5', '6', '7','8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};        //convert a byte to a hex string        public static string ToHexString(int i)        {            int b1 = i & 0x0F;            int b2 = (i >> 4) & 0x0F ;            int b3 = (i >> 8) & 0x0F;            int b4 = (i >> 12) & 0x0F;            if (b4 > 0) return "" + hexDigits[b4] + hexDigits[b3] + hexDigits[b2] + hexDigits[b1];            else if (b3 > 0) return "" + hexDigits[b3] + hexDigits[b2] + hexDigits[b1];            else if (b2 > 0) return "" + hexDigits[b2] + hexDigits[b1];            else return "" + hexDigits[b1];        }        internal Request(Socket Client, char[] Data)        {            client = Client;            ProcessRequest(Data);        }        // Request method        public string Method        {            get { return method; }        }        // URL of request        public string URL        {            get { return url; }        }        // Client IP address        public IPAddress Client        {            get            {                IPEndPoint ip = client.RemoteEndPoint as IPEndPoint;                if (ip != null) return ip.Address;                return null;            }        }        // Send a response back to the client        public void SendResponse(string response, string type = "text/html")        {            if (client != null)            {                string header = "HTTP/1.1 200 OKrnContent-Type: " + type + "; charset=utf-8rnContent-Length: " + response.Length.ToString() + "rnConnection: closernrn";                client.Send(Encoding.UTF8.GetBytes(header), header.Length, SocketFlags.None);                client.Send(Encoding.UTF8.GetBytes(response), response.Length, SocketFlags.None);            }        }        public void SendHeaderChunkedResponse(string type = "text/html")        {            if (client != null)            {                string header = "HTTP/1.1 200 OKrnContent-Type: " + type + "; charset=utf-8rnTransfer-Encoding: chunkedrnConnection: closernrn";                client.Send(Encoding.UTF8.GetBytes(header), header.Length, SocketFlags.None);                Thread.Sleep(10) ;            }        }        public void SendNextChunkedResponse(string response)        {            if (client != null)            {                string hex = ToHexString(response.Length);                client.Send(Encoding.UTF8.GetBytes(hex + "rn"), hex.Length+2, SocketFlags.None);                client.Send(Encoding.UTF8.GetBytes(response + "rn"), response.Length+2, SocketFlags.None);                Thread.Sleep(10) ;            }        }        public void SendNextChunkedResponse(byte[] response, int length)        {            if (client != null)            {                string hex = ToHexString(length);                client.Send(Encoding.UTF8.GetBytes(hex + "rn"), hex.Length + 2, SocketFlags.None);                client.Send(response, length, SocketFlags.None);                client.Send(Encoding.UTF8.GetBytes("rn"), 2, SocketFlags.None);                Thread.Sleep(10);            }        }        public void SendFinalChunkedResponse()        {            if (client != null)            {                client.Send(Encoding.UTF8.GetBytes("0rnrn"), 5, SocketFlags.None);                Thread.Sleep(10);            }        }        public void Send404()        {            string header = "HTTP/1.1 404 Not FoundrnContent-Length: 0rnConnection: closernrn";            if (client != null)            {                client.Send(Encoding.UTF8.GetBytes(header), header.Length, SocketFlags.None);            }        }        private void ProcessRequest(char[] data)        {            string content = new string(data);            string firstLine = content.Substring(0, content.IndexOf('n'));            // Parse the first line of the request: "GET /path/ HTTP/1.1"            string[] words = firstLine.Split(' ');            method = words[0];            url = words[1];            // Could look for any further headers in other lines of the request if required (e.g. User-Agent, Cookie)        }        #region IDisposable Members        public void Dispose()        {            if (client != null)            {                client.Close();                client = null;            }        }        #endregion    }}

 

Here is the code of the Listener:

using System;using System.Net.Sockets;using System.Net;using System.Text;using System.Threading;namespace WebSpace{    public delegate void RequestReceivedDelegate(Request request);    public class Listener : IDisposable    {        const int maxRequestSize = 1024;        readonly int portNumber = 80;        private Socket listeningSocket = null;        private RequestReceivedDelegate requestReceived;        public Listener(RequestReceivedDelegate RequestReceived)            : this(RequestReceived, 80) { }        public Listener(RequestReceivedDelegate RequestReceived, int PortNumber)        {            portNumber = PortNumber;            requestReceived = RequestReceived;            listeningSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);            listeningSocket.Bind(new IPEndPoint(IPAddress.Any, portNumber));            listeningSocket.Listen(10);            new Thread(StartListening).Start();        }        ~Listener()        {            Dispose();        }        public void StartListening()        {            while (true)            {                try                {                    using (Socket clientSocket = listeningSocket.Accept())                    {                        IPEndPoint clientIP = clientSocket.RemoteEndPoint as IPEndPoint;                        // Debug.Print("Received request from " + clientIP.ToString());                        var x = clientSocket.RemoteEndPoint;                        int availableBytes = clientSocket.Available;                        // Debug.Print(DateTime.Now.ToString() + " " + availableBytes.ToString() + " request bytes available");                        int bytesReceived = (availableBytes > maxRequestSize ? maxRequestSize : availableBytes);                        if (bytesReceived > 0)                        {                            byte[] buffer = new byte[bytesReceived]; // Buffer probably should be larger than this.                            int readByteCount = clientSocket.Receive(buffer, bytesReceived, SocketFlags.None);                            using (Request r = new Request(clientSocket, Encoding.UTF8.GetChars(buffer)))                            {                                // Debug.Print(DateTime.Now.ToString() + " " + r.URL);                                if (requestReceived != null) requestReceived(r);                            }                        }                    }                }                catch (Exception e)                {                    Thread.Sleep(10);                    Program.Log("!!! ERROR !!! " + e.Message + "n" + e.StackTrace);                }            }        }        #region IDisposable Members        public void Dispose()        {            if (listeningSocket != null) listeningSocket.Close();        }        #endregion    }}

And finaly, here is some example code to send a large XML stream. Note the following three calls:

            req.SendHeaderChunkedResponse("text/xml");
                        req.SendNextChunkedResponse(chunk, length);
            req.SendFinalChunkedResponse();

An example to send the contents of a large files in simple XML as <data> ..... </data>

        private const int CHUNK_SIZE = 4096;        private static void List(Request req, string file)        {            byte[] chunk = new byte[CHUNK_SIZE + 256];            int len = 0;            req.SendHeaderChunkedResponse("text/xml");            len = add(chunk, len, "<?xml version='1.0'?>n<data>n");            using (var sr = new StreamReader(file))            {                String line;                while ((line = sr.ReadLine()) != null)                {                    if (len > CHUNK_SIZE)                    {                        req.SendNextChunkedResponse(chunk, len);                        len = 0;                    }                    if (sr.EndOfStream) break;                }            }            len = add(chunk, len, "</data>n");            req.SendNextChunkedResponse(chunk, len);            len = 0;            req.SendFinalChunkedResponse();        }        private static int add(byte[] chunk, int len, string msg)        {            for (int i = 0; i < msg.Length; i++)            {                chunk[ptr++] = (byte)msg.ToCharArray(i, 1)[0];            }            return len;        }

 


Firmware update

14 March 2012 - 09:26 PM

Today I updated my Netduino Plus to firmware 4.2.0.0 RC4. First I erased the Netduino and loaded the new bootloader with the SAM-BA 2.11 CDC tool. After that I updated the firmware with MFDeploy. All worked well, as you can see from the device info below. I also updated the MF SDK to version 4.2 and finally I deployed a tiny program to blink the onboard LED. But whatever I try: it just doesn't do anything. I can't even step through the program with the debugger. It just doesn't seem to react on anything. I happen to have two Netduino's (the other one still running version 4.1) and that one run's the same tiny program as expected. What can be wrong with the one that I updated to 4.2 ?? Can anybody help me? DeviceInfo: HAL build info: 4.2.0.0, Netduino Plus (v4.2.0.0 RC4) by Secret Labs LLC OEM Product codes (vendor, model, SKU): 255, 255, 65535 Serial Numbers (module, system): FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF Solution Build Info: 4.2.0.0, Netduino Plus (v4.2.0.0 RC4) by Secret Labs LLC AppDomains: default, id=1 Assemblies: mscorlib,4.2.0.0 Microsoft.SPOT.Native,4.2.0.0 Microsoft.SPOT.Hardware,4.2.0.0 Microsoft.SPOT.Net,4.2.0.0 System,4.2.0.0 Microsoft.SPOT.Hardware.SerialPort,4.2.0.0 Microsoft.SPOT.IO,4.2.0.0 System.IO,4.2.0.0 Microsoft.SPOT.Hardware.PWM,4.2.0.0 Microsoft.SPOT.Hardware.Usb,4.2.0.0 SecretLabs.NETMF.Hardware,4.2.0.0 SecretLabs.NETMF.Diagnostics,4.2.0.0 NetduinoPlusApplication3,1.0.0.0 SecretLabs.NETMF.Hardware.NetduinoPlus,4.1.0.0 SecretLabs.NETMF.Hardware,4.1.0.0

Slow webserver

27 February 2012 - 11:04 AM

Hi, I experience slow download speed from the webserver (I used the small webserver as posted bij Greg Zimmers) when the response length is larger than a few KB. (Example: a 6K response takes about 8 seconds). In earlier posts I saw that a firmware update might solve this problem. Yesterday I upgraded to 4.1.1-B1, but the webserver is still slow. I'm the happy owner of more than one Netduino, and they're all having the same problem. So it can not be a hardware problem. Yesterday I rewrote the webserver so that it now supports chunked transfer. I was hoping that keeping the chunk size low would solve the problem, but it didn't. (The chunked transfer was needed anyway, in order to be able to send data larger than the Netduino's memory size). Can anyone help me to improve the download speed? Wim Roeling

home    hardware    projects    downloads    community    where to buy    contact Copyright © 2016 Wilderness Labs Inc.  |  Legal   |   CC BY-SA
This webpage is licensed under a Creative Commons Attribution-ShareAlike License.