Netduino home hardware projects downloads community

Jump to content


The Netduino forums have been replaced by new forums at community.wildernesslabs.co. This site has been preserved for archival purposes only and the ability to make new accounts or posts has been turned off.
Photo

Memory considerations


  • Please log in to reply
9 replies to this topic

#1 Szymon

Szymon

    Advanced Member

  • Members
  • PipPipPip
  • 108 posts
  • LocationPoland, Krakow

Posted 28 September 2010 - 06:16 PM

Hi,
I'm working on a larger project for Netduino Plus. I was hoping to use System.Http.dll but it turns out that I'm running out of memory when I try to deploy the whole project. So I started looking how the memory is allocated.

First here are the assemblies that are included wiht the Netduino Plus firmware:
mscorlib (4.1.2821.0) (3880 RAM - 33236 ROM - 19134 METADATA)
Microsoft.SPOT.Native (4.1.2821.0) (1144 RAM - 6516 ROM - 4479 METADATA)
Microsoft.SPOT.Hardware (4.1.2821.0) (1752 RAM - 11440 ROM - 7371 METADATA)
Microsoft.SPOT.Net (4.1.2821.0) (704 RAM - 5060 ROM - 2452 METADATA)
System (4.1.2821.0) (872 RAM - 5992 ROM - 3206 METADATA
Microsoft.SPOT.IO (4.1.2821.0) (740 RAM - 4620 ROM - 2522 METADATA)
System.IO (4.1.2821.0) (1548 RAM - 13292 ROM - 5862 METADATA)
Microsoft.SPOT.Hardware.SerialPort (4.1.2821.0) (508 RAM - 3440 ROM - 1527 METADATA)
Microsoft.SPOT.Hardware.Usb (4.1.2821.0) (580 RAM - 3740 ROM - 1844 METADATA)
SecretLabs.NETMF.Hardware (4.1.0.0) (256 RAM - 1108 ROM - 491 METADATA)
Total: (10320 RAM - 88444 ROM - 48888 METADATA)

As you can see in total it consumes about 88KB of RAM and I think most of these libraries are essential for any project. Well maybe apart from Microsoft.SPOT.Hardware.Usb - I guess most of network devices won't need ot connect as USB client. But it gains us only about 4KB.

When I add System.Http to blank project I can deploy it but it leaves very little ROM left. Here are the additional assemblies:
Microsoft.SPOT.Net.Security (4.1.2821.0) (264 RAM - 1220 ROM - 606 METADATA)
System.Net.Security (4.1.2821.0) (664 RAM - 4544 ROM - 1943 METADATA)
System.Http (4.1.2821.0) (2976 RAM - 38160 ROM - 12831 METADATA)

So in total it will add almost 44KB ROM. Of this about 6KB is used for the security assemblies. I guess this is requried to support HTTPS. I don't need it for my project at the moment so it would be cool to turn it off. (Again, I wish .NET MF had a post compile linker like MonoTouch to strip the code that is not executed in the project).

According to the tech spec Netduino Plus has only 48KB ROM left for our code. So with System.Http it leaves us with only about 4KB. It's really not enough for anything beyond a most basic project. For example for my project I estimate at least 15KB be needed.

So for now System.Http is no go, and I will try to implement basic HTTP functions on my own. But I hope there is a way to reduce the size of these libraries somehow so we get at least 15-20KB ROM more.

#2 Chris Walker

Chris Walker

    Secret Labs Staff

  • Moderators
  • 7767 posts
  • LocationNew York, NY

Posted 28 September 2010 - 06:38 PM

Hi Szymon, Good observations. We've actually reserved 16KB of space for a future "pre-bootloader." We may be able to free up 8KB-16KB of that (and there may be ways to reduce the size of lwIP a bit--but no promises there). Finally, uIP would leave more space for the HTTP objects. And since the HTTP code is largely/all managed code, you might be able to just pull in the parts you wanted from the .NET MF PK and make a "smaller" version. Chris

#3 phantomtypist

phantomtypist

    Advanced Member

  • Members
  • PipPipPip
  • 142 posts
  • LocationNew York, NY

Posted 28 September 2010 - 07:33 PM

Hi Syzmon,

Building off what Chris said, you might want to compile your own HTTP library. I highly suggest you use code from the .NET MF Toolkit (http://mftoolkit.codeplex.com/). Michael Schwarz is the creator of that project. The .NET MF Toolkit contains a library for HTTP. Download the latest changeset from the source code, change the project references from .NET MF 3.0 to version .NET MF 4.1, and lastly compile. You will be using the dll's built from the MicroHttp project.

You can easily remove classes and what not from the project that you won't use to reduce the amount of space the assemblies take up.

#4 Szymon

Szymon

    Advanced Member

  • Members
  • PipPipPip
  • 108 posts
  • LocationPoland, Krakow

Posted 29 September 2010 - 05:48 AM

Hi Syzmon,

Building off what Chris said, you might want to compile your own HTTP library. I highly suggest you use code from the .NET MF Toolkit (http://mftoolkit.codeplex.com/). Michael Schwarz is the creator of that project. The .NET MF Toolkit contains a library for HTTP. Download the latest changeset from the source code, change the project references from .NET MF 3.0 to version .NET MF 4.1, and lastly compile. You will be using the dll's built from the MicroHttp project.

You can easily remove classes and what not from the project that you won't use to reduce the amount of space the assemblies take up.


Hi,
I already started triming down the MicroHTTP project. Initially it consumed 37KB because it also used the security libraries, but after commenting out SSL support and removing few other classes I got it down to 26K.

However this implementation is targeting scenario where .NET MF device acts as HTTP server (accept requests and sends responses). For current project I need it to act as a client and connect to remote HTTP server. So I was planing to add these functions to the library.

Chris mentionsed another way - to take the existing System.Http.dll appart and try to trim it down (I forgot it has Appache license and we can do this). If I can get it down to 30KB it should be fine.

-Szymon

#5 Charles

Charles

    Advanced Member

  • Members
  • PipPipPip
  • 192 posts

Posted 09 October 2010 - 02:43 AM

Hi,
I already started triming down the MicroHTTP project. Initially it consumed 37KB because it also used the security libraries, but after commenting out SSL support and removing few other classes I got it down to 26K.

However this implementation is targeting scenario where .NET MF device acts as HTTP server (accept requests and sends responses). For current project I need it to act as a client and connect to remote HTTP server. So I was planing to add these functions to the library.

Chris mentionsed another way - to take the existing System.Http.dll appart and try to trim it down (I forgot it has Appache license and we can do this). If I can get it down to 30KB it should be fine.

-Szymon


Parsing HTTP is no small task... I'm sure it will carry a heavy processing burden for the little N+... What exactly is it you are trying to accomplish? Are you looking to have the N+ poll a server periodically for commands? Verify the server is up?

#6 Chris Seto

Chris Seto

    Advanced Member

  • Members
  • PipPipPip
  • 405 posts

Posted 09 October 2010 - 03:08 AM

Parsing HTTP is no small task... I'm sure it will carry a heavy processing burden for the little N+... What exactly is it you are trying to accomplish? Are you looking to have the N+ poll a server periodically for commands? Verify the server is up?

Parsing HTTP requests is actually quite easy. I can do it in a matter of a few lines of code. All you need to do is break the header off, then split it to get the individual headers and split them.

#7 Charles

Charles

    Advanced Member

  • Members
  • PipPipPip
  • 192 posts

Posted 09 October 2010 - 03:19 AM

Parsing HTTP requests is actually quite easy. I can do it in a matter of a few lines of code. All you need to do is break the header off, then split it to get the individual headers and split them.


The code is easy, but the processor overhead for splitting along with all the other pattern matching that would be involved with parsing some sites is really large.... Think about it - To split on newlines, the processor has to look at every single character in the string and compare it to newline to see if it is a match or not.

And it gets even more complicated if parsing has to happen on multi-character blocks like tags. The actual code we would write is quite simple yes, but the processor still has to do the work.

#8 Charles

Charles

    Advanced Member

  • Members
  • PipPipPip
  • 192 posts

Posted 09 October 2010 - 03:32 AM

To put it into prospective, suppose you wanted to simply examine CNN.com and get the current temperature from the bottom of the page. You can't because of the active nature of the site, but for this example let's presume it was a static page that we can download and parse. First off, the page itself will consume 99K of RAM to store as a string. This in itself is probably impossible with the framework loaded on the Netduino's processor. Even if it could load it into memory for processing, searching all the way to the bottom of the page for that number would require over 98,000 operations on the processor, since it has to examine each byte of the page in succession looking for a proper match. Even if all this could occur at one operation per clock tick, which it can't, it would take over 2 ms for the processor to do that... An eternity in computer time. P.S. - My numbers are probably off here - Could someone with a more in depth knowledge of the system step in here and fill in the gaps?

#9 Quiche31

Quiche31

    Advanced Member

  • Members
  • PipPipPip
  • 87 posts
  • LocationFrance

Posted 19 November 2010 - 12:16 PM

Suppose you wanted to simply examine CNN.com and get the current temperature from the bottom of the page. You can't because of the active nature of the site, but for this example let's presume it was a static page that we can download and parse.
First off, the page itself will consume 99K of RAM to store as a string. This in itself is probably impossible with the framework loaded on the Netduino's processor


I know that this is just an example, but why not make use of some RESTful service available up-there. For example, to get the current temperature in e.g. Toulouse (in France), where I live, request for "http://www.google.co...ather=Toulouse", and you have less than 2K to parse. Moreover if the API supports fine-grained access (I don't know if Google's does) you could even get the data you request direct with minimum parse. Additionally you do not depend on how CNN presents weather etc information to end-users. A track worth exploring I think...

JP

#10 smarcus3

smarcus3

    Advanced Member

  • Members
  • PipPipPip
  • 134 posts

Posted 09 April 2012 - 04:24 AM

I need to use ssl to get a pop message count from gmail. However this library mentioned above doens't seem to work. Help.
Steve


My Other Hobby: Engineer Turned Baker




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

home    hardware    projects    downloads    community    where to buy    contact Copyright © 2016 Wilderness Labs Inc.  |  Legal   |   CC BY-SA
This webpage is licensed under a Creative Commons Attribution-ShareAlike License.