38 replies to this topic

[Quiche31]

Greetings

This project uses a NetduinoPlus to post sensor data to Pachube. Here are the steps that I followed:

• Create a pachube account (it's free)
• From http://www.pachube.com/feeds/new, create a feed and associated datastreams; example: TemperatureLivingRoom, TemperatureRoom1, TemperatureRoom2
• Keep note of your feed ID and your API Key (for the key: select "my settings" in the banner)
• Make sure that you have a SD Card in the slot of your NetduinoPlus
• Build the attached program, deploy on your NetduinoPlus, and run
• In your browser, invoke the web-server running on your NetduinoPlus, using its IP
• You'll be sent to the settings page: enter your Pachube API Key and feed ID, and click Save, saving to the SD, to SD\pachube1\pachube1.txt
• Click on Sensors, and enter the three temperatures, and click "Send"
• The entered temperatures will be saved to the SD, then are sent to Pachube
• You are then redirected to Pachube, with a view on the feed that you just updated

Apart from Pachube http-post API, this project demonstrates how the driver supports serving html forms (handling http post), url redirections, and includes a storage abstraction for the SD card

The two forms where you configure your Pachube settings, and submit data to Pachube:

The Pachube screen that you should get, after clicking "Send":


However the key element of this program is how to post data to Pachube. The gotchas: I needed to indicate host and user-agent in the http header, otherwise I got an API error report from Pachube:

    protected void submitSensorData(String ApiKey, String ID, String []values)
    {
        StringBuilder sb = new StringBuilder(20);
        StringBuilder data = new StringBuilder(210);
        foreach (String value in values)
        {           
            data.Append(value);
            data.Append(',');
        }
        String result = data.ToString();
        sb.Append("POST /api/feeds/" + ID + ".csv?_method=put HTTP/1.1\n");
        sb.Append("Accept: */*\n");
        sb.Append("Content-Length: " + result.Length + "\n");
        sb.Append("Host: www.pachube.com\n");
        sb.Append("User-Agent: NetduinoPlus\n");
        sb.Append("X-PachubeApiKey: " + ApiKey + "\n\n" + result + "\n");
        HttpResponse response = webServer.SendRequest("www.pachube.com", 80, sb.ToString());        
        Debug.Print(response.Contents);
    }

JP

Attached Files

•  Pachube1.zip   42.76KB   224 downloads

[Chris Walker]

Nice. Maybe you can detect when people are in or leave a room by the temperature variations?

[Quiche31]

Nice. Maybe you can detect when people are in or leave a room by the temperature variations?

Probably not with a DS1820 (unless the intruder lits his cigarette with a flamethrower), but that will work with a MIR motion detector. However the initial idea of using Pachube was more to monitor an aquarium (hence the temperature) than doing intrusion detection

[Innovactive]

Nice job! I think add digital input processing will be simple. What about processing pachube feed triggers to let Netduino drive some digital output? With actuators remote activation your solution will be perfect!

[Quiche31]

What about processing pachube feed triggers to let Netduino drive some digital output?
With actuators remote activation your solution will be perfect!

This is technically possible already, as my NetduinoPlus WebServer can be reached from the Internet (as in the HoneyPot project), and it can process http post (as it does in this project with the html form processing). The issue with Pachube feeds is that I don't see any security model: my understanding is that you provide a URL to Pachube that will invoke it (with http POST) whenever the event condition is reached. But how can my Netduino trust that the http call comes from the trigger that I had configured, and is entitled to perform the appropriate action?
There is a lack for a trust model, and I'd suppose that Pachube did not invest in some server-side authentication (whether PKI-based or otherwise), unless I missed it...

Wouldn't it be nice if Servers in the Cloud such as Pachube supported open-ssl, with the minimum set of associated services, and that could issue certificates to authenticate either endpoints (the triggers that user configure and the devices that update feeds or receive trigger events)?

[zerov]

Are there any other similar services like Pachube? It's a fantastic idea, but it's a bit to expensive. If I want to monitor my whole house and keep a infinite history, $99 is to much for me.

[Quiche31]

Are there any other similar services like Pachube? It's a fantastic idea, but it's a bit to expensive. If I want to monitor my whole house and keep a infinite history, $99 is to much for me.

What would prevent you from monitoring your house using a webserver that sits in front of your sensors? What does it give you to push sensors data to Pachube? By nature some sensors information are to be shared (examples: meteo, road traffic), but light condition or room temperature are not to be shared isn't it? I know that my project involves temperature precisely, but this is just a proof of concept, not what I'll be doing. I'll end up with just the webserver running in my NetduinoPlus (in fact the target will run on a Mini).

[zerov]

What would prevent you from monitoring your house using a webserver that sits in front of your sensors? What does it give you to push sensors data to Pachube? By nature some sensors information are to be shared (examples: meteo, road traffic), but light condition or room temperature are not to be shared isn't it? I know that my project involves temperature precisely, but this is just a proof of concept, not what I'll be doing. I'll end up with just the webserver running in my NetduinoPlus (in fact the target will run on a Mini).

I though I could use Pachube for long term data logging.

[Quiche31]

I though I could use Pachube for long term data logging.

2GB is a lots of sensor data that you can store on the SD. I needed to have an idea for myself: 48 bytes of data record (8 bytes of timestamp, 40 bytes of data for 10 sensors) every minute, 24x7, will take you 728 days to fill the entire SD. It wil be less if accounting for the overhead of one file a day, but still you have near to 2 years of data for much less than the $99 yearly fee on Pachube that you quoted, unless you want disaster recovery too

[jdsmith]

Greetings

This project uses a NetduinoPlus to post sensor data to Pachube. Here are the steps that I followed:

Thanks for posting the updates to your HTTP lib. Have you ever seen an issue with the call to System.Net.Dns.GetHostEntry()? It's throwing a "An unhandled exception of type 'System.NotSupportedException' occurred in System.dll" exception on my setup (I'm using WiFly and the straight-up Netduino board; Micro Framework v4.1). The MSDN docs say that the Dns class doesn't support IPv6. I don't think I'm running IPv6, but I can't figure out what else would be causing the exception.

Thanks, again, for the excellent samples/code!

[Chris Walker]

Hi jdsmith, The WiFly shield doesn't have integrated System.Net support (at least not yet). You'll want to look at how other community members have been using this shield in their applications. Chris

[jdsmith]

The WiFly shield doesn't have integrated System.Net support (at least not yet). You'll want to look at how other community members have been using this shield in their applications.

Bummer (and thanks). I'm trying to make POST requests to via a wireless board + Netduino. I'll hunt around the forums and keep hoping that the WiFly board supports it someday.

[Quiche31]

Bummer (and thanks). I'm trying to make POST requests to via a wireless board + Netduino. I'll hunt around the forums and keep hoping that the WiFly board supports it someday.

I am stuck on the POST as well. I'll investigate more and wil report what I've got.
Don't hesitate to let us know if you find how to do it after browsing the forums
At least I can do a GET, example below:

public static void Main()
    {
        HttpWiflyImpl wifly = new HttpWiflyImpl(null, 80, HttpWiflyImpl.DeviceType.crystal_14_MHz, SPI.SPI_module.SPI1,
        SecretLabs.NETMF.Hardware.NetduinoPlus.Pins.GPIO_PIN_D10);
        wifly.Open();
        wifly.SendCommand("set com remote GET$/");
        wifly.SendCommand("set option format 1");
        wifly.SendCommand("set ip proto 18");
        wifly.SendCommand("open www.secretlabs.com 80\n\n", "AOK", "ERR", 0, 3000);
    }

[jdsmith]

I am stuck on the POST as well. I'll investigate more and wil report what I've got.
Don't hesitate to let us know if you find how to do it after browsing the forums

I'll absolutely report back if I figure out POST. I'm also wondering if I'm running into something like a threading or blocking issue when trying to use HTTPRequest and using your HTTPLib sample...was thinking that perhaps the board is in server/listen mode while I'm trying to send requests back out. I'll give that a whirl, but based on Chris's comments regarding the board's (dis)ability to fully support System.NET, I'm pessimistic.

[Innovactive]

Hi. We have tried your beautiful astra.http class lib together with your Pachube sample, updating some datastreams every 10 seconds with sensor data coming from a DangerShield we often use to do prototyping, but after a dozen of successful updates, the http sending method starts to give Socket Exceptions we cannot recover from. We have to say that we had also to reduce stringbuilder's ctor specified size since before getting those exceptions we had frequent OutOfMemoryExceptions. We used NetDuinoPlus with several firmware updates, but without success. Could you please try to slightly modify your sample to do periodic (and unattended) updates, even with random data, just to be sure that astra.http and NetDuinoPlus tcp stack are able to run indefinitely? BTW, is that any way to do "PUT" requests instead of GET and POST ones? Thanks a lot!

[Quiche31]

Could you please try to slightly modify your sample to do periodic (and unattended) updates, even with random data, just to be sure that astra.http and NetDuinoPlus tcp stack are able to run indefinitely?
BTW, is that any way to do "PUT" requests instead of GET and POST ones?

How did you test it exactly? Did you use the sample as-it, by clicking on the "Send" button of the html form, or did you modify the sample to automatically read the sensors values and call submitSensorData() every 10 s? I hope the later...
Yes I'll take some time to see where the problem is. Thanlk you for field testing it!
I'll look into the PUT as well.

JP

[Innovactive]

We added a 10 seconds periodic thread that calls your submitSensorData().

At this time, we have to call PowerState.Reboot() every 60 seconds to let it work continuously.

BTW, we modified submitSensorData() method body to use PUT and v2 API as follows:

[...]

for (int i = 0; i < values.Length; i++)
        {
            sbData.Append(i.ToString());
            sbData.Append(',');
            sbData.Append(values[i]);
            if (i != values.Length - 1) sbData.Append('\n');
        }

        String result = sbData.ToString();

        sbRequest.Append("PUT http://api.pachube.com/v2/feeds/" + ID + ".csv HTTP/1.1\n");
        sbRequest.Append("Content-Length: " + result.Length + "\n");
        sbRequest.Append("Host: api.pachube.com\n");
        sbRequest.Append("X-PachubeApiKey: " + ApiKey + "\n\n" + result + "\n");

[...]

[Quiche31]

We added a 10 seconds periodic thread that calls your submitSensorData().
At this time, we have to call PowerState.Reboot() every 60 seconds to let it work continuously

Add a garbage collection within the loop, and this will last forever.

        while (true)
        {
            submitSensorData(apiKey, id, new string[] { t0.ToString(), t1.ToString(), t2.ToString() });
            t0 = getNextValue(t0);
            t1 = getNextValue(t1);
            t2 = getNextValue(t2);
            Thread.Sleep(10000);
            Debug.GC(true); // Insert this!
        }
}

This is what I've got after a night running (and on):

Attached Files

•  Program.cs   1.91KB   45 downloads

[jdsmith]

A regular garbage collection within the loop, and it will last much longer. My test below has been running for 40' already and is still running. I also re-used the two StringBuilder's over and over, as shown below:

I've run a version of the astra http code for 24+ hours albeit with a much lower GET/POST frequency ~once every 45 seconds. I do see some hiccups - none that require a power restart - but I feel like the issue is on the network card side (I'm using a WiFly shield) more than the astra code. Specifically, I can't find a way to predictably flush the WiFly's UART data buffer. Any ideas? The WiFly user guide talks about timed flushes, key character flushes, etc. Curious if there's a more definitive "flush" command for the board or some other call into the register?

Looks like you got POSTS working!

[Quiche31]

Looks like you got POSTS working!

No POST yet, the version above for Innocactive uses the socket version. On the flush issue, have you changed the UART rate? I have not noticed flush issues, aasuming that you have ssen incomplete/garbled outgoing traffic. You may also try to upgrade the firmware, for example, by uncommenting the two SendCommand() below (the second one displaying everything, including your firmware version):

public void Open()
        {
            if (!m_opened)
            {
                Init();
                Thread.Sleep(200);
                enterCommandMode();
                SendCommand("reboot", "Listen on", "ERR:", 0, 500);
                enterCommandMode();
                Send("\r");
                //SendCommand("ftp u wifly-221G.img");
                //SendCommand("get e");
                m_opened = true;
            }
        }