The Netduino forums have been replaced by new forums at community.wildernesslabs.co.
This site has been preserved for archival purposes only
and the ability to make new accounts or posts has been turned off.
I am looking for some GOOD library to communicate with a GSM Modem (SIM 900 TTL UART)
Unfortunately all code examples including netduino2seedgsm libraries are simply dumping the AT commands sequentially to showcase SMS/Get/Post examples. Even the modem manuals have the similar examples.
While these examples work as demonstrations, but in real time operations they fail because there is hardly any checking for returned response codes, timing, prompts etc.
Is there any qualitative library around which can be used. (I don't mind if you point out any which is on some other platform or language, I can do the porting for the community ) but it should cover all standard SMS and GPRS functional needs.
I did a SIM900 interface recently, but it was contract work, so I can't post source right now without explicit permission from my client (I did talk to them already about posting non-core-ip source, and they were receptive, but I haven't made actually doing it a priority yet since I'm still pretty busy in pilot). If you want to forge ahead in advance of the coming of that day, here at least are my experiences with it. Shortest story: SIM900 works, but is quite quirky!
OK, first, let me say that I was focused on the TCP client feature, and I did not do SMS. My application supports wired ethernet (on the NP2), and also TCP/IP connectivity over the SIM900 module, (and some other communications channels I won't mention since they are off topic). Truthfully, I think SMS is easier than TCPIP since it's oriented towards short, atomic, messages, rather than a stream, but much of what follows will still be relevant.
The SIM900 has an /embedded/ TCP/IP stack that exposes connection functionality via ancient-style Hayes modem-y AT commands. You don't get to do tcpip proper, but rather you squirt chunklets of data back and forth via AT commands. This has some challenges:
since you are mixing channel data with control data (e.g. the AT commands themselves, and error responses), you've got to demultiplex that. There is a 'transparent mode' that works a little like an old-school modem, switching inline between control and data, but you still have to look out for and use the +++ escape to get into and out of command mode. Blick! I chose to use 'non-transparent' mode, where the delineation is unambiguous, but you've got more parsing to do. Also, if you are going to support multiple client connetions, or a single server connection, you will need this mode. I would suggest coding against non-transparent unless you are really sure you will never need anything else, or if you are supporting a dialup modem as your fallback where this mode would be a natural analog.
there is a single-connect and multi-connect mode. I implemented multi-connect because, why not? Single connect only gets you limitations as far as I can tell, and it doesn't really simplify coding. And you have to use multi-connect if you ever want to do 'server'.
it's not really documented, but the SIM900 is an idle-request design; i.e., you must wait until you have received some acknowledgement to an issued command before issuing another one. Don't pour out a sequence of commands then eat the various responses. Wait for each and every command to complete with a response; and, if you time out getting a positive/negative acknowledgement, you might be best off just throwing and letting some recovery logic get things back into a know state. Oh!...
the responses to all the standard GSM AT commands are minimal, clear, and parseable with a handcoded parser. The reponses to all the simcom extensions, and certainly to all the TCPIP stuff, are not so much so. Horror! However I had an early epiphany in that the Netduino supports regular expressions. You'll take a 30k or so hit on your app's image size, but parsing will actually become the pleasant part. I was concerned about RAM usage, and so I did not pre-compile the regexes, and rather did that each time needed. I was concerned about speed, but this is actually quite quick relative to the 19.2 data rate.
there are so many error codes/scenarios that you will be happiest if you define an exception that you can throw for undefined/unknown/unhandled scenarios. That way the calling module can catch it and at least recycle to recover. My servicing thread looks like vaguely this:
while ( keepOnKeepinOn ){ try { initModemStuff(); //setting all the registers/options/whatnot doAllModemyStuffs(); //actually pumping data teardownModemStuff(); //down, you go! } catch ( ModemBooBoo e ) { log ( e ); //Maybe Sleep() a bit before recycling around }}
?
(^^ sorry, can't make this look right. looks correct while editing; at least its super short)
In this way you can have something functional, yet robust against errors, right away, and then you refine the error handling as a separate task (e.g. maybe you don't need to throw, but can adapt to the condition).
the AT interface to TCPIP is so peculiar that I created an abstraction that mostly looks like a socket. This way I was able to strap on my protocol handler code on top of a real socket, or on top of a connection provided by the SIM900 with minimal effort. You'll need to implement internal buffers for each channel (the module supports up to 8 client connections, or 7 client + 1 server).
you send channel data by telling the modem that you want to do so, and it will present you with a prompt (which they call a 'promoting mark' haha). Then you emit your data. Then the modem returns automatically to command mode and you carry on. This has some caveats:
they mention using Ctrl-Z to terminate the data. There is no escaping mechanism, so if you happen to need to send an 0x26 yourself, you are hosed. I designed my protocol to be textual (against my better wishes) because of this. However, I made a booboo. I failed to notice that the AT+CIPSEND= can take an optional 'length' parameter, in which case you /don't/ send the Ctrl-Z to terminate -- the modem will simply know you are done, and automatically return to command mode. So now you actually can send binary with impunity!
the modem will asynchronously notify you of receive data. Consequently, you must be prepared to receive it at all times and buffer it for your client's subsequent recv(). You must be prepared to throw away any overrun data.
the send channel has a limit on how much you can write at any given time. This is like an MTU, and maybe it event is derived from the MTU because it is not fixed. It depends on which network you connected to. You must be prepared to take client send() calls and chunk the data up into multiple AT+CIPSEND= subject to this limit. In my case, I keep an array of these per-channel limit values, which I query and setup as part of my connect() process. That way they are always up-to-date when I need to do a send().
because of all these things, I structured my SIM900 interface module somewhat like this:
receive data event handler (more on this later)
incoming data buffer, used for building textual lines. This is a fixed 128 bytes which is more than long enough for any text lines that the modem produces (channel data doesn't go through this)
an array of received text lines; an event is signaled when new ones are added. I limit this to 20 max, which really is way more than enough for all the commands the modem supports.
an array of channel info, including:
an event for 'received'
a receive data buffer
a 'max send len' value
state machine with states of:
Normal -- making text lines, for command mode
PromotingMarkWaitGreaterThan -- first waiting state for the signal to send channel data
PromotingMarkWaitSpace -- second waiting state for the signal to send channel data
PromotingMarkPendingNormal -- we were waiting for a promoting mark, but are receiving an asynchronous text line (like a status change) before the promoting mark came, and so need to switch back to PromotingMarkWaitGreaterThan when it is finished
NormalReceive -- receiving channel data; put in channel buffer instead of using to build text lines
the receive data event handler notionally drives the state machine, taking the raw data and either building text lines (which will be interpreted as responses or asynchronous notifications), or buffering into the relevant receive channel, or the special case of noticing that the promoting mark has been received. It signals an event when these things happen, on which the client thread can synchronize to pace it's activities. For what it's worth, my code has 10 events -- one for each of 8 possible data channels, 1 for 'text line received' and one for 'promoting mark found'.
I originally implemented using a dotNet SerialPort.DataReceived event for the COM port. This works for a little while, then the event stops getting fired. I don't know what this is, and others have complained about it. I think it's a bug in the firmware. I was distraught at the propect of redesigning until I hit upon this idea:
worker thread
while ! should end
if valid com and com is open and 0 != BytesToRead
read into buffer, signal 'come and get it' event
else sleep for a time guaranteed not to cause an overrun, which in my case for 19.2 and a 256 byte buffer in the firmware was 100ms
This scheme functionally resembles the DataReceived approach, so I was able to shim it in without disrupting the rest of the design. In fact, since it is a fundamental problem with the com port, and because I use three com ports, I factored this out, and made a virtual method for the 'DataReceived'. Then it really looked a lot like the dotnet mechanism.
The rest of the module consisted of methods that wrapped the AT command dialogs that I was interested in exposing. They all followed a general form:
lock() //the modem has an idle-request design, and an issued AT command should be fully processed before issuing another
generate the AT command text from parameters
send the command, collect received text lines subject to various conditions such as count, timeout, and where the response is expected to be found in the sequence
inspect the result; perhaps parsing out return data. As mentioned, I use regex's for the non-trivial ones, but you can certainly hand parse if you can't or don't want to use the regex assembly
OK fun stuff, the SIM900 is not consistent across commands as to where things like 'OK' are emitted. Most commands response by sending info, and then finish with an 'OK'. So you can pull text lines until you hit 'OK' or an error indicator for most things. But not all! Some (particularly the SimCom extensions) return 'OK' and /then/ return the requested data, so you have to just know how many lines that will be on a command-by-command basis. Because of this, I have a couple helper methods like 'sendATCommandGetResponse' and 'sendATCommandGetNLines' so I can tweak those particulars as needed.
So, once you have all that framework in place, you can just crank out all the AT commands you want with relative ease, since you're now working at a higher level of abstraction (completed text lines and chunks of channel data). This is also why I said that adding SMS will be easy; it's just another AT command.
OK, hardware quirks. The particulars will depend on the module/shield you are using, but in my case:
Power. There is no pin you can read to know if the modem is on! WTF? So, the way I do it is:
send empty 'AT' command. This will alway return 'OK' if it is in command mode and your serial port is set up. It has a secondary benefit in that this is what you must do to trigger autobaudding. Note, for autobauding, the AT is case-sensitive. Anyway, I interpret a failure of this command to mean 'power is off' which is not strictly true but is good enough for my purposes.
Power. To turn on the power, you simulate a button push by pulsing D9. Haha, yes. OK, that's how it's done. Be sure to hold it long enough and wait after releasing it for the modem to 'boot' before sending the intial blank 'AT' command for autobauding (and sanity check that it is indeed on).
There is no handshaking. The SIM900 module has RTC/CTS lines, but on my module they are not brought out. This normally gives me the heebiejeebies, but since the modem has an idle-request design, and because there are well-defined maximal sized data chunklets involved in the various interactions, you can robustly get away without the handshaking so long as you are punctual about consuming the receive dataas it comes.
Some of the early boards out there will suck the living daylights out of your Vcc when powering the radio up, causing a spurious reset, so you really need to power the board from the jack. Later rev boards are more gentle.
OK, GPRS quirks:
you need to get a SIM card. I used EmbeddedWorks.
the GPRS acts a bit like a dialup mode, and uses a thing called an 'Access Point Name', which looks like a DNS name, but don't get too hung up on that. Its a magic text string that you must provide at certain times to make your board able to get to the Internet. You do this by 'activating a PDP context', which requires the APN to do so.
Your wireless provider will tell you what it is. I don't know why they don't just put it on their web site -- its not a security issue because its not actually a DNS name and you have to have a valid SIM card to use it, but there it is. Save yourself hours of trouble and just ask support immediately what the APN you need is.
the SIM900 needs the APN at various times. Moreover, sometimes it needs the APN indirectly, by referencing an 'index' to some stored APNs, and at other times it needs it explicitly in the AT command. Sheesh. Anyway, when it is 'indexed', the index is into some NVRAM, which you only need to set once. In fact, you want to avoid writing to it too much for wear reasons, so what I do is:
query what's there
see if it's already set to what I want
if so carry on
if not set it
that way the board is self-initializing.
be prepared for the mobile provider to terminate your PDP context out of the blue. If you implemented the exception mechanism I described, you will be robust against this automatically, but you can also handle the asynchronous notification from the modem and try to restablish connection more 'surgically' if you want.
be aware that the mobile provider will consider hammering out reconnect attempts as abusive, and will make it so you cannot reconnect! I did this several times while in my code/test/debug process. You just have to wait a while (hours) and then you can resume. I put a Sleep in my recovery loop to avoid hammering their system.
I do not know why, but I seem to pay for about 4 times as much actual channel data that I send. I think they charge even for IP headers, but even that seems like way too much. I need to follow up with my mobile provider about that, but I'm still within my monthly limit so it's not a crisis for me to understand. I do mention this so you can take it into consideration for planning.
little secret: these devices operate on some different 'part' of the cellular network, and the connection is stony-solid even when festivals are in town, which normally result in a no-talk/no-text zone for consumer's handsets. Hmm!
So my modem init sequence looks like this:
setup COM port
send empty AT command to test for power on
flick power switch if needed, and retest power on
reset modem with ATZ
turn off echo mode with ATE0
query stored APN, set if needed. There are two slots (indices) you can store it in, and I arbitrarily store mine in slot 1.
query SIM card ICCID. I do this to detect if there is a SIM installed. You won't get far without it, but you will get far enough that you will not know you are without it.
give the modem a chance to get on the network, which can be slow the very first time; presumably because there is first-time activation stuff going on in the mobile provider's system.
activate the PDP context (this will refer to the APN by index) with an AT+CGACT. This can be very very slow the first time; I use a timeout of 60 sec
set the TCP to the 'multiconnect' option with an AT+CIPMUX. This will imply 'non-transparent'; don't issue a 'set non-transparent' command, because that will now return an error (even though the option value is correct, hmm!).
start the 'TCP task' wtih an AT+CSTT. Here you will need the APN again, but this time not by index; rather it's part of the AT command itself. Hmm! Whatever. I am guessing that this is some sort of separate program running in the SIM900 module that you have to crank up.
'bring up the wireless' for the TCP 'task' with an AT+CIICR. You just have to do it, and it can take a while; I use a timeout of 30 sec.
'get local IP address' with an AT+CIFSR. This is more than a gratuitous query, it is required to make you actually have an IP address assigned. You can throw it away or just log it for fun as I do.
Now you're ready to make connections! You make a connection by supplying host and port number, and a connection slot number. As mentioned, the SIM900 will support 8 connections 0-7, but since the first connection is special, I use connection 1-7. The first is special in that it is the only one that can support server functionality. So if you make a client connection on 0, you can forget about server. You establish the connection with AT+CIPSTART. Internally, if successful, I immediately also issue a AT+CIPSEND? to update the connection max length values.
Now you can finally do useful things with your emulated socket Send() and Recv() (and Poll(), isConnected(), etc).
My teardown does:
shutdown the TCP task
detach from network
pulse power to turn off (there is an AT command for this as well, but I don't use it)
Oh, one last thing worth mentioning. Your connected modem will probably NOT be internet reachable. You get a non-routeable address like 10.x.x.x. If you need your units to take inbound requests, you will need to either:
a) work with your mobile provider, who will probably set up a VPN into your device's subnet. This will be thousands of dollars up front with also some monthly costs.
b) set up a 'relay agent' that your devices connect out to, which will forward inbound requests (and responses) through that single device connection.
I did 'b' since I've done this many times before to achieve firewall/NAT traversal.
Oh, one last last thing:
avoid some self-inflicted baldness by investing in one of those FTDI modules. You can develop all your code on the desktop, which is way way easier (I find) to develop and debug than on the device.
you can also use this same module to 'monitor' the communications to the modem using something like putty, etc. I only found it necessary to monitor the modem data back, so I only needed one, but if you want to monitor both sides, you can use a pair of them.
If you do develop on the desktop, remember that not everything there is available on dotNetMF. I routinely did trial compiles to validate that as I went; especially when developing fancy stuff to avoid coding myself into a porting corner.
Also be aware that some of those platform differences will be visible only at runtime. E.g. the regex impl on dotNetMF actually derives from a Java project, and not all the features are there. In my case, I had designed my regexs with 'named capture groups', which aren't supported on netmf, so I had some boring porting work to do. You only find out that at runtime, however, with a rather uninformative error message in the exception thrown.
unknown, however the stuff above arises mostly from the fact that these modules are effectively 'systems-on-the-side' of the *duino, and communicate via a high-level application protocol rather than a low-level driver protocol. I just took a look at the Spread Spectrum SM5100B module and it has an eerily similar AT command set for performing the TCP/IP stuff (for some reason I thought the SIM900 commands were their own proprietary creation).
On the plus side, maybe, I think the TCP/IP stuff was the most complex on the module's features (well, maybe except for Server mode, which I haven't yet implemented). But there's still a fair amount of AT commands that are needed to be issued just to get it up on the network and ready to do what you wanted.
On the other plus side, once you have bit the bullet and made a library that is robust, then you can use the module with relative ease.
The modem-style AT command approach is established practice and indeed much of the command set is derived from the GSM standards, and it's probably a great boon to the 8-bit folks. But if I were a module designer, I think I would have preferred to expose an SPI interface to IP-level functions, and have the module integrate more-or-less transparently into the existing TCPIP stack. Then your app could use sockets like anything else (you'd still need to do some stuff somehow to get the radio up and on GPRS beforehand, but please no AT commands).
