Multiple Serials Ports in Threads - Best Practices

Guys

 

I have 4 serial devices I need to talk to at different baud rates all at the same time.  Ive been trying different things from the forums (aka polling vs datareceived events, etc).  For the higher speed devices 115200 baud they just stop responding after a while.

 

What are best practices to pull this off and keep the data flowing.

 

Datareceived events vs polling (while loops with a thread sleep in them)

single thread programming vs 4 threads for 4 serial ports

 

sometimes the device (ND+2) just stops working all together (no exception thrown, nothing in the firmware).  I saw some posts about Debug.GC(true) as a way to see memory usage and perhaps force GC.

 

I will need to deliver the ND+2 in a product soon and I really need it to stay up.  As another complication I am needing to runs a TCP and UDP socket (the TCP listener and client connections are also threads).  The UDP socket is just a broadcaster not a listener.  I also have a FTPServer running (another tcp listener thread).  So all in all right now I have 4 serial port device threads, 2 tcp listeners, and tcp client thread (if a connection made) running.  Maybe that's too many threads? 

 

thanks in advance for best practices advice on multiple serials ports, threads, and best practices!

 

Homey

I've got a 115200 GPRS modem and a 9600 LCD display working together without issue, however I'm not actually doing anything with any data coming back from the LCD. I did find that I had to clear the the buffer on the LCD com port because of this or I had some weird stuff happening. In my case I do it every time I print to the LCD.

 

use

 

serial.DiscardInBuffer();

 

My modem data is handled via an eventhandler

Thanks for the reply.

 

I generally do a DiscardInBuffer after each full read.  The serialport datareceived event works mostly but ive seen it stop.  Im also monitoring memory about every 5 seconds and it never exceeds 70k.  Im guessing its the threading but I don't know how to debug something that doesn't give clues before it stops.

 

The other problem (likely a dumb question but here goes) is that with the datareceived event I don't have a good way of blocking the next write to the port until data are received.  Most of the units I talk to I need to write bytes (request data) and then receive bytes (response data) over and over.  So when I use the datareceived event, im not sure the best way to know when its ok to try and write to the port for the next sequence.  In normal c# I use a timeout exception but that doesn't exist on serialports in MF.  So with MF I seem to have to "while (sport.bytestavailable != numberiwant) loop it (aka poll) until I can write again.

 

Keep em coming!

 

Homey

Hi Homey, I'm guessing that the issue is network-related, rather than SerialPort-related, if your app stops executing. If you're not talking to the network...does the serial communication run forever? [Generally speaking your Netduino's app should just run and run and run...] On polling vs. DataReceived: generally speaking, event driven is always best. And if you need to synchronize different threads, use WaitHandles (ManualResetEvents or AutoResetEvents). You can .Wait() on one of them and .Set() it from another thread...and there are ways to use those as custom pseudo-timeouts. Chris

Thanks for the reply chris.   Im testing no network stuff now.  It seems to run better but without networking getting data is not useful for me since the data from the devices have to get to a central location on the network for use. Although actually, all I have to do is plug in an Ethernet cable and it locks after a while even if I have no code using networking.   When I try datareceived it just stops after a while (never fires).  So im not sure I can depend on that or im missing something there.   I don't really need to sync different threads I don't think.  I want all the serial ports to get data from the devices as fast as they can and then put that data on the network.  Enlighten me on how to use Waithandles as custom pseudo-timeouts?   Are there issues with SPI, Networking on 4.2.2 firmware?  Maybe I can try downgrading to earlier ones unless 4.3 is coming soon.  One of the devices I previously had on SPI but that didn't work very well; hence the SPI question.   Homey

Hi Homey, Except for one edge case (which consistently affects a subset of devices in one of the 4 SPI modes), there are no known issues with SPI on the 4.2.2 firmware. Networking with lwIP...it's a big beast, and new versions of NETMF refine the open source networking stack integration more and more over time. For DataReceived...the events are queued up, so if you block in one of them then you won't see the other events fire. We often create a "WaitForSerialData" function and call it on a new thread in our application. Then we create an AutoResetEvent which we autoResetEvent.Set() from the DataReceived event. And then in the WaitForSerialData function, we run a loop and the first line of code in that loop is autoResetEvent.WaitOne(); Chris

Chris   Which SPI case has the bug.  The device I was trying SPI on was   clock idle state false clock edge true   And is the bug only in 4.2.2 (aka if I got to 4.2.0 its not a problem)?   I found an example on autoresetevent so I think I get it. Like a gate keeper to sync all threads. Im not really sure how that would help me when I really just want all the serialports to runas fast as they can. But I would love to be enlightened. Can you post an example of how you use it for datareceived events when you have multiple serial ports getting data in threads? Im also wondering about best practices regarding instantiating serial ports in threads, thread limits, etc.   Homey

Update

 

after many more hours of trial and error I can confirm that at least with the SPI device (a YEI technology IMU) the program runs fine as long as I never connect to the netduino using TCP.  Im guessing Chris's comment about Ethernet above is correct.  I am able to UDP broadcast the data from the device fine, it seems to be related solely to using the TCP stack.  The IMU device manufacturer verified the issues I was having was not related to their IMU by duplicating my code on an arduino uno.

 

Im thinking of trying to revert my ND+2 to an earlier firmware version (right now in at 4.2.2) to see if it does any better with the other serial devices I have running in their respective threads.  Hopefully my trial and error will help others who need a lot of IO from a ND.

 

more later

 

Homey

Hi all!

 

I've got some problems with reading from serial - if You guys could please take a look at my code and tell me what I'm doing wrong

 

What I have already done and what I'm trying to have:

 

Netduino with its COM1 is connected to Arduino MEGA Serial1

Arduino MEGA is running MultiSerialMega app from examples provided in IDE examples.

 

In short - everything what I write in serial monitor in Arduino IDE is sent to Netduino, and everything I see there is sent by Netduino

 

So I wrote this program for Netduino ( I re-wrote some examples and added a little bit extra code).

using System;using System.Net;using System.Net.Sockets;using System.Threading;using Microsoft.SPOT;using Microsoft.SPOT.Hardware;using SecretLabs.NETMF.Hardware;using SecretLabs.NETMF.Hardware.NetduinoPlus;using System.IO.Ports;using System.Text;namespace NetduinoPlus_serial_test{    public class Program    {        static SerialPort serial;        static string msg = "";        public static void Main()        {            // initialize the serial port for COM1 (using D0 - RX & D1 - TX)            serial = new SerialPort(SerialPorts.COM1, 9600, Parity.None, 8, StopBits.One);            // open the serial-port, so we can send & receive data            serial.Open();            // add an event-handler for handling incoming data            serial.DataReceived += new SerialDataReceivedEventHandler(serial_DataReceived);            var thread1 = new Thread(recive);            var thread2 = new Thread(transmit);            thread1.Start();            thread2.Start();            Thread.Sleep(Timeout.Infinite);        }        static void serial_DataReceived(object sender, SerialDataReceivedEventArgs e)        {            // create a single byte array            byte[] bytes = new byte[1];                                    // as long as there is data waiting to be read            while (serial.BytesToRead > 0)            {                // read a single byte                serial.Read(bytes, 0, bytes.Length);                msg += (char)bytes[0];                                            }            Debug.Print(msg);        }        static void recive()        {            while (true)            {                if (msg.Length > 0)                {                    Debug.Print(msg);                    msg = "";                }                Debug.Print("recive thread works");                Thread.Sleep(1000);            }        }        static void transmit()        {            while (true)            {                Debug.Print("transmit thread works");                const string input = "Test$n";                byte[] sender = Encoding.UTF8.GetBytes(input);                serial.Write(sender, 0, sender.Length);                Thread.Sleep(1000);            }         }    }}

so the transmit thread works perfectly - it sends every second "test$" message and I can see it in serial monitor.

 

But when I'm trying to send something to Netduino nothing happens

( I can see only this >> Debug.Print("recive thread works"); )

 

What I want to do is make Netduino read something from Serial, create a string from it that would be accessible from other parts of program (other threads etc.)

 

Many thanks for any help!

I too am having serial issues with ND+2 even after reverting to firmware to 4.2.1.2. It seems received are getting lost somewhere whether I poll or use datareceived events. Im not using the SD card (known issue with lost d ata). Im hoping 4.3 firmware will be out soon to address serial, spi, sdcard, and network issues with ND+2. In your case, you can get statements from bytes by just using. //get the bytes off the port int bytesToRead = _sPort.BytesToRead; //start reading the stream if (bytesToRead > 0) { // get the waiting data byte[] buffer = new byte[bytesToRead]; _sPort.Read(buffer, 0, buffer.Length); } //copy the byte array to a readable string String str = new String(System.Text.Encoding.UTF8.GetChars(buffer)); hope this helps! Homey

Hi Homey!

I'm actually on ND+ (not 2) - Hope no issues here

 

So I'm posting working examples made from other examples that:

-use threads

-one thread is posting test message every 5sec

-second is printing in debug received messages

 

Everything works for me - thanks to all who made some code for serial and sharing it 

using System;using System.Net;using System.Net.Sockets;using System.Threading;using Microsoft.SPOT;using Microsoft.SPOT.Hardware;using SecretLabs.NETMF.Hardware;using SecretLabs.NETMF.Hardware.NetduinoPlus;using System.IO.Ports;using System.Text;namespace NetduinoPlus_serial_test{    public class Program    {        static SerialPort serial;        static string msg = "";        static object monitor = new object();        public static void Main()        {                                    //serial(using D0 - RX & D1 - TX)            serial = new SerialPort(SerialPorts.COM1, 9600, Parity.None, 8, StopBits.One);            serial.ReadTimeout = -1;             serial.DataReceived += new SerialDataReceivedEventHandler(serial_DataReceived);            serial.Open();                                  //threads            var thread1 = new Thread(recive);            var thread2 = new Thread(transmit);            thread1.Start();            thread2.Start();            Thread.Sleep(Timeout.Infinite);        }        static void serial_DataReceived(object sender, SerialDataReceivedEventArgs e)        {            lock (monitor)            {                int bytesToRead = serial.BytesToRead;                byte[] buffer = new byte[bytesToRead];                if (bytesToRead > 0)                {                    serial.Read(buffer, 0, buffer.Length);                }                String str = new String(System.Text.Encoding.UTF8.GetChars(buffer));                msg = str;                Debug.Print(msg + " in serial_DataReceived");            }        }        static void recive()        {            while (true)            {                lock (monitor)                {                    if (msg.Length > 0)                    {                        Debug.Print(msg + " in thread");                        msg = "";                    }                                       }                            }        }        static void transmit()        {            while (true)            {                lock (monitor)                {                    Debug.Print("transmit thread works");                    const string input = "Test$n";                    byte[] sender = Encoding.UTF8.GetBytes(input);                    serial.Write(sender, 0, sender.Length);                                    }                Thread.Sleep(5000);            }         }    }}

Ahh, nice. What im stuggling with is Sending and Receiving Serial Byte in the same thread off Main AND doing it as fast as possible. Your example is once every 5 seconds. In my case, I need to do it as fast as possible likea "Send Request Messge", "Receive Response Message" concept. For the device Im using I send it 14 bytes and should always get back 72 bytes. I do see your use of a lock object so maybe I can try that to automatically block Tx and Rx based on the datareceived event until I get the full 72 bytes. But sometimes I don't. I only get 68, or 56, or something less. In my code tho, there are cases where (a) the datareceived stops firing, and ( the data coming off the port is missing bytes. It works for a while then starts dropping or stopping. Thanks for the lock idea! Homey

Then, I think, You can get rid of receive() thread and put everything You want to do with Your received msg to  static void serial_DataReceived because it is executed while something is to read (so ASAP ).

Right! I also note you keep the datareceived event and serial port init in the main thread. Im doing that in the device thread it self. Again, another good suggestion I will try when I get back to the hardware! Homey

One more thing I discovered:

 static void serial_DataReceived(object sender, SerialDataReceivedEventArgs e)        {            lock (monitor)            {                Thread.Sleep(100);   << ADD THIS!!!!!!!!                int bytesToRead = serial.BytesToRead;                byte[] buffer = new byte[bytesToRead];                if (bytesToRead > 0)                                       (code continues....)

Why? I found out when I tried to send longer message i.e. "Test1234567890" I received "Test1234567" and few chars got lost, but in step mode (I analyzed every step so it took a while) everything goes well. So I realized the thread have to wait a little bit to get whole message and then process it

I ran into the same thing when using a GPRS shield. I had to sleep for a little to allow the incoming buffer to fill before trying to do anything with it or I'd lose data. In my case I'd get all the header info from SMS messages but would lose the actual message

Aye, there in lies the crux of one of the problems.  I expect 72 bytes from my device .. .always. How do I know how long to sleep?  If I put a

 

if (bytestoread == 72)

 

wait in there it sometimes never finishes.  l Also put a print statement to print how many bytes are on the port every time the event fired, and it took 9-12 fires to reach 72 ... on the times it actually got that far.  The port is supposed to be at 115200 baud so it shouldn't take 100ms to get 72 * 8 bits over the wire.

 

In the example here you wait 100 ms.  That means the fastest you can get data from the serial port for just a few bytes was 10 Hz! And your only writing every 5 seconds. That's awful in my application.  I need to get at least 30 Hz off this unit for a realtime app.

 

I do have the hardware back, so I'm gonna try a few things suggested here.  I also have a FitPC-2 (300 bucks and much bigger than a netduino) but I have not seen any issues talking to the same device, same code, at 70Hz over serial with that.

 

More later and thanks for all the ideas!

You can calculate that from baud rate.

Baud rate = bits per second so if You're using 9600 it is 9600/8=1200 bytes per second, then You have 72 bytes to transmit so 72/1200=0,06 seconds.

 

0,06 seconds mean that You have to Thread.Sleep(60) at least. Need more Hz? Increase baud

 

You need to sleep BEFORE doing anything with serial.BytesToRead() so:

static void serial_DataReceived(object sender, SerialDataReceivedEventArgs e)        {            lock (monitor)            {                Thread.Sleep(100);                   int bytesToRead = serial.BytesToRead;                byte[] buffer = new byte[bytesToRead];                if (bytesToRead == 72)

but I don't see a point in bytesToRead == 72. If You have a fixed number of bytes its better to do something like this:

byte[] yourBytes =  new byte[72]static void serial_DataReceived(object sender, SerialDataReceivedEventArgs e)        {            lock (monitor)            {                Thread.Sleep(100);                   int bytesToRead = serial.BytesToRead;                byte[] buffer = new byte[bytesToRead];                if (bytesToRead >0)                {                     Array.Copy(buffer, 0, yourBytes, 0, 72)

and then You have Your 72 bytes to use as You wish

Yup, im at 115200 not 9600.  That's the max baud for the device.  So I should be able to get close.

Ok, here is an illustration of serailport issues with ND+2 even with reverted firmware to 4.2.1.2. first my code. Im using events where I can and ports are initialized in the main thread.  

using System;using System.Net;using System.Net.Sockets;using System.Threading;using Microsoft.SPOT;using Microsoft.SPOT.Hardware;using SecretLabs.NETMF.Hardware;using SecretLabs.NETMF.Hardware.Netduino;using System.IO.Ports;namespace UwbTest{    public class Program    {        static SerialPort m_sPort;        static bool isRunning = true;        static int iTry = 0;        static int iEvent = 0;        static int iDataIndex = 0;        static InterruptPort m_button = new InterruptPort(Pins.ONBOARD_SW1, false, Port.ResistorMode.Disabled, Port.InterruptMode.InterruptEdgeLow);        static object monitor = new object();  //lock        static AutoResetEvent DataReceivedGate = new AutoResetEvent(false);        static byte[] response = new byte[72];  //my 72 byte response        static double millisElapsed = 0;        static DateTime start = DateTime.Now;        public static void Main()        {            // write your code here            m_button.EnableInterrupt();            m_button.OnInterrupt += new NativeEventHandler(m_button_OnInterrupt);            m_sPort = new SerialPort("COM2", 115200, Parity.None, 8, StopBits.One);            m_sPort.ReadTimeout = -1;            m_sPort.Handshake = Handshake.None;            m_sPort.DataReceived += new SerialDataReceivedEventHandler(m_sPort_DataReceived);            m_sPort.Open();            Thread thread1 = new Thread(GetUWBData);            thread1.Start();            Thread.Sleep(Timeout.Infinite);        }        static void GetUWBData()        {            byte[] msg = RCM_SEND_RANGE_REQUEST();            Debug.Print("[UWB] Start Ranging");            while (isRunning)            {                lock (monitor)                {                    //write to the port                    Debug.Print("[UWB] Writing to Port");                    iEvent = 0;                    start = DateTime.Now;                    m_sPort.Write(msg, 0, msg.Length);                    //flush the buffer to ensure the write finishes                    //m_sPort.Flush();                }                DataReceivedGate.WaitOne();                iTry++;                millisElapsed = ((TimeSpan)(DateTime.Now - start)).Milliseconds;                Debug.Print("[UWB - GATE] Gate Set so Got 72 Bytes " + "iTry " + iTry.ToString() + " Elapsed Millis " + millisElapsed);                //reinit the response to zeros                for (int i = 0; i < 72; i++)                    response[i] = (byte)0x00;            }            m_sPort.Close();            Debug.Print("[UWB] Program Stopped");                        }        static void m_sPort_DataReceived(object sender, SerialDataReceivedEventArgs e)        {            lock (monitor)            {                iEvent++;                int bytesToRead = m_sPort.BytesToRead;                Debug.Print("[UWB - DATARXEVENT] Bytes to Read " + bytesToRead.ToString() + " IEVENT " + iEvent.ToString());                byte[] buffer = new byte[bytesToRead];                m_sPort.Read(buffer, 0, bytesToRead);                Array.Copy(buffer, 0, response, iDataIndex, bytesToRead);                iDataIndex += bytesToRead;                if (iDataIndex == 72)                {                    iDataIndex = 0;                    DataReceivedGate.Set();                    //m_sPort.DiscardInBuffer();                    iEvent = 0;                }            }        }        static void m_button_OnInterrupt(uint data1, uint data2, DateTime time)        {            isRunning = false;        }        static byte[] RCM_SEND_RANGE_REQUEST()        {            byte[] serialPrefix = new byte[2] { 0xA5, 0xA5 };            byte[] numBytes = new byte[2] { 0x00, 0x0C };  //no "data" is 12 bytes in message            byte[] msgType = new byte[2] { 0x00, 0x03 };            byte[] msgIdOut = new byte[2] { 0x00, 0x00 }; ; //2bytes            byte[] rspID = new byte[4] { 0x00, 0x00, 0x00, 0x70 };  //4 bytes  (unit 112)            byte[] antMode = new byte[1] { 0x00 };            byte[] reserved = new byte[1] { 0x00 };            byte[] bdataSize = new byte[2] { 0x00, 0x00 };            byte[] msgMinusHeaderandLength = new byte[12];            Array.Copy(msgType, 0, msgMinusHeaderandLength, 0, 2);            Array.Copy(msgIdOut, 0, msgMinusHeaderandLength, 2, 2);            Array.Copy(rspID, 0, msgMinusHeaderandLength, 4, 4);            Array.Copy(antMode, 0, msgMinusHeaderandLength, 8, 1);            Array.Copy(reserved, 0, msgMinusHeaderandLength, 9, 1);            Array.Copy(bdataSize, 0, msgMinusHeaderandLength, 10, 2);//            byte[] crc = crc16(msgMinusHeaderandLength, msgMinusHeaderandLength.Length);            byte[] crc = new byte[2] { 0x65, 0x7C };  //101,124            byte[] req = new byte[18];            Array.Copy(serialPrefix, 0, req, 0, 2);  //bytes 0 - 1            Array.Copy(numBytes, 0, req, 2, 2);  //bytes 2 - 3            Array.Copy(msgType, 0, req, 4, 2);  //bytes 4 - 5            Array.Copy(msgIdOut, 0, req, 6, 2);  //bytes 6 - 7            Array.Copy(rspID, 0, req, 8, 4);  //bytes 8 - 11            Array.Copy(antMode, 0, req, 12, 1);  //bytes 12            Array.Copy(reserved, 0, req, 13, 1);  //bytes 13            Array.Copy(bdataSize, 0, req, 14, 2);  //bytes 14 - 15            Array.Copy(crc, 0, req, 16, 2);  //bytes 16 - 17            return req;        }    }}

What basically happens is the datareceived event fires for a while but a few bytes at a time until it finally reaches the correct amount of 72. I use a autoreset event to notify the sender thread (GetUWBData) we got all the data. It works for a while (152 times at about 15ms intervals - 67Hz - and then just stops firing like so. Its interesting that the first fire of the event has all 72 bytes but thereafter it trickles in. [UWB] Start Ranging [UWB] Writing to Port [UWB - DATARXEVENT] Bytes to Read 72 IEVENT 1 [UWB - GATE] Gate Set so Got 72 Bytes iTry 1 Elapsed Millis 182 [UWB] Writing to Port [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 1 [UWB - DATARXEVENT] Bytes to Read 10 IEVENT 2 [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 3 [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 4 [UWB - DATARXEVENT] Bytes to Read 11 IEVENT 5 [UWB - DATARXEVENT] Bytes to Read 10 IEVENT 6 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 7 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 8 [UWB - DATARXEVENT] Bytes to Read 11 IEVENT 9 [UWB - GATE] Gate Set so Got 72 Bytes iTry 2 Elapsed Millis 16 [UWB] Writing to Port [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 1 [UWB - DATARXEVENT] Bytes to Read 10 IEVENT 2 [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 3 [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 4 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 5 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 6 [UWB - DATARXEVENT] Bytes to Read 11 IEVENT 7 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 8 [UWB - DATARXEVENT] Bytes to Read 9 IEVENT 9 [UWB - GATE] Gate Set so Got 72 Bytes iTry 3 Elapsed Millis 15 [UWB] Writing to Port and so on until it just stops [UWB] Writing to Port [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 1 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 2 [UWB - DATARXEVENT] Bytes to Read 0 IEVENT 3 <--- why would this happen? [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 4 [UWB - DATARXEVENT] Bytes to Read 13 IEVENT 5 [UWB - DATARXEVENT] Bytes to Read 11 IEVENT 6 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 7 [UWB - DATARXEVENT] Bytes to Read 11 IEVENT 8 [UWB - DATARXEVENT] Bytes to Read 9 IEVENT 9 [UWB - GATE] Gate Set so Got 72 Bytes iTry 152 Elapsed Millis 15 [UWB] Writing to Port [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 1 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 2 [UWB - DATARXEVENT] Bytes to Read 2 IEVENT 3 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 4 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 5 [UWB - DATARXEVENT] Bytes to Read 12 IEVENT 6 [UWB - DATARXEVENT] Bytes to Read 11 IEVENT 7 [UWB - DATARXEVENT] Bytes to Read 7 IEVENT 8 <---- never got to 72 bytes! only 70! The same thing happens if I replace the Gate (Autoresetevent) with just a very long thread.sleep in GetUWBData sender and datareceived event receiver. I have tried different com ports with this device. Ive tried with other devices (and their different baud settings). Its always the same, eventually I don't get all the data of ND+2 serial ports either using events or polling even when the devices say they are 3.3v serial port logic tolerant. I also found it interesting that the datareceived event fires when there are 0 bytestoread? UGGGGGH!