7 replies to this topic

[<Jeremy>]

Hi everybody I've got a question about Serial port communication using the netduino. This is the first time I've built something like this - my first week with a Netduino - but I hope this isn't a dumb question!!!. Where I've got to so far: I'm using a barebone FTDI USB to Serial mini adapter - I open COM1 at 115200, and repeatedly send a byte array (each packet being 16 bytes long) from my netduino. I also have a C# client application which reads from the com port at 115200, and successfully sees the stream of bytes that I'm sending from the netduino. So far so good. Where I'm hitting the problem: The client application isn't able to determine where my packets of 16 bytes start and end. I mean if I tell the my client app to read 16 bytes from the port, I find that it's possible it'll pick up the last 7 bytes of one packet and then the first 9 bytes of the second. I think this is probably expected behaviour. My current workaround is a bit of code that looks for a certain byte pattern in what comes in from COM1 - when it finds that, then read the next 16 bytes of data and assume it's a valid packet of my data - and I just repeat that. But do you guys know if there's a more elegant or robust solution to my problem, or if there's a protocol I should look into using to tokenise my data stream? Thanks in advance for any help! Jeremy

[Stefan W.]

Hi everybody

I've got a question about Serial port communication using the netduino. This is the first time I've built something like this - my first week with a Netduino - but I hope this isn't a dumb question!!!.

Where I've got to so far:

I'm using a barebone FTDI USB to Serial mini adapter - I open COM1 at 115200, and repeatedly send a byte array (each packet being 16 bytes long) from my netduino.

I also have a C# client application which reads from the com port at 115200, and successfully sees the stream of bytes that I'm sending from the netduino. So far so good.

Where I'm hitting the problem:

The client application isn't able to determine where my packets of 16 bytes start and end. I mean if I tell the my client app to read 16 bytes from the port, I find that it's possible it'll pick up the last 7 bytes of one packet and then the first 9 bytes of the second.

I think this is probably expected behaviour.

My current workaround is a bit of code that looks for a certain byte pattern in what comes in from COM1 - when it finds that, then read the next 16 bytes of data and assume it's a valid packet of my data - and I just repeat that.

But do you guys know if there's a more elegant or robust solution to my problem, or if there's a protocol I should look into using to tokenise my data stream?

Thanks in advance for any help!
Jeremy

One general solution to this kind of problem is using a Self-Synchronizing code, another one is using markers to delimit token boundaries. How I would design the protocol specifically is dependant on the exact setting. If there's a byte (sequence) that will never appear in the datastream, you can use it between every N tokens (low N -> fast synchronization, high overhead, high N -> slower synchronization, lower overhead) to mark a boundary.

[liqdfire]

Hi everybody

I've got a question about Serial port communication using the netduino. This is the first time I've built something like this - my first week with a Netduino - but I hope this isn't a dumb question!!!.

Where I've got to so far:

I'm using a barebone FTDI USB to Serial mini adapter - I open COM1 at 115200, and repeatedly send a byte array (each packet being 16 bytes long) from my netduino.

I also have a C# client application which reads from the com port at 115200, and successfully sees the stream of bytes that I'm sending from the netduino. So far so good.

Where I'm hitting the problem:

The client application isn't able to determine where my packets of 16 bytes start and end. I mean if I tell the my client app to read 16 bytes from the port, I find that it's possible it'll pick up the last 7 bytes of one packet and then the first 9 bytes of the second.

I think this is probably expected behaviour.

My current workaround is a bit of code that looks for a certain byte pattern in what comes in from COM1 - when it finds that, then read the next 16 bytes of data and assume it's a valid packet of my data - and I just repeat that.

But do you guys know if there's a more elegant or robust solution to my problem, or if there's a protocol I should look into using to tokenise my data stream?

Thanks in advance for any help!
Jeremy

Probably one of the easiest things to do is put a single byte at the start of your byte array that would otherwise be invalid data, say byte 0 (null) then you can look for that byte and read the next N number of bytes for a completed message.

I have some code for the desktop client from a my personal stock that you can use, I need to clean it up; I will do that after work and post it for you.

[liqdfire]

Here is one chunk of it, after you read bytes off of the serial port you call AddData on this object, it will look for the start of the header byte and read the next N# of bytes for a completed message, this particular example though there were 2 bytes indicating the length of the message right after the header byte, you can just remove the read of those bytes and set length to a fixed value; I will work up an example for using it with a serial port later. Though I left off the dispose and cleaning up the memory stream.

will repost after i fix some things in it.

[<Jeremy>]

Hey Stefan and Liqdfire Thank you for the assists - it sounds like the answer for me is to develop a custom solution, like you say maybe have a unique "end of packet" sequence. Liqdfire - If you do get a chance to post some code that would be really great - in the meantime, I'll spend a bit of time coding up a solution and post my code when I'm done. Hopefully this thread will help someone else out if they hit the same problem. Thanks again, Jeremy

[liqdfire]

using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;

public class DataHolder :IDisposable
{

	#region Private Variables
	private Queue<byte[]> _completedMessages = new Queue<byte[]>();
	private object _syncObject = new object();
	private System.IO.MemoryStream _pendingMessages = new System.IO.MemoryStream();
	private byte _headerByte = Convert.ToByte(0);
	#endregion

	#region Properties
	public object SyncRoot
	{
		get { return _syncObject; }
	}
	public Queue<byte[]> CompletedMessages
	{
		get { return _completedMessages; }
	}
	#endregion

	#region Constructors
	public DataHolder(byte headerByte)
	{
		_headerByte = headerByte;
	}
	#endregion

	#region Public Methods
	public bool AddData(byte[] data)
	{
		bool messageWasCompleted = false;
		// The synclock ensures only one thread at a time can modify the queue
		lock (_syncObject)
		{
			_pendingMessages.Write(data, 0, data.Length);
			// This takes all of the data from the memory stream and puts it in an array
			byte[] cachedData = _pendingMessages.ToArray();
			//Uh we have no data?? then we cannot continue, so exit
			if (cachedData == null)
				return false;
			//Trim the start of the array to the start of the first message
			int indexOfStartofMessage = Array.IndexOf<byte>(cachedData, _headerByte);
			if (indexOfStartofMessage > 0)
				cachedData = cachedData.Skip(indexOfStartofMessage).ToArray();

			//If there are at least 4 bytes then we know how long the first message is supposed to be
			if (cachedData.Length > 4)
			{
				//
				// A couple different ways to check message size
				//

				// Check Message size by reading the two bytes after the header byte
				byte[] lengthBytes = { cachedData[2], cachedData[3] };
				int length = BitConverter.ToInt16(lengthBytes, 0);

				// Use a fixed message size
				//int length = 17; (16 byte message + 1byte header)

				//We have a complete message
				if (cachedData.Length >= length)
				{
					byte[] message = new byte[length];
					Array.Copy(cachedData, message, length);
					// Copy the bytes into the message array
  				_completedMessages.Enqueue(message);
					// Add the new byte array to our list of completed messages
					byte[] leftOverData = new byte[cachedData.Length - length];
					//Copy the left over data into the new array
					Array.Copy(cachedData, length, leftOverData, 0, cachedData.Length - length);
					//Reset the memory stream
					_pendingMessages.SetLength(0);
					_pendingMessages.Position = 0;
					//Write the left over data back out
					_pendingMessages.Write(leftOverData, 0, leftOverData.Length);
					messageWasCompleted = true;
				}
			}
		}
		return messageWasCompleted;
	}

	public void Dispose()
	{
		if (_pendingMessages != null) 
		{
			_pendingMessages.Close();
			_pendingMessages.Dispose();
		}
	}
	#endregion
}

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;


public class Serial
{
	#region Private Variables
	//The Convert.ToByte(0) assumes you are using the NULL byte, byte 0 as the indicator to the start of your packet
	DataHolder _messageHolder = new DataHolder(Convert.ToByte(0));
	System.IO.Ports.SerialPort _serial;

	//Processing Threads
	System.Threading.Thread _processingThread;
	bool _stop = false;
	#endregion

	#region Constructor
	public Serial(string comPort, int baudRate)
	{
		//Setup the processing threads
		_processingThread = new System.Threading.Thread(ProcessSerialData);
		_processingThread.IsBackground = true;
		_processingThread.Start();

		_serial = new System.IO.Ports.SerialPort(comPort, baudRate);
		_serial.Open();
	}
	#endregion

	#region Public Methods
	public void Stop()
	{
		_stop = true;
		if (_processingThread != null && _processingThread.IsAlive) _processingThread.Join();
		if (_serial != null)
		{
			if (_serial.IsOpen) _serial.Close();
			_serial.Dispose();
		}
	}

	#endregion

	#region Private Methods
	private void ProcessSerialData()
	{
		while (!_stop)
		{
			//TODO: You can process the message here or below
			System.Threading.Thread.Sleep(100);
		}
	}

	private void Serial_DataRecieved(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
	{
		using (System.IO.MemoryStream ms = new System.IO.MemoryStream)
		{
			while (_serial.BytesToRead > 0)
			{
				byte[] data = new byte[1024];
				int bytesRead = _serial.Read(data,0,1024);
				ms.Write(data,0,bytesRead);
			}
			if (_messageHolder.AddData(ms.ToArray()) == true)
			{
				
				lock(_messageHolder.SyncRoot)
				{
					//TODO: You can process the message here
					// If you process it in the seperate processing thread you can commed out the lock part
				}
			}
		}
	}
	#endregion
}

[Mario Vernari]

Hello AlphaReever.

The client application isn't able to determine where my packets of 16 bytes start and end. I mean if I tell the my client app to read 16 bytes from the port, I find that it's possible it'll pick up the last 7 bytes of one packet and then the first 9 bytes of the second.

I think this is probably expected behaviour.

Absolutely, and it's because the PC management. Your Netduino is not guilty!
I can't rely on timings.

My current workaround is a bit of code that looks for a certain byte pattern in what comes in from COM1 - when it finds that, then read the next 16 bytes of data and assume it's a valid packet of my data - and I just repeat that.

That's also normal. You are adding the missing "OSI layer" to the protocol.
There are plenty of UART-specific protocols, the most used are XON/XOFF, and they rely on a certain pattern for the start and for the termination.

But do you guys know if there's a more elegant or robust solution to my problem, or if there's a protocol I should look into using to tokenise my data stream?

Always we need to define what "elegant" means.
Generally, such a protocol could be composed as follows:
- start marker: often it uses the SOH character (0x01), that has historically just this function;
- address: (optional) if your UART is connected to many slaves, and you need to address one of them;
- length: the remaining count of bytes;
- body: the useful data;
- crc/checksum: a redundancy check to verify the message integrity.
I would suggest to take a look at the Modbus-RTU specs.
Cheers

[<Jeremy>]

Hey liqdfire and Mario Thank you so much for this help and information - I've been getting through it this weekend and think I've got a better (and more object oriented) solution to my problem with your help. Thanks again, Jeremy