Actually I managed to get some random results changing the size of the image chunks I fetch down to 128.
const int IN_BUFFER_SIZE = 512;
change into
const int IN_BUFFER_SIZE = 128;
However, it is not reliable at this stage.
Now I'm going to use a logic level converter since the N+2 is said to be 3.3v and the camera is said to be 5v. I will hopefully get some more reliable communication.