You would not need to do it for the logic speed, but consider whether those grounds are carrying power between the boards.
If they are, you may still need a few grounds (rather than just one) to ensure they provide a low resistance path between the boards. The fewer grounds the higher the resistance, the hotter they could get, and there would be a voltage differential between the boards which could confuse the logic gates.
Paul
Thanks for the reply. The grounds aren't carrying power, they're strictly logic reference. I've got many other power grounds.