The plane of the RGB emitter is higher than that of the light sensor so any interference will probably be mostly due to reflection from whatever is above the shield. Try to keep reflective objects away, or if that doesn't work perhaps glue in some cardstock around the LED or light sensor.
Oh maybe a tiny prism on top of the sensor and the sides of the prism covered with the cardstock. Clever clever thanks a lot
