Never. Puppeteers watch what they are doing using a video split from the camera and you can see the scene unfold as a viewer would so you become all the more drawn in to it.

Although some actors scoot off set for "reverses" on their scenes and let the Assistant Director read in their lines, on Farscape the cast would stay and read in opposite the puppet, which made the scene feel much more real and provided a continuity. I think they realised just how important their input was in making the character seem alive.