Guests During the Game

I am finding that having people "just watch" during my D&D game is very distracting. I don't like telling people that they can't hang.