I often pop into rooms that are open to see what people are playing. Hang around for a few minutes then come back later if the game interests me.
I do not ask for permission to enter in advance, I think having to answer such requests all the time is more disturbing to the players than what I do. Have never had a private message saying they did not like my viewing.
If you do not want your game to be visited, just lock the door. It is not rude, it is just your privacy (and possibly the troll thing).
To Tater: if you are in a locked room you have created and there is another guy in there, you can right click on your room status and press "Kick". That automatically removes the player from the room. So you do not have to ask anyone to leave and if they do not respond make a new room.
Anyone who has "build the wall" in their signature line should of course lock their VASL rooms all the time.