Today, I looked at an article about Mark Sanford, and I started to read the emails between he and his mistress. Then I stopped. Those aren't any of my business.
I don't care who he's sleeping with, I don't. I do care that he disappeared for several days when he should have been in contact with, oh, say, his OFFICE. He's governor of a whole state. That's his job. He should do his job, or else he becomes unworthy of keeping it.
What he does with his private life matters only to he and his family, and anyone else directly involved with those parts. When he neglects his public duty, then it's the absolute right of the public to get all uppity about it.