That's always been the fate of true patriots. Even in Revolutionary times, my understanding is that well over half of the colonial population had a neutral or even positive attitude towards the British. Surprisingly few people wanted to rock the boat.

It's always easy to do the right thing when everybody agrees with you.