I think it’s likely that Thoros and even Melisandre’s abilities are rare
The average Red Priest may not get much aside from vague visions from the flames
They get special powers
Also, they hang around people, not necessarily planning on doing a war, maybe do some sightseeing and the like...