There were never many Brit troops in India. They didn't rule by conquering it, they just ran the place on money and better tech, that they were sharing.
Being a Brit colony wasn't even close to the worst thing that could have happened to the Indians.