They can, but Metropolitan France doesn't want to get rid of them. They're only pulled by the lure of profit not also pushed by religious persecution or running out of good farmland. The Black Death emptied a lot of farms, but in England they were mostly on what is marginal land after the end...