There's a *lot* of Roman structures in France to this day, but a lot of it is partial and fragmentary because it continued to be maintained, added onto, modified, and merged with later construction. In particular, the city walls in the core 'old city' of just about every significant French town include Roman-era sections that were integrated into larger medieval-era walls, and sometimes further walls all the way to the Napoleonic era.By the time Saxons really rocked up in force, most of Roman Britain had crumbled to the point that the local Britons themselves couldn’t repair it. It’s not that any of them were stupid. They knew it was good building material which was why they scavenged a lot of it, but they did not have the means to maintain it.
I remember going around the ruins of a Roman villa near where I live, and when asked about something like this the guide said “alright, you could move into it for a while. But you’ve got no way to repair it, and a lot of it is empty space with water leaking through the roof.” Not of very much use to a Saxon ring giver, is it?
The stuff that's in *ruins* is typically that which was already abandoned *during the Roman period*, and had decayed to the point of not being useful enough to maintain or restore by post-Roman civilization. For example, the great Roman bathhouse in Paris was no longer functional by the 3rd century, long before Rome actually ceased to govern proto-France.