My two cents on this is:
Why bring them down? The entire suite of Confederate symbology should be kept as a cautionary tale. Remove that and we run the risk of repeating past mistakes.
Also, I notice that the loudest campaigners for destroying Confederate symbology are either eerily quiet or...