That's certainly true to the extent that immigration hurts the wages and/or bargaining power of any native-born Americans. In this regard, immigration restrictionism would benefit all native-born Americans, though out of these, white Americans are by far the largest group.
And some Mexicans are...