The former president said a large group of people, including "waitresses and caddies and drivers," are being hurt.