Hi Craig,
I would suggesting trying something other than chamois on your buffs.
I never had any success with chamois. I tried it multiple times, and always ended up returning to cotton velvet. Microsuede or microfiber cloth might be an alternative that work for you. I have found some that work as as well, and occasionally better than cotton velvet.
Did you see this thread? https://cdags.org/forums/topic/plate-polishing/
Lots of helpful hints on using a random orbit sander. That said, I never liked the results I got from my sander, and returned to 6″ wheels. I too struggled with swirly marks, and even when I got rid of them found the polish always felt flat.