Hi Read! I wanted to add that the times when I (a non-binary person that is generally assumed to be a man) have encountered neighbor men who refuse to swing with me, it has often been accompanied with either thinly-veiled disgust or even an overt "ew I'm not dancing with a man." I'm so sorry that you were physically assaulted, though I'm not surprised.
I agree with Read that this is just general homophobia, and again it makes me worry about the women these men are dancing with, simply because they clearly don't want to "hold a man" in the same way that they are "holding" all the women they neighbor/partner with.
Polite declining is 100% acceptable and I've never been upset by it. But more often than not, men refusing to dance with me has been a rather dramatic production that leaves me feeling icky. 🤷