The swing is arguably the move that requires the most trust in the other person, and not being able to choose someone that you're doing half the swings in a dance with can cause some unsafe situations for various reasons. A single swing with a neighbor can usually be accommodated, but much more than that can become dangerous.
Also, people are allowed to just not like shadow-swings without giving a reason or explaining themself. They are allowed to find them confusing/disorienting. Just as others are allowed to enjoy them.
People are also allowed to avoid dancing with certain people for any reason whatsoever; yes we want to be inclusive, but inclusivity to the point of ignoring the safety/comfort of the individual runs into the Horseshoe Problem (ie. pushing an issue so far that you end up causing the problems you set out to avoid).
To bring it back to the topic at hand, I think a good rule of thumb as a caller (and one I employ on the rare occasion I choose a dance with a shadow swing) is telling people that the next dance has a shadow swing in it, in advance of the walkthrough. And then giving people a chance to choose to sit out or even join a different line if they have concerns. That way, those that enjoy the move can have a nice time, and those that don't aren't forced into an unsafe situation until they can get to the end of a line to drop out.