I think my recommendation to our board will be to start recording some of the dances (with their knowledge, of course). This will help us, both for feedback for those callers who want this feedback to improve their calling, and for helping us with having evidence for those callers that aren't meeting the needs of our dances. Then we can provide specific areas for improvement, and if improvement isn't made, then we can have a procedure in place for taking them "off the rotation". This will require a little more work for a reviewer, but probably only in circumstances that require intervention.
Thanks to everyone. Specific feedback is below:
Walker, I agree that "banning" is too absolute.
Mac, yes, my situation is for local callers who are part of our community and whom I don't want to exclude in other ways. We've also developed several new talented callers through our New Mexico Callers Collective (NMCC), so there is less pressure to rely on callers who don't meet our needs.
Laur, yes, I agree that the challenge is honoring the relationships and getting consensus. Also, the opinions of a community shift slowly, so implementing change takes time.
Jeff, I agree that we shouldn't hire this sort of caller.
David, I like your idea for collecting evidence (through video or audio recording) that make it clear what is working and what isn't, and that this can simply apply for everyone.
Lindsay, I agree that feedback strategies are really helpful. I like the recordings, they are objective.
Peter, the NMCC has been developing training materials and caller skills for callers, with evaluation by our local org's board (FolkMADS). When we've asked for feedback via online forms, all we get are the most vocal being brutal, so we've stopped that practice. If the comments were representative, it would be different.
Martha, I love Joseph in a Box!