The late Larry Jennings, who did the booking for the Thursday Concord MA series for some time used the "no, and if you are interested, I can tell you why".   When asked, Larry was honest (blunt) because that was the way he was.  He was probably more critical of callers than bands, given his expertise and interests.

Applicants deserve being handled with honesty, but do remember they are people.  If I were handling booking, my refusal would be on the lines of "we don't think that what you offer is consistent with our vision of our series and the personality we want to project at our evenings."  What they have to offer might be perfectly fine for another series.

You need to do your research and be able to back up your position by making every effort to see the performer(s) in action before you form an opinion.  If you can be encouraging, or helpful in a specific way, that would be good. ("You've got issues with tempo control", "You always seem unprepared and make a lot of mistakes").