I tend to work mostly in a trio or quartet, and we don't use written music. We also have terrible memories. So we tend to limit ourselves to one "thing" - rhythmic variation, etc. - per tune or set, and everyone knows what it is and we've practiced it so if someone calls "thing!" it happens pretty much together. But that's the benefit of working in a smaller group, I think. It's a lot easier to make grooves up on the fly when it's only four people.
The times I've been involved with a large band arrangement (dance camp, etc.) with 15-20 people on stage, we did make handwritten signs and it was someone's job to make decisions about when different things were going to happen, hold up the sign and cue the group. It worked fine, but it's inelegant conducting, imo. And it was someone's full-time occupation to drive the bus, so to speak.
I like Sarah's idea - it seems to make a lot of sense and be workable, and you don't have to worry about someone not happening to see a sign that the leader held up, etc. Thinking back to when I played a lot of percussion and did a lot of hand drumming, that was kind of how things worked in drum circles.
I hope some of this is helpful?
Meg