Sounds like what is happening is that a fast retrieve turns them on by mimicing a true fleeing baitfish. Perhaps a slow retrieve until they are either close or following, then try to rip the fly out of the water. It's what bitfish would normally do.
Another note: I checked out the pix gallery today and read a story from last year about a guy on the North Shore (sorry can't remember his name but it was a pic of a 31 lb bass) who sight fished with crab patterns. He was convinced that you retrieve a crab until you get a follow then STOP the retrieve, to mimic the bait trying to bury in the sand.
Interesting, very interesting.