You're doing something wrong. Even though I stayed inshore on Saturday, my friend at the dock boated at least 14 fish. It was because of him that I was in that area on Sunday. I only got six, but I only fished for about an hour and a half, and got very late. My friend also told me that the fog was thick out there on Saturday, but on Sunday it cleared once you were around 8 miles off.
Listen, go out to 650/775, set out a spread of small or regular unwieghted green machines, clones, small jets, bars, or anything else you have confidence with and continue trolling south. Forget zig - zag...... Keep it straight south unless you see fish, birds or nervous water.....otherwise just go straight at 6.5 - 7.0 knots, or what ever speed you need in the prevailing conditions that make your spread "smoke".....no jumping.....just smoking.
On the first knockdown, hit the save button, land your fish, and stay within that area unless there are no fish.....and always keep your eyes peeled.
Quite frankly, you should have caught them. Perhaps you were not far out enought. Remember, going due south from the point puts you on the numbers. But most inexperienced guys get spooked when they hit Cartwrights, which is around 4 miles off. The water gets a bit messy there, and many inexperienced guys turn back thinking it's gonna get worse. It gets much better in fact, and flattens out within a mile or so. Once out in Butterfish hole area.....which is where those numbers are, you should have fish all over you if you are fishing your spread correctly.