I've dove/snorkeled that rubble a few times. I was diving with a spearfishing buddy of mine a couple years ago in March and it was covered in sheepshead. My buddy speared a couple decent sized ones and tried to hand off the stringer to me since I wasn't carrying anything. Of course I folded my arms and shook my head emphatically considering the size of a few sharks I've seen around that area before. If anyone knows Walter (the approx. 12foot great hammerhead) then they would have done so also.
The rubble is only about halfway down the pier, so it sounds like you guys where too far out. It's only 12-16ft deep, depending on tide, but there should be sheepies on it this time of year. Did you see any birds diving on the horizon? There should be little tunny out there.
Alex
P.s. Check your PMs.