Since this isn't the Spoiler section, I won't outright spoil anything.

*****SPOILER*****
Here's a Bloodmoon map. It's not very helpful, but the general layout is correct. If you use the face of the castle as a guide, the entrance is nearly in line with the wall. Just follow that line to the coast and dive. The door will be a little to the left of the line of the castle face. See Below, x= cave entrance. CK= the castle.
____x_______________
[|||||]
[CK|]
[|||||]