The ferry takes 45 minutes to an hour one way. If the water is rough or choppy, there will be a LOT of folks who get seasick. Have to look at what time you get into port and what time back on-board time is and plan accordingly.
Forget about swimming in the ocean. Cold water, strong currents and wicked waves. The locals don't even go in. Now on the gulf side, way much better, just have to put up with the constant stream of vendors on the beach.