VAfamcrusin

I cannot address how easy or difficult this would be with special needs, and of course depends on the specific needs, but I have absolutely done this and is my number one recommendation for St Thomas. Here's how we have done it:
 
1) Get off the ship ASAP, grab the first taxi, take it to the ferry. Know the schedule of the return ferries, and ask your driver to meet you back there so you don't need to wait for any taxi when you get back. Have his number (assuming you're American, most phone companies treat US Virgin Islands like any other state so calling/texting is easy and included) just in case.
2) Ferry across, grab a taxi on the other side to the beach of your choice.
3) Reverse the trip for the way back. Make sure there's at least one more ferry that can get you back to the ship on time in case there's any issue...so for example (just making up times here) if you know you need to be back on the ship by 7:30pm, and there's a ferry that drops you back to your taxi at 6:00pm, leave on the one that gets you there at 5:00pm...knowing the 6:00pm ferry is there, just in case).
 
With a nine-hour port day, we have been able to spend four hours easily on the beach at St. John, plus stopped for lunch and beers before the return ferry, and returned to the ship over an hour before "all aboard".

My son has autism and adventure ocean is very accommodating. We were on harmony last October and have to say the Adventure ocean staff there was the best and most accommodating of all the ships we’ve been on, but to be fair all the ships we’ve been on we’ve had great experiences with adventure ocean. The first day go in to adventure ocean and ask for the special needs form. You can also discuss any concerns you have with them then. You can also call the special needs department at 1(866)592 7225 and can setup pier assistance and a special muster drill in a quite and non crowded space. 

My 5 year old daughter has significant behavioral/emotional/social issues, to the point she goes to a special school.  Adventure Ocean was great with her.  I explained to them (in tears, terrified they were going to turn us away) what her issues are, and the staff was amazing.  Each time she went (we never sent her for very long and someone always stayed close just in case) they gave us a phone so they could call us if she had any issues.  They contacted us quickly, before it turned into a huge issue, when she "didn't like her pants" "was cold" - things that wouldn't be a big deal for other kids, but they recognized could become a problem for her.  And the last night of the cruise, when we decided to push our luck and let her stay longer because her twin sister was doing the pajama party, they were FABULOUS when she had a complete break down and couldn't be controlled.  There was no judgement, they didn't get mad, they just helped me catch her, contain her, and get her out without any issues.
My one complaint is that the phones don't work very well.  They would call, but we couldn't hear anything most of the time.  The phone would ring and it would be silence or static - each time we just ran there quickly to see what the issue was.
AND they got her to not only participate in the talent show, but they helped her wait patiently for her groups' turn (the little kids all did the chicken dance, it was adorable).  I was fully expecting to be in charge of her until it was her turn to go up, but they handled her and all the other kids very well.