...only one drawback that I can see... You'd likely miss out on a LOT of nice sailing...
This winter for me was a perfect example... The best sailing I had, by far, was during the trips down and back... I had some really nice sailing down in the Keys, and a few days of perfect sailing in the Bahamas, but it's a bit depressing to imagine how much fantastic sailing I would have missed had I motored the 2,000 miles down and back on The Ditch, instead... Not to mention, I don't even want to think about what the final fuel tab would have been, with diesel now averaging well over $4/gal...
Funny, but 2 of the highlights of my cruise this winter were the first and final legs between Barnegat Inlet and the Chesapeake Entrance, and return... Kick ass sailing in early January behind a strong cold front, I just flew down the coast... Coming home, carried the spinnaker all the way from Cape Charles to Atlantic City, first time I've ever managed that, just a marvelous sail...
Seems a pity that so many cruisers view the trip south as an ordeal, and the ICW as little more than a means to an end...