I get this all the time, sometimes I actually thought I was making a detour and turns out I actually made the trip longer. So what you want to do at this point is look at the map. Mini + main map. What you'll want to do next is follow the paths, it usually leads down to the cave where the ferry are at.
Check out my awesome adventure at http://fantaji.com/-/5Bc
. If you don't know about Fantaji then you're missing out!