I'm staying on Durandal too, as a FFXI player I appreciate the maturer community you'll find on a JP server and have made many awesome JP friends both in FFXI and FFXIV. I played WoW for a bit and really, I don't look forward to going to something like that again, way too many immature kids shouting their mouths off.
If you wanna be more serious and go for the hard content, then yes the latency is a good reason, I have Titan HM on farm now only because I memorized everything so I have an extra 500ms head start on the dodge. It's a challenge I'm willing to live with and lots of players respect that I do well despite my latency. But I'm a casual player so I'm fine not getting the best gear, I like to just hang out and be laid back with nice folks, hardly care if I get nothing done all day!
I loved how in FFXI you could dedicate a day to crafting or questing, and hey you still can! And latency has nothing to do with that, now imagine trying to do the same thing with kids running around shouting all day, well then I'd have to turn shout off and that kinda detracts from the experience of being in a community. Then you pop into a DF group and get elitists who pick on every little thing you do wrong, maybe that's the problem, everything in WoW was always rush rush rush to end game and end game gear, with JPs it feels more casual, we're just here to have fun and get along!