The trick is that highly goal-driven people don't want to play with fun-driven people, and fun-driven people don't want to be excluded from play or play with people who harass them for their performance.
You're making the assertion that "fun" and "lacking direction" are the same thing, which is false. "Fun" is subjective, so it can't be directly attributed to one thing or the other.
The issue here is between having direction vs. lacking it. In this case, you'd be right: people who have clear-set goals in mind generally don't want to play with people who lack them, which is fine. At the same time, people with no direction don't want to be excluded or harassed for not having goals, which is also fine. Ultimately though, only one of these sides is going to be successful in high-end endgame content; guess which one, and plan accordingly.
The pie in the sky solution here for everyone to do their best, be goal-driven if you absolutely must, but keep it in perspective. Even goal-driven people are -attempting- to have fun (though they tend to fail somewhat. Setting goals to achieve fun is generally a red herring versus the goal of having fun), but they should ideally learn how to have fun even when they're not making the most satisfactory progress towards their goals. When they don't, at the very least their own fun suffers, and often the fun of the more "casual" (not unskilled) players as well.
You're equating "casual" with "lacking direction" as well, which is also a false assertion. "Casual" simply means that you've less time to spend playing; it doesn't imply that you're not hardcore enough.
I'll agree that people's fun can be diminished by focusing too hard on the goals, but I wouldn't call it a red herring. If anything, the red herring is the assumption that "fun" and "goals" are mutually exclusive things.
Personally I just don't plan to party with people who impose performance expectations on others (like it's a social expectation, rather than a team effort), even if I have to party with sh*tty players. Don't care if the party lives and dies by you. If you depend on people meeting a certain level of performance to have fun-- later. I depend on people not being needy and self-absorbed to have fun, so that'll just be where we differ. And I'll have more fun than you. But hey, maybe your character could beat my character in a fight, which is always an impressive achievement on the playground.
There's two sides to this: on the one side, you have people that are trying to achieve goals as a group, and on the other you have people who are just trying to accomplish their own goals and don't care about the group as a whole. You seem to be leaning in the latter direction, which is fine. Just don't assume that the other side is being greedy or self-absorbed because they aren't adapting the group to meet the needs of the one outsider; if anything, having that expectation makes the outsider
the greedy or self-absorbed one. Don't assume you'll have more fun, either; the path you're planning on treading sounds lonely, after all.