Gotta remember, when the game first came out in japan, nearly TWO years before it came out in the states, the level cap was 50. AF armor did not exist, the only AF piece you had at the time was an AF weapon.
They then raised the level cap to 55, and a while later to 60. It was during the level 60 cap raise they introduced AF gears. At the time people playing the game were less intelligent, less min/maxing, less demanding.
The AF sets were not made to maximize damage, and as time progressed, people got better, more knowledgeable, and learned how mechanics worked.
People realized which stats were 'useless' and which were important for weapon skills and dps. This is not unique to FFXI, wow had the EXACT same problem. The tier gear(afs) were poorly itemized. As people got smarter they realized off spec gear which is more specialized was actually better. Subsequent tiers were better itemized and specialized, and blizzard went back and changed tiier1/2. Again, FFXI did something similar and introduced AF+1. More intelligently itemized gear.
FFXI did the same thing. AF2 was introduced shortly after the final level cap of 75. If you notice, it is MUCH better itemized and set up. SE learned that people knew how mechanics worked and was required to allocate items to meet the demands of the player base.
Now, as more expansions came out, new forms of end game gear arrived. No longer is it AF tailored for one class, but tailored for groups of similar classes. This reduces rng and allows for less wasted loot. Similarly, wow developed the same thing around the same time. Tokens with 3-4 classes were added to the same token to reduce waste. However, wow continued the one tier per class set up.
For FFXIV, I think SE is better prepared this time around. They went away from the 1 AF rng drop from dynamis' and went towards the 'token' system. This time I am certain, AF pieces will be better itemized because in general MMo players are MUCH more knowledgeable and more min/maxing.
MMO playing has changed, and FFXI was slow to adapt to the changes. Recently, they realized this and have been adding these changes but things are still missing. Transparency and item budgeting will be required this time around. A level 7 item should not have a better item budget than a level 62 item. It makes no sense at all.
Since FFXI has moved away from the one endgame armor set per class in recent expansions, it is fairly safe to assume they will incorporate something similar in FFXIV. People responded well to token systems. I don't think they will appear late in the games life as in FFXI, but instead be there from day one.
I am certain AF gear will make a return. However, if there will only be one AF set or multiple we do not know. I will gaurentee you there will be multiple endgame sets though.