Forum Settings
       
This Forum is Read Only

WOOT! I Give you Combat Log Parser!(kinda)Follow

#1 Oct 16 2010 at 4:43 PM Rating: Excellent
8 posts
(working on the coding as we speak)



So after spending about 14 hours figuring out where/how
data for in game logs is stored.

I figured out the weird logging method they have with 8 billion .Log files that can be opened with notepad.
once you do that as many people already know, your looking at logs, but its such a mess its almost painful to even think about sorting out the Non ASCII characters at the beginning and sometimes ends of the files as well as what seemed to be the random assignments of numbers and decimal/ colons after every line of a chat/combat window line feed

Well i sat and i looked because after trying to do a crap-shoot way of writing an LUA script for my G15 i saw a pattern...

I want to refer to these as Unique Hexadecimal Action Identifiers or "UHAID" basically the numbers you see in the .Log files after each combat line is parsed, well that is a reference to a specific set of actions to you, a PC, or an NPC whether it be buffs/debuffs/evades you name it

i wont have this up and running for a bit because i want to make sure it runs flawlessly as a .gadget sidebar app
and i still have to map out the rest of the UHAID, and im working on the code now to just read from the files and sort the codes for me, i haven't been in my programming classes in about 3 years and i don't program/design for a living so im a bit slow =)
here are a few so far.
42= all SP and EXP gain notices by you 
50=all my attacks that land 
51= all auto-attacks that mobs land on me, even crits / side attacks 
54 = monster readying special ability 
55= all friendly AND hostile attacks on/by npc's near me 
56 = all my misses Vs monsters as well as their evades vs me 
57= all misses vs me 
6D = status affects being inflicted on monsters near you AND players 
61= players other than me casting heals on themselves/PC's  as well as HP absorb messages 
67 = appears to be buffs/debuffs on players that have just been removed 
69= status effects just being inflicted upon me via monsters 
005c= so far it shows everytime i drain health with lancer speed surge 
20 = "ready (inswert combat skill)..." as well as loot obtain


Note: excuse my horrible Post syntax, im kind of excited and exhausted at the same time, and Spell checker can only help me so much when im this tired /lol


If anybody has experience with desktop gadgets and LUA scripts and feel they could speed the process along feel free to PM me here, thanks!

Edited, Oct 16th 2010 6:53pm by akill3s

Edited, Oct 16th 2010 6:56pm by akill3s
#2 Oct 16 2010 at 4:45 PM Rating: Decent
Scholar
***
1,104 posts
Sounds great, will definitely be keeping an eye on this. Thanks for all the hard work and effort so far.
____________________________

Melaahna Valiera
#3 Oct 16 2010 at 4:55 PM Rating: Decent
Scholar
**
626 posts
i will deffinately use a parser if created, good going man!
____________________________

#4 Oct 16 2010 at 5:02 PM Rating: Excellent
**
723 posts
A parser is definitely needed. You can check who is doing the most damage and remove them from the party since that is counteractive to gaining the most SP. I'm kidding of course (Or am I?), thank you for the effort you're putting into this.
#5 Oct 16 2010 at 6:18 PM Rating: Good
Thanks for the hard work!

Where are these files in the game folders?

Also, are you talking about Windows Vista/7 sidebar gadgets?
____________________________
#6 Oct 16 2010 at 6:28 PM Rating: Good
Sage
***
1,675 posts
There was this thread:

http://www.zam.com/forum.html?game=268&mid=128647768828613162#msg128663461519908734

Maybe this could help you out, or be some sort of basis?
#7 Oct 16 2010 at 6:39 PM Rating: Excellent
Anterograde Amnesia
Avatar
*****
12,363 posts
Good stuff. Not to deter you, but this won't be easy to put into a gadget without some kind of standalone app parsing the logs. I could incorporate this into my G-series app if you want and give you credit. If not I'll respect that and do my own research on this when the time comes.

Gadgets run on HTML, JavaScript, and CSS primarily. There are ways to bring .NET code into a gadget, but I'm not sure how robust it is for what you want to do. Also, I've heard the buffer on the logs is 60-80 lines, so it's not exactly a real time thing without going directly to the memory, which I won't do.

What languages do you have experience with?
____________________________
"Choosy MMO's choose Wint." - Louiscool
The greatest trick the devil ever pulled was to convince the world he didn't exist.
Keyser Soze - Ultros
Guide to Setting Up Mumble on a Raspberry Pi
#8mitmystria, Posted: Oct 17 2010 at 12:14 AM, Rating: Sub-Default, (Expand Post) ban hammer with this. square is parsing you as long as your logged in, go in ffxiv config and look at your system info it shows everything. and who knows whats hidden from our view.
#9thehellfire, Posted: Oct 17 2010 at 12:30 AM, Rating: Sub-Default, (Expand Post) Thinks SE will be banning him soon...
#10 Oct 17 2010 at 2:40 AM Rating: Good
**
532 posts
I have broken down the binary header format (its not random!) of the FFXIV log files over at FFXIVcore, fell free to take a look:

Here:
http://www.ffxivcore.com/topic/12124-ffxiv-log-file-analysis/page__view__findpost__p__256893

In more detail:
http://www.ffxivcore.com/topic/12124-ffxiv-log-file-analysis/page__view__findpost__p__257708

Hope this helps!
____________________________
Formerly: Iaito on Odin (XI)
Currently: Adorthan on Selbina (XIV)
#11 Oct 17 2010 at 6:08 AM Rating: Decent
Scholar
***
1,017 posts
If SE does anything they should encourage this. It does not give ANYONE an advantage over anyone else. It can only benefit the game and the players. With this we can learn more about the game by gathering data more efficiently and I cant see a reason why SE would not want that.

PS: you ******* rock.
____________________________
-Character Name: Djinnrb (old)
-Server: CaitSith
-Jobs:BLU75 BRD75 THF67 PLD 44 DRG 44 DNC39 NIN37
----------------------------------
-Character Name: Lightpalm
-Asura
-Jobs: WAR54 WHM53 RNG35 SAM70 RUN70
----------------------------------


http://ffxiv.zam.com/wiki/FFXIV_Leveling_Guide

Mithsavvy wrote:
Everything Square Enix does puts out a vibe that says, "I was programmed by someone who read C++ for Dummies after obtaining my degree in MIS"
#12 Oct 17 2010 at 11:53 AM Rating: Decent
8 posts
hmm laito i like your work there

I don't plan on sorting it on the fly just using some lua functions compatible with g15 that i can io.read(line) for now,
Then once ill get the g15 all set i want to take a stab at a gadget
#13 Oct 17 2010 at 11:57 AM Rating: Default
Sage
**
447 posts
kind of off topic, but there's no development SDK for the game right? that is, no potential 3rd party (or even 1st party) add ons?
____________________________
Djigga, please. Highland Hyurs can't jump.

#14 Oct 18 2010 at 8:27 AM Rating: Good
**
532 posts
I'll consider making a Java library JAR for reading the logs. I could also do C#, or C++.

Need to make a googlecode or sourceforge project first, though.
____________________________
Formerly: Iaito on Odin (XI)
Currently: Adorthan on Selbina (XIV)
#15 Oct 18 2010 at 8:34 AM Rating: Excellent
Anterograde Amnesia
Avatar
*****
12,363 posts
C# would be good, I could use that in my G15 app I already have written.
____________________________
"Choosy MMO's choose Wint." - Louiscool
The greatest trick the devil ever pulled was to convince the world he didn't exist.
Keyser Soze - Ultros
Guide to Setting Up Mumble on a Raspberry Pi
#16 Oct 18 2010 at 9:22 AM Rating: Excellent
*
211 posts
DjinnRB wrote:
With this we can learn more about the game by gathering data more efficiently and I cant see a reason why SE would not want that.


lol.. Well I guess you don't know SE very well then. If they want us to thoroughly understand something about the game, they will release the info on it. Some things they do want us to know. Others they want to leave shrouded in mystery, and have players gradually discover them. Gradually... which doesn't mean a couple people break the game into a science and figure it all out.

They have good reasons for this too. When there is some mystery surrounding the specific mechanics, people just play the jobs, wear the gear, and use the abilities that they want to use. If things aren't working they try something else. In other words, they have fun, which believe it or not is an alien concept to some gamers.

Now I've seen what happens when the "best" is figured out. You end up like 2 or 3 jobs, pieces of gear, and abilities that every last person uses. It's like watching a bunch of clones go to war with another bunch of clones. I played WoW for a couple months, and that game was so late in it's lifetime, it was plagued by that problem.(not to mention design flaws with the endgame which compounded the problem)

And here is the worst part... Once the "best" is figured out, everything else becomes the "worst", and the elitist nerds start proclaiming what's okay and what's not okay, sucking all of the fun out of the game and ostracizing anyone who doesn't have a cookie cutter build. Again, I'll point to WoW to illustrate the worst case scenario... For anyone who played it, remember gearscores? How about "dps meters"... People ****** and moaned about it so much it was easy to forget you were in a casual mmo. It was literally bad enough to completely ruin the limited PvE endgame that was there, and it turned the PvP into a way too serious nerd feud. I actually went out of my way to reduce my dps when I was on top, because I really didn't want to be in that top bracket while good players got chewed out for not playing the game according to rigid formulas. Not to mention people completely lose sight of jobs that offer utility or survivability, and just get tunnel vision about raw dps.

I just thought I'd post my 2cents on the issue. I won't use any programs like this, but I'm not saying others shouldn't. I just want to stress that you shouldn't take things too seriously. Like really, if someone does 10% less damage over the course of 3 hours, big deal. If someone isn't using the most efficient sp gain combo of abilities possible, oh well... And for crying out loud, there had better not be another "lol drg" era because of some idiots observation of a couple poor players.

On the other hand, maybe I'm just taking players who take the game too seriously too seriously, lol... So take it with a grain of salt... Just don't forget to play for fun, that's the point of the game after all. I just hate to see people get out of hand with this stuff. Even if the people in this thread don't, others will.
____________________________
PvR fun!
http://www.youtube.com/watch?v=-KYqooGHd2g




#17 Oct 18 2010 at 10:19 AM Rating: Decent
Scholar
***
1,151 posts
I don't think I would ever admit that it took 14 hours to figure out where the game logs are located at.

If you want a perl based parser look here
http://www.ffxivcore.com/topic/11874-pseudos-ffxiv-log-parser/

If you want more information about parsing Iaito response has another good thread from core.

My current parser (The perl one is not mine and I have nothing to do with its' development) can do pseudo real time. I am always one file back from the one SE is using. It does have a dps component but to use it requires setting up a bunch of echo macros.
#18 Oct 18 2010 at 10:37 AM Rating: Decent
Scholar
***
1,104 posts
Tiger228 wrote:
...If they want us to thoroughly understand something about the game, they will release the info on it. Some things they do want us to know. Others they want to leave shrouded in mystery, and have players gradually discover them. Gradually... which doesn't mean a couple people break the game into a science and figure it all out... They have good reasons for this too...
There is NEVER any good reason for concealing relevant ingame mechanics. All that accomplishes is allowing broken systems to persist - because if nobody knows how something works, they cant tell if its NOT working. Transparency also allows users to test mechanics and frees up the dev team to actually create content.

The only reason SE has EVER had to not reveal game mechanics was to hide something that they didn't want the general gaming populace to know was ***-busted-broken and/or malfunctioning, and as we're seeing there is a LOT of that in XIV.

Tiger228 wrote:
...I've seen what happens when the "best" is figured out. You end up like 2 or 3 jobs, pieces of gear, and abilities that every last person uses.
Where TRUE game balance exists, this isn't an issue whatsoever. Sure min/maxers will always look for that extra 1%, but if your game is balanced then 1% isn't going to make or break any class or encounter. I doubt SE has the slightest notion about balance.

I personally want to know how accurate my character is, or how much of an improvement that latest gear upgrade really was, or be able to tell exactly where I need to improve my character's statistics. That requires data.

Theorycraft doesnt just apply to damage and healing output, it works for crafting also. Its an essential part of any game and simply can't be ignored just because you dislike being held to a standard.
____________________________

Melaahna Valiera
#19 Oct 18 2010 at 4:02 PM Rating: Good
**
532 posts
Another reason for these types of tools is curiosity.

For example, in Closed Beta I got a Conjuror to 20ish. From 10-20 I had access to the Radiance ability. And honestly it seemed like it rarely ever hit, even with a more level appropriate weapon. I started planning on parsing the logs because I wanted to know how often it hit and much mana it returned on average.

Not every tool "that makes a game about math" is intended to be used as a method of comparing one player to another, or one class to another. Some are, in fact, created for discovery. Let's say that you think dexterity affected the accuracy of ability X, but ability X is a Conjuror ability. You added some dexterity while leveling another class and ability X started hitting more often, but you weren't sure how much. How would you find out? The scientific method.

1. Use your experience.

Ability X misses quite a lot, and when you added some additional dexterity it /seemed/ to hit more often.

2. Form a conjecture.

Dexterity affects ability X's accuracy by some amount.

3. Deduce a prediction from that explanation.

Ability X gains 50% of the melée accuracy that you gain from dexterity.

4. Test.

This is where the log parsers come in. Testing requires being able to reproduce the conditions of the experiment in a reliable manner. It also requires you to have a sufficiently large data set. A log parser would allow you to go kill things for an hour, check the hit/miss/etc ratios for ability X, then respec your stats or otherwise adjust your attiribues and then go kill *the same things* for an hour and compare the results.

Result:

You test a non-dexterity build verses a dexterity heavy build while grinding on moles for an hour each. You find no significant increase in your accuracy from the dexterity heavy build. You can then conclude one of two things: ability X's accuracy is not affected by dexterity, or, it's accuracy is also affected by a >= amount of another stat (that you may have decreased to create your dexterity heavy build).

Obviously all of the above is simplified/hypothetical/etc...

TL;DR: Not every theorycrafting tool is used for evil. ;)

Edited, Oct 18th 2010 5:09pm by Iaito
____________________________
Formerly: Iaito on Odin (XI)
Currently: Adorthan on Selbina (XIV)
#20 Oct 21 2010 at 10:06 AM Rating: Decent
11 posts
Hello, here it is my Alpha version of FFXIVBreport, a program im doing , that in future will support much more stuff, right now its only kinda barebones, displays chat log, basic damage count, nothing too advanced and still buggy, its just to show now, in a near future will work much better :)

Just put some name in the box and YOUR char id, you can see your char id on:

C:\Users\YourUsername\Documents\My Games\FINAL FANTASY XIV\user\02699E4E\

Its those 8 digits after \User , if you have more then 1 user, choose the one u play the most, should be the one with most LOG files inside the log folder, put that in the program and press ADD, then press Start Log, and play the game, Currently it has some delay in appearing in the program, just play some time and it will appear.

Any Suggestions? im open for them....

The Program: http://www.sendspace.com/file/fknh31
#21 Oct 21 2010 at 2:45 PM Rating: Decent
Scholar
***
1,017 posts
are you still working on this?
____________________________
-Character Name: Djinnrb (old)
-Server: CaitSith
-Jobs:BLU75 BRD75 THF67 PLD 44 DRG 44 DNC39 NIN37
----------------------------------
-Character Name: Lightpalm
-Asura
-Jobs: WAR54 WHM53 RNG35 SAM70 RUN70
----------------------------------


http://ffxiv.zam.com/wiki/FFXIV_Leveling_Guide

Mithsavvy wrote:
Everything Square Enix does puts out a vibe that says, "I was programmed by someone who read C++ for Dummies after obtaining my degree in MIS"
#22 Oct 21 2010 at 3:18 PM Rating: Decent
11 posts
DjinnRB wrote:
are you still working on this?


if thats to me, yeah, i will be for some time
#23 Nov 06 2010 at 6:05 PM Rating: Decent
Scholar
**
626 posts
sooooo hows it comen along?

just hoping this guy isnt one of the many that actually left the game.

Edited, Nov 6th 2010 8:05pm by pixelpop
____________________________

#24 Dec 04 2010 at 1:57 PM Rating: Decent
2 posts
Raiding life in wow was .........wow.
#25ghosthacked, Posted: Dec 04 2010 at 2:01 PM, Rating: Sub-Default, (Expand Post) good: all we need now is gearscore and we're ready to rock!
#26 Dec 04 2010 at 2:36 PM Rating: Decent
32 posts
Now we can parse those 10 second fights
#27 Dec 04 2010 at 4:34 PM Rating: Good
**
696 posts
ghosthacked wrote:
good: all we need now is gearscore and we're ready to rock!


fekkin number crunchers: ruining MMOs since...the inception of mmos?


Um... mmos are nothing but 1 gigantic bunch of math problems. To know anything about the game you have to know the math. What was your complaint? I mean really? We'll skip the 20 pages I could write about how ignoring these things is the reason a player sucks at a game and leave it at simply that.
#28 Dec 04 2010 at 5:04 PM Rating: Decent
Scholar
**
322 posts
maxmad wrote:
Now we can parse those 10 second fights


Hahahah xD
____________________________


#29 Dec 04 2010 at 10:45 PM Rating: Decent
*
158 posts
zoltanrs wrote:
ghosthacked wrote:
good: all we need now is gearscore and we're ready to rock!


fekkin number crunchers: ruining MMOs since...the inception of mmos?


Um... mmos are nothing but 1 gigantic bunch of math problems. To know anything about the game you have to know the math. What was your complaint? I mean really? We'll skip the 20 pages I could write about how ignoring these things is the reason a player sucks at a game and leave it at simply that.


I mean "number crunchers" ruin MMOS by enabling idiots who abuse the stuff they create.

And about sucking, lol. I'm sort of like upper-end of average. I'm not **** enough to go psychotic over bullcrap parses/statistics...I used to--but it's horrible having to deal with the clowns that abuse these tools.

I research my classes/jobs before I even begin playing them -- I know the drill. I know the optimal gear bullsh*t when I'm actively playing an MMO: I just don't care about the environment these types of addons create in MMOs: it's all hostile.

People are so ******* paranoid about under-performing, they become hostile towards everyone who even slips up a bit, they judge people by numbers, -- as if numbers alone encompass the ability and potential of a player in an MMO. That one-dimensional bullsh*t is so limiting.

I remember playing MMOs like warcraft before idiots ruined it with their addon rape: we used to clear content just fine, and no one was scrutinizing everyone else like the ******* statistics police, or some sh*t. Back then, it was more about the community, and the game/storyline than measuring up e-penises with other one-dimensional clowns.

But I wouldn't expect the lot of you to understand that. You're all the victims of an entire culture that has grown from, *ahem*, "inadequacies."

A key part of being human in this life is about overlooking "flaws" or "mistakes" you, or anyone else may have, or make--and focusing on the greater goal. You'd be amazed how a person can "improve" when you actually have faith in them, instead of waiting for them to slip up.


Edited, Dec 5th 2010 12:21am by ghosthacked
#30 Dec 06 2010 at 5:35 PM Rating: Good
****
6,898 posts
ghosthacked wrote:
zoltanrs wrote:
ghosthacked wrote:
good: all we need now is gearscore and we're ready to rock!


fekkin number crunchers: ruining MMOs since...the inception of mmos?


Um... mmos are nothing but 1 gigantic bunch of math problems. To know anything about the game you have to know the math. What was your complaint? I mean really? We'll skip the 20 pages I could write about how ignoring these things is the reason a player sucks at a game and leave it at simply that.


I mean "number crunchers" ruin MMOS by enabling idiots who abuse the stuff they create.

And about sucking, lol. I'm sort of like upper-end of average. I'm not **** enough to go psychotic over bullcrap parses/statistics...I used to--but it's horrible having to deal with the clowns that abuse these tools.

I research my classes/jobs before I even begin playing them -- I know the drill. I know the optimal gear bullsh*t when I'm actively playing an MMO: I just don't care about the environment these types of addons create in MMOs: it's all hostile.

People are so ******* paranoid about under-performing, they become hostile towards everyone who even slips up a bit, they judge people by numbers, -- as if numbers alone encompass the ability and potential of a player in an MMO. That one-dimensional bullsh*t is so limiting.

I remember playing MMOs like warcraft before idiots ruined it with their addon rape: we used to clear content just fine, and no one was scrutinizing everyone else like the ******* statistics police, or some sh*t. Back then, it was more about the community, and the game/storyline than measuring up e-penises with other one-dimensional clowns.

But I wouldn't expect the lot of you to understand that. You're all the victims of an entire culture that has grown from, *ahem*, "inadequacies."

A key part of being human in this life is about overlooking "flaws" or "mistakes" you, or anyone else may have, or make--and focusing on the greater goal. You'd be amazed how a person can "improve" when you actually have faith in them, instead of waiting for them to slip up.


There are a **** of a lot more uses for some of these applications than just stroking ones epeen. Lumping them all together is just absolutely ignorant. Sometimes the utility of these is a GODSEND in the face of poor programming or UI (think external macros in ffxi, where only allowing 6 lines made swapping full sets of gear an absolute chore, or recast to know exactly when timers were up).

While I agree that some addons do encourage elitist behavior, you really can't lump them all into this category. Also, making comments about an entire culture being "victims" is a quick way to get an immediate rate-down from me. I can't stand when people try to make assumptions about an entire group of people, it just makes you look arrogant and ignorant.
____________________________
Bartel Hayward--- Ultros Server
The Kraken Club <ZAM>
50 WAR • 50 MNK • 50 MIN • 50 GSM • 50 ARM • 50 LTW • 50 CUL • 50 WVR
thekrakenclub.shivtr.com
#31 Dec 06 2010 at 6:40 PM Rating: Decent
*
158 posts
BartelX wrote:

There are a **** of a lot more uses for some of these applications than just stroking ones epeen. Lumping them all together is just absolutely ignorant. Sometimes the utility of these is a GODSEND in the face of poor programming or UI (think external macros in ffxi, where only allowing 6 lines made swapping full sets of gear an absolute chore, or recast to know exactly when timers were up).

While I agree that some addons do encourage elitist behavior, you really can't lump them all into this category. Also, making comments about an entire culture being "victims" is a quick way to get an immediate rate-down from me. I can't stand when people try to make assumptions about an entire group of people, it just makes you look arrogant and ignorant.


I'm not arguing against the fact that there are "other" uses for these tools, I'm just saying that from what I've seen over the years in MMOs, the majority of players misuse the tools--they always, always do.

I'm not against add-ons. I use addons readily in MMOs that allow them, but I don't support the use of addons--particularly "parsing" and other statistical type addons by infantile minds.

And about the rate-down thing? You think I give two ***** about rating? I don't speak to gain favor with anyone.

#32 Dec 06 2010 at 6:57 PM Rating: Decent
***
2,261 posts
If you like addons, then your rant would be better placed in a thread where people are performing the actions you're describing.
#33 Sep 08 2011 at 5:39 AM Rating: Decent
Avatar
3 posts
Even if I dont agree with Ghost entirely, I dont think that rating down someone with a coherent argument or point of view is appropriate. Rating down is for the morons that post comments like "F*ck yeah" and "WTF... anyone know where I could find some bot code"
____________________________


Necro Warning: This post occurred more than thirty days after the prior, and may be a necropost.
This forum is read only
This Forum is Read Only!
Recent Visitors: 16 All times are in CST
Anonymous Guests (16)