Forum Settings
       
This Forum is Read Only

Log Parser and WebsiteFollow

#1 Feb 06 2011 at 2:59 PM Rating: Good
Scholar
26 posts
I just wanted to let everyone know I created a log parser along with a website that tracks stats and shows some statistics and graphs for battles and character information. So far the list includes:

Character Stats with:

Total Hours Logged (This is actually the amount of hours that were recorded by the logger)
Number of Deaths
Min, Max and total damage broken down by class
Graphs showing Exp / SP Gains over time broken down by class.

A battle log containing:

Monster - The name of the monster that was logged.
Class - The class your character was using when the monster was killed.
Hit % - Hit percentage based on number of misses over the total number of hits.
Ttl Dmg - The total damage delivered to the monster.
Ttl Avg Dmg - The average damage across both regular and critical attacks per hit.
Crit Avg Dmg - The average damage across critical hits.
Reg Avg Dmg - The average of normal attacks (non-critical) to the monster.
Ttl Dmg Taken - The total damage taken from the monster.
Ttl Dmg Taken Avg - The average over both critical and normal attacks by the monsters to you.
Crit Dmg Taken Avg - The average over critical hits performed by the monster to you.
Dmg Taken Avg - The average damage taken by you for normal (non-critical) hits.
Exp - Experience awarded for this monster.
Skill Pts - Skill points awarded for this monster.
Value - A calculated worth based on damage dealt, received and exp/sp awarded.
Date/Time - The date/time of the log file that contained this fight. because the logs do not have a time the file time is used instead.

I have quite a few more additions that I would like to add but I think its now at a point where it can be announced.

You can find it at: www.ffxivbattle.com

A sample character display at: http://www.ffxivbattle.com/character.php?charactername=0xc55f7c74fL
The list of logged battles are at: http://www.ffxivbattle.com/characterbattle.php

I hope everyone enjoys it as much as I do. Please let me know if there are any statistics you want to see or graphs you would like. There are quite a few things I want to do still that are not quite finished yet. Crafting and Gathering are being collected by the log parser but I don't have the database setup for them quite yet. I think the script can parse about 95% of all messages from the logs now.

* Important *
By Default the log parser generates a unique ID but you can put whatever name you like. This is so you can remain anonymous if you like since the log parser never collects a single character name from in game from you or others. Very soon I will add a password so others don't collide with your data should you want to use a custom name.

Best,
H3lls
#2 Feb 06 2011 at 3:21 PM Rating: Default
42 posts
Shouldnt all attributes be listed to? Link to archer level 1: killed 5 "x-rats" min atk:Y max atk:z level 2: ... etc. if you understand what i mean ^^
#3 Feb 06 2011 at 3:25 PM Rating: Decent
Scholar
26 posts
I would love to have the ability to get the current rank. The only time you can tell that a rank up happened is when it actually occurs. This presents a problem when you are doing continuous parsing of the logs. I am considering it though.

I had an idea to just log the rank up / level up events in the database then later stitch those into the reporting based on the date/time it happened. I will see if I can get that working as it would make the graphs and min/max stats really interesting.
#4 Feb 06 2011 at 10:24 PM Rating: Decent
****
6,898 posts
I think this is a really good start. I'd really like to see averages added in, like show average accuracy/damage/damage taken/etc. for an SP session, unless I missed that somewhere there and if so I apologize. I'd like to know average damage/battle, average acc/battle, etc. over the course of a session also. It is a very good start though and I commend you on it.

Edit: Eventually, parses of other characters in the party would be great too, so that you can get damage comparison data, and % of damage per battle/party. Stuff like that, and sorry, these ideas just keep comin to me.


Edited, Feb 6th 2011 11:52pm by BartelX
____________________________
Bartel Hayward--- Ultros Server
The Kraken Club <ZAM>
50 WAR • 50 MNK • 50 MIN • 50 GSM • 50 ARM • 50 LTW • 50 CUL • 50 WVR
thekrakenclub.shivtr.com
#5 Feb 06 2011 at 10:42 PM Rating: Decent
*
120 posts
Make a windower for FF14! XD
#6 Feb 06 2011 at 10:56 PM Rating: Default
**
602 posts
SWEET! Now would someone prove to me that DEX does SOMETHING?
____________________________
FFXI: Dashiel. (Asura) Puppetmaster.
FFXIV: Majidah Sihaam. (Besaid)
Marauder, Weaver & Alchemist.

#7 Feb 07 2011 at 12:45 AM Rating: Decent
18 posts
Not bad, I think the site is a little confusing in navigation and such.

For parsing realtime in game I think apps like Crabby is much better as the ddl for stat parsing is open source so you can parse anything you wish.
____________________________

#8 Feb 07 2011 at 2:09 AM Rating: Decent
Scholar
26 posts
Quote:
I think this is a really good start. I'd really like to see averages added in, like show average accuracy/damage/damage taken/etc. for an SP session, unless I missed that somewhere there and if so I apologize. I'd like to know average damage/battle, average acc/battle, etc. over the course of a session also. It is a very good start though and I commend you on it.

Edit: Eventually, parses of other characters in the party would be great too, so that you can get damage comparison data, and % of damage per battle/party. Stuff like that, and sorry, these ideas just keep comin to me.


on the character page if you click on the View Fight Data link it brings you to the battle page. You can click on the names of each monster in the table to see a breakdown of each fight. I need to improve the visibility of the links to make it easy to see that. I will also add the averages for that fight on the fight detail page as well.

I plan to add a graph that shows your damage, your hits taken, others damage, others hits taken over the last few fights overlayed on one. This way you can quickly see the last few fights and how well you did in comparison to thers. I don't plan to track the other users char names in case SE decides it would be cool to ban people for using log parsers so its just generic data. I think I can show 5-10 fights but im not sure since it will be about 200-500 plot points.

Quote:
Not bad, I think the site is a little confusing in navigation and such.

For parsing realtime in game I think apps like Crabby is much better as the ddl for stat parsing is open source so you can parse anything you wish.


I agree on the navigation I will see what I can do to make it a better flow overall. While Crabby is great it also is doing memory reads from the app which is something that I don't want to do. I don't have any moral objection to how Crabby works but I wanted this to just stick with the logs. I have about 95% of the log messages figured out and parsed it is more a matter of displaying it correctly on the website. I am also thinking of nice things to do with the UI but that will come later.


Thanks for all the ideas, I hope to have some more additions every week or so. It has an auto-updater when there are changes to make life easy when new features are available.
#9 Feb 07 2011 at 2:10 AM Rating: Good
*
211 posts
I can't decide whether to love or hate those parser things...

On the one hand, they provide a lot of very useful information that can be used to improve your ability to play your role well.

On the other hand, a very ugly thing happens when you mix elitist players and DPS meters of any sort... So I hope anyone who uses those things takes them with a grain of salt. Especially if you ever use those things as a measuring stick between players... Raw damage isn't everything after all.

It's also nice to see people trying a variety of things instead of just going with cookie cutter methods that parsed the highest.

But either way, good job on the thing. I've no use for it, but it can be a good thing to have around.
____________________________
PvR fun!
http://www.youtube.com/watch?v=-KYqooGHd2g




#10 Feb 07 2011 at 3:36 AM Rating: Good
Scholar
26 posts
I rearranged things a bit and made it a bit easier to find the different pages and made the links look like links (that will teach me to start with a template). I also linked the monsters you have killed at the bottom so you can see each fight with that type of monster.

Next step is the damage graphs which are a bit more scary but that's for tomorrow maybe.

Quote:
On the one hand, they provide a lot of very useful information that can be used to improve your ability to play your role well.

On the other hand, a very ugly thing happens when you mix elitist players and DPS meters of any sort... So I hope anyone who uses those things takes them with a grain of salt. Especially if you ever use those things as a measuring stick between players... Raw damage isn't everything after all.


I agree with this completely. I originally wrote the parser as a way of trying to compare monsters to find out which give the highest rewards without having to think too hard about it. It looks like a few come out on top. I like to reduce the time I spend grinding so for me this was the best way to do that. From there it grew into something more fun. Like you said, it can be a nasty thing for some folks when endgame comes around or trying to form the *Perfect* party.

Thanks for the great feedback everyone!
#11 Feb 07 2011 at 3:15 PM Rating: Decent
18 posts
Keep up the good work, going to check it out in a week and see your progression.
____________________________

#12 Feb 07 2011 at 3:31 PM Rating: Decent
*
219 posts
Really cool!

I'd like to see a further breakdown of damage done and received based on skills and spells.
____________________________

#13 Feb 08 2011 at 11:05 AM Rating: Decent
Scholar
26 posts
Just about finished with the Japanese localization then I will add the damage graphs. Great Idea for damage / attack type! I will break it down and do min, max for each attack type. It would be cool to have a graph of each fight showing the min / max damage for each attack type. I was thinking of doing a few mini-graphs that pop-up from a list when you click so you can view more detailed data.
#14 Feb 08 2011 at 2:16 PM Rating: Decent
Scholar
35 posts
In your breakdown of individual mob kills, like:
http://www.ffxivbattle.com/characterbattle.php?charactername=0xc55f7c74fL&monstername=antelope%20doe

It would be nice if you could group by job/SP gain. I think that would help a lot of people when they're running tests, they could grind on a single mob type for hours, then be able to turn it all into useful comparisons without having to add up the separate rows themselves.

I also think a lot of people might like the pre-calculated values from these columns, before averaging. # of hits, misses, evades, crits. Total regular damage, crit damage. Total WS damage (which you'd want to breakdown somewhere else).

That gets to be a lot of numbers, so there are certain data sets that make sense together, that make it easier to read the information you want, when you want it. It might improve readability if the information is categorized, and displayed as needed. Maybe with checkboxes that are tied into the table to hide a subset of columns.

Sort of like:
Outgoing Damage (Raw values, # of hits, misses, damage amounts based on type)
Outgoing Averages (Calculates averages and percentages)
Incoming Damage (Raw values, # of hits, misses, damage amounts based on type)
Incoming Averages (Calculates averages and percentages)
Non-combat Values (Exp/Sp/Value/Time)

Also, given that the site will be mainly for reading these values and seeing charts and stuff, I suggest a template/theme with a larger width for content.
#15 Feb 08 2011 at 2:29 PM Rating: Decent
***
1,313 posts
I want recount for my FFXIV.

Flame on.
____________________________
Eithne Draocht
My IG: archaicmachinery - Friend me!
#16 Feb 08 2011 at 2:31 PM Rating: Decent
***
1,608 posts
Doesn't the stamina system make it impossible to gain accurate combat data? I wouldn't find it feasible compared to parsers of FF11, as that combat system was automatic.
____________________________

#17 Feb 08 2011 at 3:37 PM Rating: Decent
Scholar
35 posts
LyleVertigo wrote:
Doesn't the stamina system make it impossible to gain accurate combat data? I wouldn't find it feasible compared to parsers of FF11, as that combat system was automatic.


What would stamina make less accurate?
#18 Feb 08 2011 at 3:54 PM Rating: Decent
Scholar
26 posts
Vapes these are great ideas. I will try to do a bit of clean-up on the battle list page as it was written in an hour or two just to give a quick debug of the database code. I will add in more interesting filters and create a dropdown that lets you select the type of table being shown. So far the only thing I am using on that chart is the "Value" column. It calculates the overall worth of a monster so you can find out which monsters are best when grinding. Next step is to make it better for those switching weapons to optimize hits or armor to optimize damage resistance. I want to see the results of doing a number of respecs so this is something I have really wanted also.

Recount is great and I could get all of the same values (Except DPS) if I recorded the other people in the party. I was thinking of doing DPM(inute) since the logs write every minute if you are continuously fighting.

I am hesitant to record other characters names you are playing with but I would love to hear your opinion. I am concerned that people may freak out that they are being recorded and shown. This may just me being overprotective though. I do already record each hit type and it would be trivial to gather the character that is hitting / being hit.

Edited, Feb 8th 2011 4:55pm by h3lls
#19 Feb 08 2011 at 4:26 PM Rating: Decent
***
1,608 posts
Vapes wrote:
LyleVertigo wrote:
Doesn't the stamina system make it impossible to gain accurate combat data? I wouldn't find it feasible compared to parsers of FF11, as that combat system was automatic.


What would stamina make less accurate?


Since stamina is manually operated DPS can vary greatly from person to person. Much more so than merely misuse of TP.
____________________________

#20 Feb 08 2011 at 5:58 PM Rating: Default
Scholar
*
88 posts
What about a character isn't operated manually? You been botting?
____________________________


DaevaofWar.com
#21 Feb 08 2011 at 6:22 PM Rating: Decent
Scholar
26 posts
I think stamina allows you to make personal changes to the flow of skills to optimize your attacks. This is one of the factors that can help your DPS. For example my conj fighting nannygoats gets knocked back 3 times or so in a fight. I found that if I start casting cure as soon as this happens the spell delay gives the stamina bar time to fill up almost perfectly enough to get in a fire before the goat comes charging in. Small optimizations can make a big difference overall. I think if you are looking to optimize for solo and party managing your stamina in the same way for every fight based on the monster type and what your party consists of makes a big difference.
#22 Feb 08 2011 at 6:45 PM Rating: Decent
Scholar
35 posts
LyleVertigo wrote:
Vapes wrote:
LyleVertigo wrote:
Doesn't the stamina system make it impossible to gain accurate combat data? I wouldn't find it feasible compared to parsers of FF11, as that combat system was automatic.


What would stamina make less accurate?


Since stamina is manually operated DPS can vary greatly from person to person. Much more so than merely misuse of TP.


Most of the data gathered currently is based on how hard you hit, how accurate you hit, how often you crit, etc. Stamina doesn't affect those values. Stamina is in a way, a function of time, and since the logs don't output very accurate timing data, most things aren't currently being calculated over time, just sticking to the basics.

Once more people are sitting at cap, I'm sure the focus will shift towards "what order of attacks is optimal" etc. Then you'll need to bring stamina into the mix, but that information would, most likely, best be obtained through genetic algorithms, not logs.
#23 Feb 10 2011 at 10:16 AM Rating: Decent
Scholar
35 posts
The new width looks good.

As far as group parsing and privacy you could replace the names with placeholders. You could just replace everyone with greek gods for instance. So the names would all map properly and it probably wouldn't take much effort on the side of the people who were actually in the fight to figure out which placeholder they are. Especially since you'd have your job identifier. Just a simple "Well, there are two archers, and.... I fired Foeseeker off as my first WS... I must be Ares."
#24 Feb 10 2011 at 12:30 PM Rating: Decent
Scholar
26 posts
I came up with a good idea for the character names last night. I am working to add a password so others cannot collide with your user. What I am thinking of doing is to allow users to log in to the character to be able to display the other people in the party. That way only someone who has access to their own data can see the individual users. I love the idea of using greek / roman gods for name replacements on the anonymous browsing though. I will add that on the next update of the client, just doing some testing of the password functionality and doing a bit of cleanup from a code review. Later I could allow you to hand out a link with an embedded code that lets others view fights or date ranges of fights so you could hand out the link to the party so they can view for 24 hours or something.

Edit:

If I do this I can also send chat logs so you can search your previous chat logs. I will need to build out a full text search but I was wanting to do this anyway.

Edited, Feb 10th 2011 2:02pm by h3lls
#25 Feb 11 2011 at 11:31 PM Rating: Decent
Scholar
26 posts
The new version of the parser is available. It should automatically download / install it for you if you launch a recent version. I also made quite a few changes to the detail page to show more pertinent data for fights. I will be adding more but it is a start. This version includes a password that you can log in to your character with. It creates the password if you haven't had one before.

I made the fight detail page return Greek Gods and Titans if you are not logged in to your character.

Here's an example:

http://www.ffxivbattle.com/fightdetail.php?id=862e2068cf650ca9ff1567e6c1451921

If you uploaded data before it will need the newer version before it starts showing the names. The new version will update the existing records you already sent so as long as the logs aren't too old or overwritten it will update.

Please let me know how I'm doing so far with the latest stats. My next goal is to do some better graphs and data over the last few fights so you can decide at a glance who is the best / worst in a party and how you are doing in comparison to others. Another thing that will likely come at the same time is healing. Right now healing is not recorded.
#26 Feb 12 2011 at 8:55 AM Rating: Decent
Scholar
35 posts
In your example, based on damage done, and misses, etc. It doesn't look like the user's character is included in the graph, or the total/average break down below it? Is that the case? It might be useful to compare yourself along with everyone else.
#27 Feb 12 2011 at 1:11 PM Rating: Decent
Scholar
26 posts
I later added a line graph to show the user to quickly show a comparison. I am trying to get it to map across the entire graph rather than start at the 0 position. Unfortunately I don't record the other party member hits in the same sequence as the logger. I am thinking of added a sequence number to each entry so I know the order in which they come up in the logs. Basically a 0-infinity number that starts with the first log entry and increments forever. It's not as nice as having a date time but would help quite a bit for several different situations.

Now that I feel the password stuff is working I am going back to do some major cleanup on the code so it can easily support other languages. Right now it is very English specific.

I will be adding a call to lodestone to get the current rank of the class you are using. This will make for some much more interesting results so it can be tied to rank and stats. To ease the burden on the lodestone site it will make this call infrequently in the background but should give a good general rank / stats to each battle.
#28 Feb 13 2011 at 12:24 AM Rating: Decent
Scholar
26 posts
There was a bug with the 2.2 version that would prevent you from submitting your password correctly. This has been fixed now and the DB cleaned up so update to fix the problem. Once you launch it should auto-update.

I have quite a few new changes for the fight details page. I found a novel way of determining the class of the other people in your party. It does a check for the type of attack being used and ticks up a count for that class. The class that has the most counts in a fight is the assumed class. It isn't perfect but it seems to work quite well in most instances. The trouble would be if someone were on Marauder and never hit but instead stood back and healed the whole time it would assume you were a conjurer. Past that it is shaping up nicely.

Solo Fight: http://www.ffxivbattle.com/fightdetail.php?id=862e2068cf650ca9ff1567e6c150cf13
Two Member Party: http://www.ffxivbattle.com/fightdetail.php?id=862e2068cf650ca9ff1567e6c1458a88
Small Party: http://www.ffxivbattle.com/fightdetail.php?id=862e2068cf650ca9ff1567e6c14ffd2c
Small Behest: http://www.ffxivbattle.com/fightdetail.php?id=862e2068cf650ca9ff1567e6c145e062

I fixed all of the issues where it wasn't including the main logging char in the rest of the information so it should now show everything integrated.
#29 Mar 04 2011 at 3:51 AM Rating: Good
Scholar
26 posts
Just wanted to drop a note saying I added support for Japanese in the log parser and website. It will handle parsing for English and Japanese without any trouble. I also added in a feature request page and quite a few fixes for the entire site.

Thanks for the feedback here previously it has really helped shape the growth. There are now about 13 thousand battles logged so it is very interesting. I plan on doing some higher level stats. With all of the changes going in things are going to be interesting keeping up with SE but I have a feeling I can keep ahead of them.

Also on the fight detail page I added in some of the Zam hover thinggies for each ability and will be adding more. Does anyone know if there is a nice list of all of the abilities and IDs associated so it will be easier to set this up?
This forum is read only
This Forum is Read Only!
Recent Visitors: 15 All times are in CST
Anonymous Guests (15)