User Name
Password

Go Back   Planetarion Forums > Planetarion Related Forums > Planetarion Discussions

Reply
Thread Tools Display Modes
Unread 17 Apr 2006, 03:29   #1
Shyne
Flash in the PAN
 
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
Shyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud of
New Fast Dump Processing Code

I've put together a bundle of my recently completed dump processing php scripts.

The slowest - the planet processor, completes in well under 2 seconds on my entry level webhosting.

Note that these automatically calculate ALL ranks as well.

Included is an sql file for the database, and the php code.

Download it at http://www.fxmy.co.uk/pa/dump.zip

The code is a tiny bit messy, and its not commented. If you arent proficient with php just make the tables and run it, and the data is in the mysql ready to go

Updates:

* Fixed a minor bug with alliance ranks.
* All 3 Dumps are backed up to a 'dumps' folder each tick. (Thanks Banned for the idea). Be sure to CHMOD777 a dumps folder or this will not work.

Last edited by Shyne; 17 Apr 2006 at 13:24.
Shyne is offline   Reply With Quote
Unread 17 Apr 2006, 10:07   #2
Banned
Banned
 
Banned's Avatar
 
Join Date: Jul 2003
Location: ******
Posts: 2,326
Banned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so little
Re: New Fast Dump Processing Code

MySQL

I've been playing around with the same recently, but in pl/pgsql. Mine takes about 7-8 seconds to run on my p3-700, with about half the time being spent generating ranks for planets. I might try to improve this (I'm quite sure it's possible), but at the moment I have other stuff I'd prefer to work on.

Sharing tech is I mean to release my tech stuff at some point, but really, it requires a little more than a normal shell account. It runs python/postgresql, and stores every dump.
Banned is offline   Reply With Quote
Unread 17 Apr 2006, 11:51   #3
Shyne
Flash in the PAN
 
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
Shyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud of
Re: New Fast Dump Processing Code

Well, the code should work fine on postgresql with little adaptation. If theres any demand ill convert it.

I know a lot of people even now are working on alliance tech applications and things, and seeing as I was working on this anyway I thought I'd share it.

If anyone has suggestions for improvements, let me know
Shyne is offline   Reply With Quote
Unread 18 Apr 2006, 22:49   #4
DJet
Wearer of The Hat
 
Join Date: Mar 2001
Location: Bedfordshire, UK
Posts: 252
DJet is a jewel in the roughDJet is a jewel in the roughDJet is a jewel in the rough
Re: New Fast Dump Processing Code

Any chance you can add the code for inc/include.php?

from my basic knowledge of php in guessing it defines the various do-dahs (technical term there) like LISTING_LOCATION


edit: thanks, something like this has been missing for a while!
__________________
- [ROCK] Retiree- www.rock-alliance.com - #rock -
DJet is offline   Reply With Quote
Unread 19 Apr 2006, 00:17   #5
Banned
Banned
 
Banned's Avatar
 
Join Date: Jul 2003
Location: ******
Posts: 2,326
Banned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so little
Re: New Fast Dump Processing Code

Quote:
Originally Posted by Shyne
* All 3 Dumps are backed up to a 'dumps' folder each tick. (Thanks Banned for the idea). Be sure to CHMOD777 a dumps folder or this will not work.
Code:
patools16=> select count(*) from planet_dump ;
  count
---------
 4476164
(1 row)
That's my r16 database

The way I figure it, if you're going to use a db, you should do something actually takes advantage of the db's primary purpose

This saving only one tick in the db bullocks is a bunch of hogwash.
Banned is offline   Reply With Quote
Unread 19 Apr 2006, 00:28   #6
Shyne
Flash in the PAN
 
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
Shyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud of
Re: New Fast Dump Processing Code

:O

My current applications will only have interest in the latest tick. I do obviously see the use of a history table, but perhaps would need to investigate the best way of doing so.

A 4.4m row table might be rather slow - but having never used one, I'll leave it to you to comment

I heard that sandman says flatfile storage would have made his site faster than the current mysql site, but that may be untrue.
Shyne is offline   Reply With Quote
Unread 19 Apr 2006, 01:10   #7
Banned
Banned
 
Banned's Avatar
 
Join Date: Jul 2003
Location: ******
Posts: 2,326
Banned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so littleBanned contributes so much and asks for so little
Re: New Fast Dump Processing Code

Quote:
Originally Posted by Shyne
My current applications will only have interest in the latest tick. I do obviously see the use of a history table, but perhaps would need to investigate the best way of doing so.
I don't actually use it for much in the way of history stuff. I do occasionally look up the old information. Plus I have a command for our IRC bot that shows score gained in the last 72 ticks. (So at worst the bot needs the last 72 ticks).

Quote:
A 4.4m row table might be rather slow - but having never used one, I'll leave it to you to comment

I heard that sandman says flatfile storage would have made his site faster than the current mysql site, but that may be untrue.
Flatfile will always be faster than a database application if you write the code right. With that said, the main bottleneck on our bot's speed is the IRC connection, which it throttles pretty heavily to avoid getting thrown off. I've occasionally run into problems when designing queries, but with the right indexes and structurings it's not a big deal.
Banned is offline   Reply With Quote
Unread 19 Apr 2006, 12:31   #8
Ramihyn
Emperor
 
Join Date: Jul 2001
Location: in front of a computer
Posts: 490
Ramihyn has much to be proud ofRamihyn has much to be proud ofRamihyn has much to be proud ofRamihyn has much to be proud ofRamihyn has much to be proud ofRamihyn has much to be proud ofRamihyn has much to be proud ofRamihyn has much to be proud ofRamihyn has much to be proud of
Re: New Fast Dump Processing Code

Quote:
Originally Posted by Banned
Flatfile will always be faster than a database application if you write the code right.
Jup, very much so. My own tools did use a packed binary data format stored in files and using a 512MB Ram Server meant it could have the whole 45000 ticks of a speedgame in memory or alternatively a whole round of PA with 188000 planets. As it allowed to create graphic charts of up to 9/10 different planets/galaxies/clusters showing roid/score growth in one dynamic chart on sometimes low-end hardware (300mhz servers), it needed to quickly access all data for some of those.

Quote:
Originally Posted by Banned
With that said, the main bottleneck on our bot's speed is the IRC connection, which it throttles pretty heavily to avoid getting thrown off. I've occasionally run into problems when designing queries, but with the right indexes and structurings it's not a big deal.
Thats why i gave up ordinary "query" connections to my old irc-bot interface. On networks like netgamers where it wasnt possible to run it as "service", i just told people to use the DCC chat interface instead. The flood restrictions on many irc networks are just too strict otherwise.
Ramihyn is offline   Reply With Quote
Unread 19 Apr 2006, 22:44   #9
Altor
Registered User
 
Join Date: Jul 2005
Location: Belgium
Posts: 30
Altor is just really niceAltor is just really niceAltor is just really niceAltor is just really niceAltor is just really nice
Re: New Fast Dump Processing Code

For xVx I only keep the tick%24==0 ticks
I use those to calc score gain/loss etc.

I'd post our scripts here as well, but I need to clean them up a bit first
And they calc anything you'd like and put it in db (score, size, value, xp diffs, uni, para, cluster, gala ranks for planet, gala, cluster, para, rank diffs etc)
You get tables with a lot of columns that way

It does it all with MySQL queries though, using temp tables
TheShadow wrote some perl code once to do it in memory, but it didn't get finished
Maybe when I have some time I'll see if I can make all the calcs in PHP... It'll probably be faster then doing it with queries
Altor is offline   Reply With Quote
Unread 20 Apr 2006, 00:01   #10
Shyne
Flash in the PAN
 
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
Shyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud of
Re: New Fast Dump Processing Code

Take a look at my code - personally I dont know a faster way of doing it (programatically at least). 2 seconds is adequate for me anyway, compared to my slower processor in the past.

Storing data at midnight is one way of doing it - statistics may tell you a best time of the day to take snapshots of planets
Shyne is offline   Reply With Quote
Unread 20 Apr 2006, 12:05   #11
Altor
Registered User
 
Join Date: Jul 2005
Location: Belgium
Posts: 30
Altor is just really niceAltor is just really niceAltor is just really niceAltor is just really niceAltor is just really nice
Re: New Fast Dump Processing Code

2 seconds is a lot faster then wat our script takes atm, but calculating the universe ranks is the fastest/easiest anyway, and I guess the time for those is also about 2 secs...
It's the calculation of planet ranks per galaxy etc that takes longest
And that part is also harder to do then just sorting the array on score etc...
There's probably a way, but I've been too lazy to find another one after I found a way to do it in a MySQL/PHP script
Altor is offline   Reply With Quote
Unread 20 Apr 2006, 12:24   #12
Shoshuro
Registered User
 
Join Date: May 2004
Posts: 134
Shoshuro is just really niceShoshuro is just really niceShoshuro is just really niceShoshuro is just really niceShoshuro is just really nice
Re: New Fast Dump Processing Code

Good code Shyne

Our dump parser stores every single tick and takes on the average 1.58 seconds to do the related stuff (php/mysql).
__________________
Omen
Shoshuro is offline   Reply With Quote
Unread 29 Jan 2007, 09:57   #13
Lockhead
Cabeza Coder
 
Join Date: Oct 2000
Posts: 212
Lockhead is a jewel in the roughLockhead is a jewel in the roughLockhead is a jewel in the rough
Re: New Fast Dump Processing Code

Nice code.

I like it

You are processing exiles, are you?
__________________
Lockhead
Developer, Solutions Architect, DevOps Engineer

lockhead.net

Quote:
Round 24 Conspiracy HC Comment at my planet
<Germania> 4.9.1
<Germania> hes our top hostile
Lockhead is offline   Reply With Quote
Unread 29 Jan 2007, 12:30   #14
Shyne
Flash in the PAN
 
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
Shyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud of
Re: New Fast Dump Processing Code

I'm glad all that stuff is working for you guys. The code should be fairly clear so you can add whatever information you like in there.

The ranks are calculated in memory using the sort function - it took a long time to get that working but its fast now.

As for processing exiles - I'm not sure what you mean. I did make something that logged new planets, and their out-of-protection tick though - bit mean eh !
Shyne is offline   Reply With Quote
Unread 29 Jan 2007, 13:06   #15
Heartless
CRASHING BEATS 'N FANTASY
 
Heartless's Avatar
 
Join Date: Mar 2001
Location: Cold Country.
Posts: 1,912
Heartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like him
Re: New Fast Dump Processing Code

Quote:
Originally Posted by Shyne
I'm glad all that stuff is working for you guys. The code should be fairly clear so you can add whatever information you like in there.

The ranks are calculated in memory using the sort function - it took a long time to get that working but its fast now.

As for processing exiles - I'm not sure what you mean. I did make something that logged new planets, and their out-of-protection tick though - bit mean eh !
What he wonders is whether your importer keeps track of all coords a planet has been at or not (and mines corresponding planet data accordingly).
__________________
Ią! Ią! Munin F'tagn! - [*scendancy]
Heartless is offline   Reply With Quote
Unread 29 Jan 2007, 23:35   #16
Shyne
Flash in the PAN
 
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
Shyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud ofShyne has much to be proud of
Re: New Fast Dump Processing Code

I dont - but you quite easily could.

The data is updated in that code, and the planets table always reflects the universe.
Shyne is offline   Reply With Quote
Unread 7 Feb 2009, 22:29   #17
Rio
Denial
 
Join Date: Dec 2008
Posts: 41
Rio is on a distinguished road
Re: New Fast Dump Processing Code

Sorry to dig up an old thread, but anyone know where i can get this file or something similar? Thnks
Rio is offline   Reply With Quote
Unread 7 Feb 2009, 22:47   #18
Heartless
CRASHING BEATS 'N FANTASY
 
Heartless's Avatar
 
Join Date: Mar 2001
Location: Cold Country.
Posts: 1,912
Heartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like himHeartless is an inspiration to us all and we should try to be more like him
Re: New Fast Dump Processing Code

Quote:
Originally Posted by Rio View Post
Sorry to dig up an old thread, but anyone know where i can get this file or something similar? Thnks
No problem digging this up. I don't know what Shyne did with that file, but if you are looking for a pa dump file importer, check out Munin:

http://github.com/andreaja/munin/
__________________
Ią! Ią! Munin F'tagn! - [*scendancy]
Heartless is offline   Reply With Quote
Unread 23 Apr 2011, 15:05   #19
dizzy
Registered User
 
Join Date: Jan 2011
Posts: 8
dizzy is an unknown quantity at this point
Re: New Fast Dump Processing Code

Does anyone have the said dump-read-script by anychance?
dizzy is offline   Reply With Quote
Reply


Thread Tools
Display Modes

Forum Jump


All times are GMT +1. The time now is 22:54.


Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2002 - 2018