PacketCollector Changes?

[unknwon]

Seeing how data collecting is need for the project go forward, how`s about we change the way it works.

instead of having the packet collector log all the packets and then having to run the parser to pull out the mob/npc information. How about we have it search a local database for the NPC/Mob and if not found to just add the information into the database on the fly.

This would reduce the guess work as to which mobs/npcs have been collected. Than all we have to do is everyone run around the game either eq2 or eq2x with the new packet collector running in the background, it will than update the local database with any new mobs/npc, after which we can than sync up the changes to a central database server.

i have the basic working on my custom packet collector now, it shouldn`t be much to change the eq2emu collector as (both use the same orginal source code from Acid). minus the radar server application i added onto it

So basically, you start the collector, it sync the local database from the central server, you launch eq2, run around the zones as it search`s for new mobs/npcs and updates the local database, after you finished gaming, the you than sync up the local database the central server which adds the changes.

This is just an idea to help the project get going again, now with the eq2x coming out, everyone will have access to collect the need information from the soe servers.

[Zcoretri]

We don't really need more collecting, what we need is more c++ coders to work on the emulator.

I don't see how your system is gonna work...so you have a collector and a parser going at the same time writing to a database?

[John Adams]

Zcoretri, I see what he's saying... and yes, we do need more collecting Since we never know when the current collector will be rendered useless again (and there is no more LE to fix it, does anyone else know how? no), I'd like to get 100% of the spawns collected and in a database for our own use populating this game. We won't use most of the data before it becomes obsolete, but as I said, once this collector stops working - and it will - we'll at least not have empty zones.

As for his idea... I get what he's after, but I think he's forgetting that the collector will collect the same exact NPC everytime you run passed it - regardless of weather or not it's currently in your database. Also, the collector (because it only see's SOE-provided spawn configs) cannot determine if it is a single spawn using random appearances (randomize) or ranges in level/difficulty set once again on a single spawn parent, just appears differently. It also cannot determine the structure of a spawn "party" - multiple mobs that work together to kick your ass.

The way we do it now is cumbersome, but far more accurate. You cannot automate this process, and expect to emulate the true Live experience, imo. Robots are for building cars. Let's keep human decision involved in building our games.

But unkwon, if you want to build this tool, definitely do so. I can't release Collector sources, but you could get it working with what you have, and offer it to the community who doesn't care about emulating Live perfectly, as we do.

[unknwon]

The way i see this collector working, is it checks the mob`s/npc`s ID (these don`t change) against the database, if the ID doesn`t exist it then adds it to the database, if it finds the id in the database it skips the ID.

if they is enough people who what this, i`ll look at completing the custom collect and release it here, if that okay

I believe the biggest problem collecting data with the old Collector, is that it didn`t like Zone changes, so users were asked to collected data in one zone at a time and stop and logout after each collection. Mine has no problems with zone changes, and you can even start the tool after you login to EQ2.

Just imagine how much faster we could collect data if we have users just gaming/running around normally and not worrying about staying in one zone. Most of my gaming night`s i may visit 10 zones in 4 hours.

I could also release the source code for others to help with the coding, minus the encrypting key routines. I agree that no one should release any source code that has the encrypting routines, as it`s way too easy to create a key logger for passwords once you have those routines.

[John Adams]

The way i see this collector working, is it checks the mob`s/npc`s ID (these don`t change) against the database, if the ID doesn`t exist it then adds it to the database, if it finds the id in the database it skips the ID.

My understanding (from years ago) is that everytime an NPC spawns in EQ2 Live, it has a unique spawn ID. It is not *our* spawn ID value, of course... but some seemingly random number (entity_id?) that is likely incremental.

When "a rat" spawns, it has an id of 1234. When you kill it, and that placement spawn respawns, it is now 1235. etc.

I may be wrong, or things may have changed, but that's how it was explained to me as to why we could not use SOE's spawn ID's to test for uniqueness.

And none of that negates all my other reasons why collecting "automatically" will not work for the core project, or valid emulation of EQ2 Live. But, it might get worlds populated pretty quick and dirty.

[unknwon]

are we sure there is no common mob id, i know from program on the wow side, there was a GUID (INT64) that had a random number and the spawn id.

Also from the testing i`ve done, spawns on one server match the mob/npc spawns on the other.

Code: Select all

199.108.13.56 -> 192.168.10.4
0000:	01 37 [b]80 00 00 00[/b] ff 1f	02 28 08 35 56 01 ff ff	.7.......(.5V...
0010:	00 00 00 00 7b 8a 53 8a	00 06 00 61 74 74 61 63	....{.S....attac
0020:	6b 00 40 1c 46 01 08 35	56 01 c7 5f a9 54 33 00	k.@.F..5V.._.T3.
0030:	00 00 f0 05 dc 5c ee 06	ee 19 38 5f c5 d9 17 01	.....\....8_....
0040:	9f 31 c0 31 c0 ff 81 04	ee 54 c0 02 25 3f dd e7	.1.1.....T..%?..
0050:	01 e9 50 6e bb 72 db 40	03 2d 87 01 a8 a5 7f 7f	..Pn.r.@.-......
0060:	7f 7f 7f 7f 0b 11 00 61	20 66 72 6f 7a 65 6e 20	.......a frozen 
0070:	67 72 61 70 70 6c 65 72	00 00 00 00 00 00 00 00	grappler........
0080	00 00 00 00 00 03                               ......

other server..

Code: Select all

199.108.12.56 -> 192.168.10.4
0000:	01 37 [b]80 00 00 00[/b] ff 1f	02 0e 2f c2 3b 00 ff ff	.7......../.;...
0010:	00 00 00 00 fb 56 05 f3	00 06 00 61 74 74 61 63	.....V.....attac
0020:	6b 00 40 1c 46 01 2f c2	3b 00 44 a0 1e 48 33 00	k.@.F./.;.D..H3.
0030:	00 00 f0 06 c0 a2 ee 4a	f0 19 93 b4 cc c3 ff f1	.......J........
0040:	df 02 01 d5 01 d5 ff 08	f7 18 a8 80 05 b3 23 ee	..............#.
0050:	b3 23 07 3d fb dd fb 18	a8 80 05 83 07 3d 7f 7f	.#.=.........=..
0060:	7f 7f 7f 7f 0a 11 00 61	20 66 72 6f 7a 65 6e 20	.......a frozen 
0070:	67 72 61 70 70 6c 65 72	00 00 00 00 00 00 00 00	grappler........
0080	00 00 00 00 00 03                               ......

The bold Hex number matches on both servers, i`ve checked it on other mobs/npcs too. this seems to be a global spawn id??

also do we have an updated Spawn structure for live, i have all the structure mapped i needed for my radar program, but not enough to create the spawn entries in the database...

I miss LE ..

[Zcoretri]

also do we have an updated Spawn structure for live, i have all the structure mapped i needed for my radar program, but not enough to create the spawn entries in the database...

You will find structs for spawns in the SpawnStruct.xml file.

Now for the packets you posted...the section you tried to bold (doesn't work inside code tags) is the packet length.
The next 3 bytes make up the opcode FF 1F 02 (543) which is the OP_EqCreateGhostCmd opcode.
The next byte is an Index
The next 4 bytes are the spawn ID, so as you can see, they are different 08 35 56 01 vs. 2F C2 3B 00.

I miss LE too

[unknwon]

guess i`ll have to do some more digging on the structure, i didn`t think the current spawn structure was up to date for Live servers? i have a feeling that one of the unknown values is the guid of the spawn. I~ll do some dumps this week and see if i can find a link between two servers. All other mmo out there have a guid of some type.

By the way, i readed that SOE will be releasing an API that will allow access to some of the information on players, spells, mobs, etc. That would help with the emulator project, i think.

[John Adams]

Unknwon, I am not 100% positive about the random entity_id's, but that's the last thing I remember LE and I discussing about spawns, which was over 2 years ago. Things may have changed, but I doubt it. Definitely let us know if you find out any differently.

Re: our Spawn structs - they seem to rarely change, so what we have now should be pretty close to Live, since I just parsed some 1046 live data last week and the spawns look fine. There may be some additional detail I'm not getting, but generally speaking they are good.