Updating PacketParser

[John Adams]

Since more folks are becoming interested in collecting and parsing using our tools, I thought it time to make a quick post about some common issues you may run into. The first and foremost being, incompatible data versions.

What is a Data Version?
The data stream that comes from an SOE server to the Live client needs to be compatible; ie., the client needs to understand what the server is telling it, in a specific order or structure. This is made possible by blocks of information being sent to and from the client as needed. The layout of these Structures are specific to a version of the Data Stream of any given client release. Simplified, if SOE patches the client and moves data around (adds or relocates a bit of data in the stream), the Data Version gets changed so the newer client knows to accept data ONLY from the new structures.

A very simplistic (and inaccurate) example

• Let's say Client version 1000 comes out, has a data version of 1, and it may look something like this:
  
  • PlayerName, PlayerClass, PlayerRace, PlayerGender
  A week later, SOE decides "oh, we need to add a `Strength` value to the player to track their current strength, so they add PlayerSTR to the structure:
  
  • PlayerName, PlayerClass, PlayerRace, PlayerGender, PlayerSTR
  This needs to become Client version 1001, Data Version 2, since now the packet for Player has a new field, and the client needs to consider it when fetching Player data from the server.

In this simple example you can kind of see why, when SOE updates the client, our server becomes incompatible, our parsings no longer work, and we have to wait til a developer decodes the new data stream and updates our opcodes and structs.


What are Opcodes and Structs?
Opcodes are nothing more than Operation Codes, some fancy term left over from the days when people felt the need to learn Assembly language But seriously, it is really that straight forward. The client/server use Opcodes to tell each other what the next action is.

Another very simple, and likely inaccurate example

• A player types /camp in-game. The client sends the Opcode 48 (OP_CampStartedMsg) to the server, so the server knows that this player is /camp'ing.

Most times, the server will acknowledge the opcode with another opcode saying it received the message, but this is basically opcodes in a nutshell. So how do Opcodes related to EQ2Emulator *Struct.xml?

While LE was off tramping around the Middle East one day, it came to him... how to make multiple clients compatible for the same emulator. As any of you old EQEmu fans know, you had 1 client, 1 server, that's it. Fanboys will bark that they had 3 clients!!1 but in fact, it was 1 - Titanium. For years. Still is that way, huh gang?

LE made our server in such a way that it can detect what "data version" your client is connecting with, and then load the appropriate "data structure" so your client can still connect - even if it's older. So anyone with an ROK box set (client 4412) can connect to our current emulator, which is compatible through client 5709!

I won't get into the details of how the structs work, since it's still pretty much a mystery to me - but if you open WorldStructs.xml, you can start seeing some of what I am telling you - the OPcodes, the data version, and the structs.


What does all this have to do with collecting and parsing?
You turn on the Collector, log into an SOE server, and you start collecting data. That data is placed in order received into a EQ2PacketLog.log file. When you are done collecting, you can open it and look inside, and what you'll see are blocks of hex/string broken into "Opcodes" and "Structs".

packetlog.jpg

In the above example, you can see in the OP_LoginRequestMsg packet, there are 2 bytes "AB 03". As explained to me, SOE uses "little endian", so you have to swap the bytes before decoding them (3AB) which is 939.

WAIT! WHOA! Hold on, how the hell did I go from AB 03 to 939?

Open your Windows (or other) calculator, and switch to Hex (I love pictures, don't you?)

calc1.jpg

Type in the AB 03 as 3AB (cuz you had to flip the bytes, remember)

calc2.jpg

and simply click Dec (for decimal). Voila, 939. Now you know what Data Version your entire log should be.

Knowing this, you can determind if you have the proper Opcodes and Structs to "decode" it.


How do I know what the latest Opcode versions are?
Run this command on your EQ2Emulator World database (that is the database your EQ2World.exe attaches to).

Code: Select all

select max(version_range2) from opcodes;

version.jpg

In this example, and as of this writing, the max "data version" we support is 973. Anything you collect that is HIGHER than that is INCOMPATIBLE - but, do not throw your logs away. The emulator is constantly being updated to support new features, and that includes new data structures.


Experimenting with new data versions/opcodes
Sometimes SOE makes very subtle changes to the data stream, maybe to add something to data we do not use currently. This means all our existing, known structures are intact. You can experiment with this by manually bumping the version_range2 of your opcodes tables (for PARSER ONLY!) to the new data version.

To do this, run this command on your PARSER database:

Code: Select all

update opcodes set version_range2 = {new_data_version} where version_range2 = {old_data_version}

Let's say SOE patches, changes our data version to 974, but the change does not effect our spawns, spells, or items structures. We would run this command:

Code: Select all

update opcodes set version_range2 = 974 where version_range2 = 973

and now parser can read the new data version.

Note:THIS WILL NOT WORK ON MAKING YOUR WORLD COMPATIBLE. Access to your worlds is governed by our Public Login Server, and if Login does not know about the new data version, you are denied access. So please, no questions on how to do this to make worlds compatible with new(er) clients.


How do I tell my Parser to use the newest Data Version?
Parser, much like World, uses Opcodes and the 3 Structs (xml) from the Config Files\World Structs folder on SVN. It's a simple matter of copying the 3 XML files you use for your World to the place where PacketParser.exe is stored.

The harder part (though still not hard to anyone who understands MySQL) is updating Parsers `opcodes` table. For this, you need to somehow export the current `opcodes` table from your EQ2Emulator WORLD database, then import that table to your parser DB.

How I do it is I use a GUI tool, like SQLYog. I open a connection to my database, highlight the eq2dev.opcodes table in the tree view, right-click on it, then Copy Table To Different Host/Database, pick the Parser database, and you're done. GUI's make things very simple... however, if you do not have a GUI, you can still achieve this using MySQL command lines.

Export:

Code: Select all

mysqldump.exe -h {your_mysql_ip} -u {mysql_user} -p {your_eq2emulator_db_name} opcodes > opcodes.sql

This will extract the opcodes table to the file opcodes.sql. Be sure to put in your own MySQL server IP, or leave -h param out if MySQl is on the same server as you are on. You need at least SELECT rights on the -u user name, and -p means "prompt me for a password" for that user. {your_eq2emulator_db_name} is the name of the database your EQ2World.exe uses.

Complete example dump command line:

Code: Select all

mysqldump.exe -h locahost -u root -p eq2dev opcodes > opcodes.sql

Import:

Code: Select all

mysql.exe -h {your_mysql_ip} -u {mysql_user} -p {your_parser_db_name} < opcodes.sql

This reads the exported opcodes.sql file into your parser database {your_parser_db_name}

Complete example import command line:

Code: Select all

mysql.exe -h locahost -u root -p eq2_rawdata < opcodes.sql

If there are any other questions related to PacketParser.exe and the DATA it uses to work, let us know and we'll add to this FAQ. Please don't ask me how to unravel the entire packet, as almost 2 years later I am still unsure how to do that myself.

[Scatman]

Nice post!

[John Adams]

Ugh... the reason I posted that is because people are asking about Parser, and I just realized we had LE so busy with crashes lately, he hasn't had time to update the new Database Structures in Parser, so -populate is not working.

I will take a crack at it this afternoon. If I am successful, you'll have a new parser to use.

It will parse the current logs fine, you just cannot get your data from RAW to DB using -populate ;( Sorry about that, totally forgot the table changes.

[LethalEncounter]

Nice post! Very informative, but I bet that took a while

[John Adams]

All these things I post... they are usually little "New Text Document(243).txt" files I started months ago and just found, and decided to finish it. So yes, this post took me MONTHS to write!

[ilythor]

Oh, am I glad you wrote that "update" up!

Nice post!!! it clearly has ~months of work~ beaming out of every orifice.