Updating PacketParser
Posted: Thu Jul 02, 2009 8:45 am
Since more folks are becoming interested in collecting and parsing using our tools, I thought it time to make a quick post about some common issues you may run into. The first and foremost being, incompatible data versions.
What is a Data Version?
The data stream that comes from an SOE server to the Live client needs to be compatible; ie., the client needs to understand what the server is telling it, in a specific order or structure. This is made possible by blocks of information being sent to and from the client as needed. The layout of these Structures are specific to a version of the Data Stream of any given client release. Simplified, if SOE patches the client and moves data around (adds or relocates a bit of data in the stream), the Data Version gets changed so the newer client knows to accept data ONLY from the new structures.
A very simplistic (and inaccurate) example
What are Opcodes and Structs?
Opcodes are nothing more than Operation Codes, some fancy term left over from the days when people felt the need to learn Assembly language
But seriously, it is really that straight forward. The client/server use Opcodes to tell each other what the next action is.
Another very simple, and likely inaccurate example
While LE was off tramping around the Middle East one day, it came to him... how to make multiple clients compatible for the same emulator. As any of you old EQEmu fans know, you had 1 client, 1 server, that's it. Fanboys will bark that they had 3 clients!!1 but in fact, it was 1 - Titanium. For years. Still is that way, huh gang?
LE made our server in such a way that it can detect what "data version" your client is connecting with, and then load the appropriate "data structure" so your client can still connect - even if it's older. So anyone with an ROK box set (client 4412) can connect to our current emulator, which is compatible through client 5709!
I won't get into the details of how the structs work, since it's still pretty much a mystery to me - but if you open WorldStructs.xml, you can start seeing some of what I am telling you - the OPcodes, the data version, and the structs.
What does all this have to do with collecting and parsing?
You turn on the Collector, log into an SOE server, and you start collecting data. That data is placed in order received into a EQ2PacketLog.log file. When you are done collecting, you can open it and look inside, and what you'll see are blocks of hex/string broken into "Opcodes" and "Structs".
In the above example, you can see in the OP_LoginRequestMsg packet, there are 2 bytes "AB 03". As explained to me, SOE uses "little endian", so you have to swap the bytes before decoding them (3AB) which is 939.
and simply click Dec (for decimal). Voila, 939. Now you know what Data Version your entire log should be.
Knowing this, you can determind if you have the proper Opcodes and Structs to "decode" it.
How do I know what the latest Opcode versions are?
Run this command on your EQ2Emulator World database (that is the database your EQ2World.exe attaches to).
In this example, and as of this writing, the max "data version" we support is 973. Anything you collect that is HIGHER than that is INCOMPATIBLE - but, do not throw your logs away. The emulator is constantly being updated to support new features, and that includes new data structures.
Experimenting with new data versions/opcodes
Sometimes SOE makes very subtle changes to the data stream, maybe to add something to data we do not use currently. This means all our existing, known structures are intact. You can experiment with this by manually bumping the version_range2 of your opcodes tables (for PARSER ONLY!) to the new data version.
To do this, run this command on your PARSER database:
Let's say SOE patches, changes our data version to 974, but the change does not effect our spawns, spells, or items structures. We would run this command:
and now parser can read the new data version.
Note:THIS WILL NOT WORK ON MAKING YOUR WORLD COMPATIBLE. Access to your worlds is governed by our Public Login Server, and if Login does not know about the new data version, you are denied access. So please, no questions on how to do this to make worlds compatible with new(er) clients.
How do I tell my Parser to use the newest Data Version?
Parser, much like World, uses Opcodes and the 3 Structs (xml) from the Config Files\World Structs folder on SVN. It's a simple matter of copying the 3 XML files you use for your World to the place where PacketParser.exe is stored.
The harder part (though still not hard to anyone who understands MySQL) is updating Parsers `opcodes` table. For this, you need to somehow export the current `opcodes` table from your EQ2Emulator WORLD database, then import that table to your parser DB.
How I do it is I use a GUI tool, like SQLYog. I open a connection to my database, highlight the eq2dev.opcodes table in the tree view, right-click on it, then Copy Table To Different Host/Database, pick the Parser database, and you're done. GUI's make things very simple... however, if you do not have a GUI, you can still achieve this using MySQL command lines.
Export:
This will extract the opcodes table to the file opcodes.sql. Be sure to put in your own MySQL server IP, or leave -h param out if MySQl is on the same server as you are on. You need at least SELECT rights on the -u user name, and -p means "prompt me for a password" for that user. {your_eq2emulator_db_name} is the name of the database your EQ2World.exe uses.
Complete example dump command line:
Import:
This reads the exported opcodes.sql file into your parser database {your_parser_db_name}
Complete example import command line:
If there are any other questions related to PacketParser.exe and the DATA it uses to work, let us know and we'll add to this FAQ. Please don't ask me how to unravel the entire packet, as almost 2 years later I am still unsure how to do that myself.
What is a Data Version?
The data stream that comes from an SOE server to the Live client needs to be compatible; ie., the client needs to understand what the server is telling it, in a specific order or structure. This is made possible by blocks of information being sent to and from the client as needed. The layout of these Structures are specific to a version of the Data Stream of any given client release. Simplified, if SOE patches the client and moves data around (adds or relocates a bit of data in the stream), the Data Version gets changed so the newer client knows to accept data ONLY from the new structures.
A very simplistic (and inaccurate) example
- Let's say Client version 1000 comes out, has a data version of 1, and it may look something like this:
- PlayerName, PlayerClass, PlayerRace, PlayerGender
- PlayerName, PlayerClass, PlayerRace, PlayerGender, PlayerSTR
What are Opcodes and Structs?
Opcodes are nothing more than Operation Codes, some fancy term left over from the days when people felt the need to learn Assembly language
But seriously, it is really that straight forward. The client/server use Opcodes to tell each other what the next action is.Another very simple, and likely inaccurate example
- A player types /camp in-game. The client sends the Opcode 48 (OP_CampStartedMsg) to the server, so the server knows that this player is /camp'ing.
While LE was off tramping around the Middle East one day, it came to him... how to make multiple clients compatible for the same emulator. As any of you old EQEmu fans know, you had 1 client, 1 server, that's it. Fanboys will bark that they had 3 clients!!1 but in fact, it was 1 - Titanium. For years. Still is that way, huh gang?
LE made our server in such a way that it can detect what "data version" your client is connecting with, and then load the appropriate "data structure" so your client can still connect - even if it's older. So anyone with an ROK box set (client 4412) can connect to our current emulator, which is compatible through client 5709!
I won't get into the details of how the structs work, since it's still pretty much a mystery to me - but if you open WorldStructs.xml, you can start seeing some of what I am telling you - the OPcodes, the data version, and the structs.
What does all this have to do with collecting and parsing?
You turn on the Collector, log into an SOE server, and you start collecting data. That data is placed in order received into a EQ2PacketLog.log file. When you are done collecting, you can open it and look inside, and what you'll see are blocks of hex/string broken into "Opcodes" and "Structs".
In the above example, you can see in the OP_LoginRequestMsg packet, there are 2 bytes "AB 03". As explained to me, SOE uses "little endian", so you have to swap the bytes before decoding them (3AB) which is 939.
WAIT! WHOA! Hold on, how the hell did I go from AB 03 to 939?
Open your Windows (or other) calculator, and switch to Hex (I love pictures, don't you?)
Type in the AB 03 as 3AB (cuz you had to flip the bytes, remember)and simply click Dec (for decimal). Voila, 939. Now you know what Data Version your entire log should be.
Knowing this, you can determind if you have the proper Opcodes and Structs to "decode" it.
How do I know what the latest Opcode versions are?
Run this command on your EQ2Emulator World database (that is the database your EQ2World.exe attaches to).
Code: Select all
select max(version_range2) from opcodes;Experimenting with new data versions/opcodes
Sometimes SOE makes very subtle changes to the data stream, maybe to add something to data we do not use currently. This means all our existing, known structures are intact. You can experiment with this by manually bumping the version_range2 of your opcodes tables (for PARSER ONLY!) to the new data version.
To do this, run this command on your PARSER database:
Code: Select all
update opcodes set version_range2 = {new_data_version} where version_range2 = {old_data_version}Code: Select all
update opcodes set version_range2 = 974 where version_range2 = 973Note:THIS WILL NOT WORK ON MAKING YOUR WORLD COMPATIBLE. Access to your worlds is governed by our Public Login Server, and if Login does not know about the new data version, you are denied access. So please, no questions on how to do this to make worlds compatible with new(er) clients.
How do I tell my Parser to use the newest Data Version?
Parser, much like World, uses Opcodes and the 3 Structs (xml) from the Config Files\World Structs folder on SVN. It's a simple matter of copying the 3 XML files you use for your World to the place where PacketParser.exe is stored.
The harder part (though still not hard to anyone who understands MySQL) is updating Parsers `opcodes` table. For this, you need to somehow export the current `opcodes` table from your EQ2Emulator WORLD database, then import that table to your parser DB.
How I do it is I use a GUI tool, like SQLYog. I open a connection to my database, highlight the eq2dev.opcodes table in the tree view, right-click on it, then Copy Table To Different Host/Database, pick the Parser database, and you're done. GUI's make things very simple... however, if you do not have a GUI, you can still achieve this using MySQL command lines.
Export:
Code: Select all
mysqldump.exe -h {your_mysql_ip} -u {mysql_user} -p {your_eq2emulator_db_name} opcodes > opcodes.sqlComplete example dump command line:
Code: Select all
mysqldump.exe -h locahost -u root -p eq2dev opcodes > opcodes.sqlImport:
Code: Select all
mysql.exe -h {your_mysql_ip} -u {mysql_user} -p {your_parser_db_name} < opcodes.sqlComplete example import command line:
Code: Select all
mysql.exe -h locahost -u root -p eq2_rawdata < opcodes.sqlIf there are any other questions related to PacketParser.exe and the DATA it uses to work, let us know and we'll add to this FAQ. Please don't ask me how to unravel the entire packet, as almost 2 years later I am still unsure how to do that myself.
