Packets: Reading and Understanding

[John Adams]

I can actually do it in PHP too, I was just hoping there was something already done.

LE/Image, in Hex, what does a Float store like? I am trying to identify element #2... a float for pos_x_0... :/ I'll google it, in case you think that's a stupid question I should just google

[LethalEncounter]

Ah hah!! See? We're getting somewhere. So what started this whole parade of silly questions was the fact that pos_size = 0 for everything parsed using 944 logs, and presumeably 936 structs. What I am hoping to see clear-as-day here is that the value for pos_size is just not in the right position anymore thus being set to 0, instead of "9" like a typical village rat or shrumbler size.

So I will go run this on some data that I know should have a size value. I think even NPCs should have a size... it should never be 0, but 32 appears to be "normal" for everything I've played with. less than 32 for smaller, greater than for larger, of course.

Like our small crabs on the Queen's Colony with 0 set as size is the same size as if it were 32. So in order to make them actuall "a small shore crab" vs "a shore crab" vs "a large shore crab", I had to make them size 24, 28 and 32 respectively. But lately... everything is parsing as size 0 (32) so I dunno wtf is going on. /babble off

Pos_size is no long used as it once was. That field will probably always be 0. They switched to using pos_size_ratio which is a floating point number. The code automatically converts the old value to the new struct. (Kinda have to support both since clients for both versions can be logged in).

[LethalEncounter]

I can actually do it in PHP too, I was just hoping there was something already done.

LE/Image, in Hex, what does a Float store like? I am trying to identify element #2... a float for pos_x_0... :/ I'll google it, in case you think that's a stupid question I should just google

int32/double/float = 4 bytes
int16 = 2 bytes
int8/char/uchar = 1 byte

So 25 7A D8 C3 is the float value for x.

[John Adams]

Pos_size is no long used as it once was.

~blinks in amazement~

You couldn't have told me this a week ago when I mentioned that this is the entire problem I've been having?

Regardless of that very frustration revelation ... mobs that should be small, are huge, in new parsings. So something isn't right, somewhere.


and thx for the float lesson =) I still need to learn this, and I think I am well on my way now ~hopes~

[John Adams]

Err, sorry if this is remedial math... but how does the value 1088914795 become 7.23504?

This is what I am looking at, first line of the packet:

Code: Select all

   0: 7C 1C E0 30 6B 85 E7 40 - 4B F7 F7 C0 02 87 79 43  | |..0k..@K.....yC

The 7C 1C E0 30 turned into the grid location perfectly. So I thought the next value was the position_pos_x_0:

Code: Select all

Name: position_pos_grid_id_0  	Index:  0 	Type:  int32		Data:  819993724 
Name: position_pos_x_0  	Index:  0 	Type:  float		Data:  7.23504
Name: position_pos_y_0  	Index:  0 	Type:  float		Data:  -7.74894
Name: position_pos_z_0  	Index:  0 	Type:  float		Data:  249.527

Btw, has something with position_pos_heading1_0 and 2_0 changed, too, while we're on the subject of things changing? These used to have the values the NPC was facing, but now seem to have odd int values like -11091 or 8019. Used to be 0-359, or so I thought.

[LethalEncounter]

Err, sorry if this is remedial math... but how does the value 1088914795 become 7.23504?

This is what I am looking at, first line of the packet:

Code: Select all

   0: 7C 1C E0 30 6B 85 E7 40 - 4B F7 F7 C0 02 87 79 43  | |..0k..@K.....yC

The 7C 1C E0 30 turned into the grid location perfectly. So I thought the next value was the position_pos_x_0:

Code: Select all

Name: position_pos_grid_id_0  	Index:  0 	Type:  int32		Data:  819993724 
Name: position_pos_x_0  	Index:  0 	Type:  float		Data:  7.23504
Name: position_pos_y_0  	Index:  0 	Type:  float		Data:  -7.74894
Name: position_pos_z_0  	Index:  0 	Type:  float		Data:  249.527

Btw, has something with position_pos_heading1_0 and 2_0 changed, too, while we're on the subject of things changing? These used to have the values the NPC was facing, but now seem to have odd int values like -11091 or 8019. Used to be 0-359, or so I thought.

http://www.cs.princeton.edu/introcs/91float/
(Do yourself a favor and just download Hex Workshop, paste the value in it, and the value will be displayed)

Nope, pos_heading1 and 2 have always been sent like that. The way that is turned into a floating point heading is:

Code: Select all

float heading = val/64;
if(heading >= 180)
	heading -= 180;
else
	heading += 180;

-11091 turns into 6.703125 and 8019 turns into 305.296875.

[John Adams]

Re: Size -- ok, I think I get it now.

Old Way:

Code: Select all

Name: position_pos_size_0  	Index:  0 	Type:  int16		Data:  12 
Name: position_pos_size_multiplier_0  	Index:  0 	Type:  int16		Data:  32 

New Way:

Code: Select all

Name: position_pos_size_0  	Index:  0 	Type:  int16		Data:  00 
Name: position_pos_size_ratio_0  	Index:  0 	Type:  float		Data:  0.4
Name: position_pos_size_multiplier_ratio_0  	Index:  0 	Type:  float		Data:  1

0.4 is to 1 as 12 is to 32.

[LethalEncounter]

Pos_size is no long used as it once was.

You couldn't have told me this a week ago when I mentioned that this is the entire problem I've been having?

And have you miss all this learning?

Nah, I must have missed your original question somehow. You have to be more direct and make a post simple and specific otherwise I miss stuff

[John Adams]

Code: Select all

float heading = val/64;
if(heading >= 180)
   heading -= 180;
else
   heading += 180;

Oh! Hey, I've seen this code. =)

Ok, I am done asking stupid questions for the night. Gotta go cook some "Carbonara to die for".


And PS: I will start asking 1 question per post, again. In bold. Red.

[John Adams]

Nope, pos_heading1 and 2 have always been sent like that. The way that is turned into a floating point heading is:

Code: Select all

float heading = val/64;
if(heading >= 180)
	heading -= 180;
else
	heading += 180;

-11091 turns into 6.703125 and 8019 turns into 305.296875.

Ok, I see how both of these turn into their floats, and that's pretty nifty. But I have another question about sint16.

In the unpacked data, I have a pos_heading1 of AD D4, little endian D4 AD. In dec, 54445. "54445 / 64 = 850.703125", which is ">= 180", so I applied the "-=180" and end up with 670.703125, which is still not a valid heading. However, after some fancy math (subtraction, ahem) I took 65535 from 54445 and ended up with the expected -11090, and then all the above math worked out as expected.

Question!
What's the deal with subracting the max INT value from the DEC value first? I haven't found that in the code, but maybe that's how the signed data types work?


Btw, I have downloaded and installed Hex Workshop. I now have another cool tool on my system I have no idea how to use.

[John Adams]

int32/double/float = 4 bytes
int16 = 2 bytes
int8/char/uchar = 1 byte

Do these same rules apply for sint8, sint16, etc? Single byte can handle a -255?

[Bion]

I think they are half their value max value -1. a single sint8 is max +/-127 I am not sure there is a way to tell just by looking at the bytes.
I did noticed that if something was signed there was a FF near it but that could have just been coincidence. good question

I think you just have to guess at what it should be by it's placement in the packet. but if you have alot of data to check it against and as long as there is something in the fields it should pop out after awhile. one thing that I have noticed is that if the bit size is correct you will be able to see data in the following fields if it is wrong or to much of a wrong type like it is suppose to be string and you have a struct trying to show it as a int. all the data after that point will be all trashy looking. so that could be a indication where something has changed.

[John Adams]

Ok, whew... after forcing myself to endure figuring this stuff out, I think I finally have a clue - but as with all other things I learn, it takes brute-force to get it through my thick skull =) I'll bore you with it only because I made so much noise about it, I want to explain how I am validating that things actually are PERFECTLY OK and not all gloomy as I had hoped -- er, I mean thought... anyway. In order for my brain to accept the SpawnStructs, I had to lay them out visually. I had to do the same thing before trying to understand the EQEmu's playerpacket structs, so luckily I had some old code to use!

What my problem was, was "all of a sudden" it appeared that some data values were not parsing correctly. So in order to ensure both the structs and parser were correct, I had to find various data in the unpacked packet, so I could visually SEE it was not bad. And after much effort, I have found that the Collector is collecting chest_type_id and legs_type_id as FF FF, but only on some mobs - so that tells me there is NOTHING WRONG... and I need to ignore this odd value.

LE showed me that pos_size has changed on SOE's side, yet I do not know if we have support for it in the emulator, since spawns that should be small, are not. I do not see new fields in the DB to support ratio and multiplier. Is this something we need? Or should our -populate be converting the new SOE size format into 'size'? If this is support to be coming soon, can we get it sooner than later? It is playing hell on my data cleanup efforts (2 weeks spent trying to determine if data is bad or not).

Just so no one else does this for me, I wrote a PHP script to 'parse' the unpacked packet (currently version 864 structs only, as I was testing with a 910 log).

Code: Select all

   0: AD E9 6E 65 9A 99 80 C3 - D7 A3 30 40 5C 0F 3E 43  | ..ne......0@\.>C
  16: 00 00 00 00 00 00 00 39 - 00 39 00 00 80 40 00 00  | .......9.9...@..
  32: 04 00 00 00 02 00 00 00 - 00 00 00 00 00 00 00 00  | ................
  48: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
  64: 00 00 0B 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
  80: 00 00 00 00 20 00 20 00 - 20 00 00 00 FF FF FF FF  | .... . . .......
  96: 00 00 00 00 00 00 03 01 - 00 00 16 00 00 00 00 00  | ................
 112: 00 00 00 00 64 00 00 00 - 00 00 00 00 00 00 00 00  | ....d...........
 128: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 144: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 160: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 176: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 192: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 208: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 224: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 240: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 256: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 272: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 288: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 304: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 320: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 336: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 352: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 368: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 384: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 400: 00 00 00 00 00 00 03 01 - 01 03 01 00 02 04 00 00  | ................
 416: 00 00 00 00 00 00 00 00 - 00 00 00 00 93 1C 93 1C  | ................
 432: CC CC CC CC CC CC E5 E5 - 66 CC CC CC CC CC CC E5  | ........f.......
 448: E5 66 00 00 00 00 00 00 - E0 18 00 00 00 00 00 00  | .f..............
 464: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 480: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 496: 00 00 00 00 6D 04 00 00 - 00 00 FF FF FF FF 6D 04  | ....m.........m.
 512: 00 00 FF FF FF FF FF FF - FF FF FF 1A 3F 1A FF FF  | ............?...
 528: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF  | ................
 544: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF  | ................
 560: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF  | ................
 576: FF FF FF FF FF FF FF FF - FF FF 00 00 00 00 00 00  | ................
 592: 00 00 00 00 00 00 FF FF - FF FF FF FF FF FF FF FF  | ................
 608: FF FF FF FF FF 2D 2D 2D - FF FF FF FF FF FF FF FF  | .....---........
 624: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF  | ................
 640: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF  | ................
 656: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF  | ................
 672: FF FF FF FF 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 688: FF FF FF FF FF FF 00 00 - 00 00 00 00 00 00 00 00  | ................
 704: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 720: 00 00 00 00 00 00 00 00 - 00 00 0B 00 00 00 00 00  | ................
 736: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 752: 00 00 00 00 0B 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 768: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 784: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 800: 00 0C 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 816: 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  | ................
 832: 32 33 00 00 00 00 00 00 - FF 01 00 00 00 00 CD CD  | 23..............
 848: CD CD CD CD CD CD CD CD - CD CD CD CD CD CD CD CD  | ................
 864: CD CD CD CD CD CD CD CD - CD CD CD CD CD CD CD CD  | ................
 880: CD CD CD CD CD CD CD CD - CD CD CD CD CD CD CD CD  | ................
 896: CD CD CD CD                                        | ....

So if you paste this unpacked stuff into the box, you can see the data (except for the huge arrays that I am not interested in atm).

Long story short... aside from the Size issue, I don't think anything is "broken". RoK+ data is just not going to be exactly like Live, since the chest/legs data is either correct, or FF FF (65535) which may actually mean 0, I dunno. But I'm done worrying about it, and hopefully will have the 3 remaining newbie areas spawned this weekend.

[LethalEncounter]

LE showed me that pos_size has changed on SOE's side, yet I do not know if we have support for it in the emulator, since spawns that should be small, are not. I do not see new fields in the DB to support ratio and multiplier. Is this something we need? Or should our -populate be converting the new SOE size format into 'size'? If this is support to be coming soon, can we get it sooner than later? It is playing hell on my data cleanup efforts (2 weeks spent trying to determine if data is bad or not).

You are right. For some reason it slipped my mind that I need to change the parser to convert from the new format to the old format. I just made the changes to the parser.

[LethalEncounter]

RoK+ data is just not going to be exactly like Live, since the chest/legs data is either correct, or FF FF (65535) which may actually mean 0, I dunno.

Why wouldn't the data be like Live? The values being parsed are the same ones being sent to your client of Live. Just because you don't agree with the values being sent doesn't make them incorrect