EQ2 Models (.draw)

[Jabantiz]

I always wanted to learn direct x and try to teach myself when I have the free time. Today I had the free time and after learning how to load and render models I ended up on one of my wild tangents, the result was rendering EQ2 models.

EQ2ModelTest3.png

EQ2ModelTest4.png

EQ2Terrain.png

I extracted the .draw files and textures with Blaz's VPK Tool and opened the .draw file in hex workshop and did my best to figure out the file format. Here is what I came up with, put it in eq2 emu xml struct format because that is what we all know.

Code: Select all

  <Struct>
    <Data ElementName="text" Type="EQ2_8Bit_String" />
    <Data ElementName="unknown1" Type="int8" />
    <Data ElementName="file_type" Type="int8" />
    <Data ElementName="unknown2" Type="int8" Size="5" />  <!-- Notice the Size 5, so the unknown is 5 bytes -->
    <Data ElementName="vertex_count" Type="int32" />
    <Data ElementName="verticies" Type="Array" ArraySizeVariable="vertex_count">
      <Data ElementName="vertex_x" Type="float" />
      <Data ElementName="vertex_y" Type="float" />
      <Data ElementName="vertex_z" Type="float" />
    </Data>

    <Data ElementName="normal_count" Type="int32" />
    <Data ElementName="normals" Type="Array" ArraySizeVariable="normal_count">
      <Data ElementName="normal_x" Type="float" />
      <Data ElementName="normal_y" Type="float" />
      <Data ElementName="normal_z" Type="float" />
    </Data>

    <Data ElementName="tex_coord_count" Type="int32" />
    <Data ElementName="tex_coords" Type="Array" ArraySizeVariable="tex_coord_count">
      <Data ElementName="tex_coord_u" Type="float" />
      <Data ElementName="tex_coord_v" Type="float" />
    </Data>

    <Data ElementName="tex_coord2_count" Type="int32" />
    <Data ElementName="tex_coords2" Type="Array" ArraySizeVariable="tex_coord2_count">
      <Data ElementName="tex_coord2_u" Type="float" />
      <Data ElementName="tex_coord2_v" Type="float" />
    </Data>
    <Data ElementName="tex_coord3_count" Type="int32" />
    <Data ElementName="tex_coords3" Type="Array" ArraySizeVariable="tex_coord3_count">
      <Data ElementName="tex_coord3_u" Type="float" />
      <Data ElementName="tex_coord3_v" Type="float" />
    </Data>
    <Data ElementName="tex_coord4_count" Type="int32" />
    <Data ElementName="tex_coords4" Type="Array" ArraySizeVariable="tex_coord4_count">
      <Data ElementName="tex_coord4_u" Type="float" />
      <Data ElementName="tex_coord4_v" Type="float" />
    </Data>
    <Data ElementName="tex_coord5_count" Type="int32" />
    <Data ElementName="tex_coords5" Type="Array" ArraySizeVariable="tex_coord5_count">
      <Data ElementName="tex_coord5_u" Type="float" />
      <Data ElementName="tex_coord5_v" Type="float" />
    </Data>

    <!-- These bytes are only in the file if file_type = 13 (0x0D) -->
    <Data ElementName="tex_coord6_count" Type="int32" IfVariableEquals="file_type=13" />
    <Data ElementName="tex_coords6" Type="Array" ArraySizeVariable="tex_coord6_count">
      <Data ElementName="tex_coord6_u" Type="float" />
      <Data ElementName="tex_coord6_v" Type="float" />
    </Data>
    <Data ElementName="tex_coord7_count" Type="int32" IfVariableEquals="file_type=13" />
    <Data ElementName="tex_coords7" Type="Array" ArraySizeVariable="tex_coord7_count">
      <Data ElementName="tex_coord7_u" Type="float" />
      <Data ElementName="tex_coord7_v" Type="float" />
    </Data>
    <Data ElementName="tex_coord8_count" Type="int32" IfVariableEquals="file_type=13" />
    <Data ElementName="tex_coords8" Type="Array" ArraySizeVariable="tex_coord8_count">
      <Data ElementName="tex_coord8_u" Type="float" />
      <Data ElementName="tex_coord8_v" Type="float" />
    </Data>
    <!-- End of the extra bytes for file_type = 13 (0x0D) -->


    <Data ElementName="face_count" Type="int32" />
    <Data ElementName="face_list" Type="Array" ArraySizeVariable="face_count">
      <Data ElementName="face" Type="int16" />
    </Data>
  </Struct>

This struct is not perfect, I still have issues opening up a few files, mainly those related to zones it seems. I also stopped after the face list as that is all I needed to render the model, there is more after including what looks like an array of strings for bone names and an array of string for shaders I think. Also I am not sure what tex_coord2-5 is for, not even sure they are texture related but the array is accurate, just assumed it was texture related (my best guess is tex_coords for the normal/bump/spec maps used by shaders).

Also for those of you interested I use these tutorials to teach myself direct x (I only made it to tutorial 8...)

[thefoof]

That's really cool.

[Lokked]

You definitely have a gift for this, Jab. Well done!

[Jabantiz]

Updated the struct in the first post, the second byte after the string was 0x0F in all the files I was opening with out a problem, the ones I had issues with that byte was 0x0D. When the byte was 0x0D there was 12 extra bytes between the last texture coord array and the face list. So far I have only seen that byte set to 0x0D and 0x0F, and am now able to open both with no problems.

Also in the struct I put it as 3 more texture arrays, I have no clue if that is accurate as the only files I have seen with 0x0D so far have had all the extra bytes set to 0x00, I am just guessing that it is more texture arrays. With this updated struct I have had no issue opening any of the .draw files so far.

[thefoof]

Now I need visual images of spawns in the zone migrater, mwuahahaha

[thefoof]

I was looking into EQ2Models a bit last night for a simple tool I've had in the back of my head for awhile and found this post by blaz

If you are interested in the 3D format, I suggest you take a look at the VeRenderMesh code first. That's the class that represents a raw mesh as stored in .draw files. The format is quite straightforward (if you ignore the likes of 'autoDropVertexIndices' and 'noPolygonShadowTriangleIndices', which I have no clue as to what they are), and you can probably have a simple DirectX or OpenGL app that renders these meshes in little time.

If you want to render the mesh with textures, it is more complex. You get the texture coordinates from the VeRenderMesh object, but you don't get the information on which textures to assign to each stage.

The process would be as follows. You would first parse the script with the definition for the object, for example:

clientvdl/appearances/accessories/wieldable_items/weapons/axe/battleaxe/battleaxe001_plain.vdl

And get the path to the VOC file with the "visual definition" of the object:

composed/characters/accessories/wieldable_items/weapons/axe/battleaxe/battleaxe001_plain.voc

You would then decode that file and end up with an object of the VeMeshGeometryNode class (or some class derived from it). This class contains information on each of the LODs of the object. For each LOD, you get the path to the render meshes that represent the object and the path to the "shader palette":

cooked/characters/accessories/wieldable_items/weapons/axe/battleaxe/battleaxe002.sp

This .sp file contains an object of the VeShaderPalette class, which in turn contains a list of scripts that define the shader palette:

shaders/characters/accessories/wieldable_items/weapons/axe/battleaxe/battleaxe001plain.vdl

Finally, this script defines the textures to apply to the object and has a reference to the shader code that is used to render the object:

shaders/templates/bclr_t2_bmp1_spwm.vdl

The problem with this approach is that there is no easy way to integrate all this script and shader code stuff in an existing 3D engine. You'd probably be better off exporting all the data you need to your own format and working with that.

If you want to work with animations, the classes to refer to are VeAnimMeshGeometryNode, VeSkeleton (.skeleton files) and VeAnimation (.anim files).

Also found a python script using the old blender API that looks to have the entire struct of .draw files, in an existing project somebody worked on a long time ago

http://code.google.com/p/openzone2/source/browse/


Just putting this here for reference


Also found the same tutorials Jab used ported over to SharpDX which I'm considering using for this tool, if it ever happens.

http://sharpdx.org/forum/7-documentatio ... mitstart=0

[Chrisworld]

This is some very impressive work.

Well done!

[Tiger]

This is excellent, expect that the current client, does not use clientvdl/appearances/
Instead the client uses the zlib compressed app_cache.dat (I'm not talking about appearance.dat)
When mine is decompressed it is 249MB in size. This file is the "brains" for all the appearance. In fact there are 262490 imbedded voc files in just one of its 23 different sections. Things such as mapping what animations and what sounds to what meshes, ect are all in here.
The file is sequential meaning you must start at the beginning to get to the end. The first two sections are for ASCII strings. They map an ID number to each string. The ID numbers are then referenced in the 21 remaining sections.
For example in my app_cache.dat from a few months ago, there were 62055 strings in the first section. Here are the first 20:

Code: Select all

10020  staticobjects/liveevent/frostfell/dpo_candy_round_red 
20728  accessories/wieldable_items/weapons/great_sword/broad_sword/broad_sword001_plain 
112097  dodge03 
132557  staticobjects/urns/dpo_slove02_vase03 
136988  accessories/wearable_items/_exp10/heroic_armors/leather_shadowred/shoulders 
156320  cooked/characters/ec/core/soga/dwarf_male/spell_bard_flute_loop.anim 
183609  cooked/characters/ec/core/mounts/pegasus_mount/barbarian_male/fly_turn_right.anim 
344896  cooked/characters/creatures/tu/goatmen/animations/goatman_swim_idle.anim 
452939  cooked/characters/ec/core/faceanims/soga_ogre_female/angry.anim 
735908  cast_bard_stringed_persist 
779162  sounds/critters/shadowed_man/shadowedman_idle01.wav 
804579  sounds/_exp08/creatures/ulthork/ulthork_vox_idle.wav 
904888  _exp06/zones/objects/plunderable_loot/vase_small_full_01 
944709  creatures/mounts/anglerserpent_mount/anglerserpent_mount_ice_transparent 
1111773  sounds/player/barbarian/barbarianm/barbarianmale_drown001.wav 
1363216  cast_mage 
1402318  accessories/wearable_items/_exp06/erudite_classic_armor/erudite_classic_armor_leather_white/chest 
1466929  sounds/combat/sheathe/slashing/1hs_sheathe.wav 
1472889  _exp10/objects/holotable_dpo 
1504070  aa_hand_leaf 

The next section is for the voc names that are imbedded in the third section of app_cache.dat There are 262494 in mine (meaning that if you don't look in app_cache.dat your missing out on nearly one third of the files in the EQ2 client (262494 imbedded voc+500000 regular files=roughly 750000)
Vocs get 2 4byte integers for IDs. Here are the first 20:

Code: Select all

1991970851  32180  composed/characters/accessories/wearable_items_armor_maker/_exp08/unknown_faction_armor/chain/soga_barbarian_male/head.voc 
-433285186  46747  _exp07/composed/fx/creatures/tserrina/tserrina_wing_buffet.voc 
-1269368923  66073  composed/characters/accessories/wearable_items_armor_maker/_exp06/dragon_chain/human_female/legs.voc 
-916720843  72030  composed/characters/accessories/wearable_items_armor_maker/_exp08/robe_elemental/air/soga_dwarf_female/skirt.voc 
11056783  103793  composed/characters/accessories/wearable_items_armor_maker/_exp07/dark_elf_armor_chain/halfling_female/chest.voc 
858277148  110773  _exp06/cooked/zones/objects/odus_south/eurudite_furniture/odso_chair01_ball01_l0.voc 
1847785265  114831  composed/characters/accessories/wearable_items_armor_maker/_exp08/short_skirts/green_01/kerra_male/head.voc 
569042216  122174  composed/characters/accessories/wearable_items_armor_maker/_exp05/void_heavycloth/ratonga_female/chest.voc 
1294104602  129805  composed/characters/accessories/wieldable_items/weapons/exp09/bow/longbow_fantasy_greengold.voc 
237074049  146888  composed/characters/accessories/wearable_items_armor_maker/_exp09/starter_armor/templar/soga_dwarf_male/legs.voc 
-1814526503  148410  composed/characters/accessories/wearable_items_armor_maker/_exp07/human_heritage_armor_chain/fae__common__male/head.voc 
-210946145  151773  composed/characters/accessories/wearable_items_armor_maker/_exp09/withering_lands/chain/halfling_female/legs.voc 
33040565  179746  composed/characters/accessories/wearable_items_armor_maker/_exp07/cryomantic_armor/soga_human_female/forearms.voc 
240932540  208009  composed/characters/accessories/wearable_items_armor_maker/_exp06/ornate_robe01/soga_dwarf_male/chest.voc 
-2025108018  257794  composed/characters/accessories/wearable_items_armor_maker/_exp08/ralloszek_armor/leather/soga_common_elf_female/shoulders.voc 
309601215  268728  composed/characters/accessories/hair/ratonga_female/glasses_stud_whiskers.voc 
266477056  271969  composed/characters/accessories/wearable_items_armor_maker/magus/robes/plain_quilted/halfling_male/pants.voc 
-1029720981  283029  composed/characters/accessories/wearable_items_armor_maker/leather/common/high/gnome_female/hands.voc 
1058801704  289408  composed/characters/accessories/wearable_items_armor_maker/_exp09/wyvern_armors/chain/dwarf_female/shoulders.voc 
1528804171  289635  composed/characters/accessories/wearable_items_armor_maker/_exp06/frostrune_armor_robe/soga_barbarian_female/skirt.voc 

The next section are the vocs themselves (though the EQ2 devs got slopply in that there some repeated names. In fact you could make all the ascii strings say johnadamsrule and the game will still load, except that a select few imbedded voc files reference other imbedded voc by name with VEFILENODE, I think the repeated name with the lowest ID number will be used then.) There are 262490 voc files (so again it is not strictly one to one in any direction (id to string) or (string to id)). I have an extractor somewhere on my computer, that I'll upload.
The next 19 sections are binary (no ascii) and progressively build up to the 23rd section which is the "cerebral cortex" of the file in the sense that while there might be a few standalone sections (not sure, but some sections are tiny, with only 8 members), the last section maps to the second last section which might map to the 12th section, which maps the 8 section to the 7th and 6th sections ect. I made a few scripts that I will post that will explain things better. The cache.dat file has been used at least since 2007 (though the app_cache.dat on very old clients might just be nonsense ASCII, that is not loaded. Hint the game will crash on the loading screen, without the app_cache.dat on the new clients, while the game works just fine with out the dummy ones on the ancient clients). Blaz did most his work on the old style client before the new client was released.
I'll also upload a more complete description of each section (its on my computer somewhere).
Think of app_cache.dat as being like WoW dbc files.

[Jabantiz]

I made a few scripts that I will post that will explain things better.

That would be great, always wanted to make a tool to display the model (looks like foof does too) just never looked into it to much, my opening post is as far as I got and that was manually extracting the files with blaz's vpk tool. Any additional info would be great to have for future reference.

[thefoof]

I made a few scripts that I will post that will explain things better.

That would be great, always wanted to make a tool to display the model (looks like foof does too) just never looked into it to much, my opening post is as far as I got and that was manually extracting the files with blaz's vpk tool. Any additional info would be great to have for future reference.

Yeah, I stopped looking at it because I'll probably eventually work on that in-game spawn editor I talked about before. That will kind of do what I wanted.

[Dello0000]

Jabantiz Where does your talent end?

Also whats wrong with the elf's breasts? birthmark's? lol

[Jabantiz]

This was actually mostly a learning experience for me, was fun but haven't put much work into it since then.

I remember in beta there was a bug that would cause armor to not show and you would see an occasional model like the above elf running around, no clue why they decided to do the texture like that but I gave up trying to understand SOE a long time ago...

[alfa]

Nice work Jab.
If I propely remember Image aleready wrote a program to open VKP file and see layer of zone.
So a question pop up in my mind instantly: Geodata for EQ2Emu ??

[Jabantiz]

I know Blaz had a tool to open vpk's, that is what I used to extract the files, didn't know image had one.

But yes in the long run it would be possible to get the geodata extracted for use by the server.

[Jabantiz]

My latest attempt at this was to convert my code from c++ to c# with SlimDX so I could use a dll from blaz's vpk tool, still a lot of work to do as the file formats are confusing but here is my result from a couple days working on it.

MapTest.jpg