Implementing: Instancing

[John Adams]

Scat, since we've talked about this before, and I know you started planning some things out, let's talk about any details here for implementing Instanced zones.

The reason this is coming up (from me) right now is that I am re-doing the Zones tables while populating some zones for Content, and realized we definitely could trim a ton of fat off our Zones tables if Instancing were available.

Now, I am not talking about full on, raid zones or housing. I simply mean, /zone WorkshopEvil and it will use a single (parent) tradeskill_workshop zone entry and append the specific (child) details to make a virtual zone.

What I have done so far, is tore the `zones` table apart. In Zones, all there is are these fields:

Code: Select all

CREATE TABLE `zones` (
  `id` int(11) unsigned NOT NULL auto_increment,
  `name` varchar(64) collate latin1_general_ci NOT NULL default '',
  `file` varchar(64) collate latin1_general_ci NOT NULL default '',
  `description` varchar(255) collate latin1_general_ci NOT NULL default '',
  `zone_type` tinyint(3) unsigned NOT NULL default '0',
  `login_checksum` int(10) unsigned NOT NULL default '0',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `ZoneNameIDX` (`name`),
  KEY `ZoneDescIDX` (`description`),
  KEY `ZoneFileIDX` (`file`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci

Those are the parent zone records. I've also created zone_details:

Code: Select all

CREATE TABLE `zone_details` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `zone_id` int(11) unsigned NOT NULL default '0',
  `safe_x` float NOT NULL default '0',
  `safe_y` float NOT NULL default '0',
  `safe_z` float NOT NULL default '0',
  `safe_heading` float NOT NULL default '0',
  `underworld` float NOT NULL default '-1e+006',
  `xp_modifier` float NOT NULL default '0',
  `min_recommended` tinyint(3) unsigned NOT NULL default '0',
  `max_recommended` tinyint(3) unsigned NOT NULL default '0',
  `always_loaded` tinyint(3) unsigned NOT NULL default '0',
  `city_zone` tinyint(3) unsigned NOT NULL default '0',
  `min_status` int(10) NOT NULL default '0',
  `min_level` int(10) NOT NULL default '0',
  `max_level` int(10) NOT NULL default '0',
  `start_zone` tinyint(3) NOT NULL default '0',
  `instance_type` tinyint(3) NOT NULL default '0',
  `default_reenter_time` int(10) unsigned NOT NULL default '0',
  `default_reset_time` int(10) unsigned NOT NULL default '0',
  `default_lockout_time` int(10) unsigned NOT NULL default '0',
  `force_group_to_zone` smallint(5) unsigned NOT NULL default '0',
  `lua_script` varchar(255) collate latin1_general_ci default '',
  `shutdown_timer` int(10) unsigned NOT NULL default '300',
  `zone_motd` varchar(250) collate latin1_general_ci default '',
  `ruleset_id` int(10) unsigned NOT NULL default '0',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `ZoneDetailsIDX` (`zone_id`),
  CONSTRAINT `FK_zone_details` FOREIGN KEY (`zone_id`) REFERENCES `zones` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci

which has all the more unique values per zone in it (just like characters + character_details, etc).

What I want to create now is a zone_instances table (just an example):

Code: Select all

CREATE TABLE `zone_instances` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `parent_zone_id` int(10) unsigned NOT NULL default '0',
  `instance_name` varchar(64) collate latin1_general_ci NOT NULL default '',
  `instance_description` varchar(64) collate latin1_general_ci NOT NULL default '',
  `instance_shutdown` int(10) unsigned NOT NULL default '300',
  PRIMARY KEY  (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci

which points to the `zones`.`id` record as a parent, so it knows what File to launch. Then we can set the instance name (for the /zone command) and the description (for screen display while zoning in). I figure the other instance-related fields from zone_details can come to this table, or another one which configures the actual running instance later (see Image's `instances` table idea).


This is where I am falling down. I am not sure how to offer an "instance /zone Name" that overrides the parent /zone Name. Unless..... because only GMs would ever use /zone Instance, maybe we can have a new command /instance DartharsLair instead of /zone? And our scripts that zone players around can be told to handle them any way we choose?


Again, I do not want to get hung up on details of raids, lockout timers or any of that... just get me to be able to /zone into the same file for Queen's Colony, only this time, it's Darthar's Lair (or whatever).


Does this make sense?

I kinda would like to do this before going any further with spawn pops, because spawn IDs are related to their zone_id, and if we're going to change things here, I'd rather not repop again later.

Get yer thinkin cap on!

[John Adams]

And let me make an addendum right away, giving you a way out of all this

Since Zone data is not all that big, we certainly could leave everything in `zones`, and dupe the crap out of our `file` entries for every instance we need, kinda like we're doing now. Those are not really virtual zones though, and wanted to get your thoughts on being fancy pants vs plain janes

[Scatman]

Yeah that makes a lot of sense. But what about instances where you have multiple copies of the same zone? Like those instance zones in Queens Colony and Outpost? How would /instance work then?

[John Adams]

Ok, I deleted my ramblings to make new ones, now that I understand your question. I think you are referring to the small instance zone (quest) within QC and OO itself, and how 10 different players could spawn their own instance of that solo zone.

We will likely need to track active instances somehow, and maybe a GM has an /instance list command or something, which will display the instances virtual zone_id/name. Each instance that is spawned should have an "owner", if it is a personal, group, guild or raid instance. The char_id owns a solo instance or housing, a group_id owns a group instance of up to 6 players, a guild_id owns a guild hall, and a raid_id owns an instance up to 24 players. Sound right? This is so that same group can return to their instance should they leave for whatever reason, and their progress remains intact.

As for instanced, public zones like Antonica and Commonlands, there is no owner of course. We'll only spawn a new instance if we exceed `max_players` in a zone Rule. The first instance is of course just called "Antonica", so /zone Antonica would work. Additional instances will be called "Antonica 2", "Antonica 3", and so on... so we can probably add a parameter to the /zone command to be able to /zone Antonica 2 just to get to the 2nd running instance of that zone.


Regarding my `zones` table revamp, and my obsession with redundant data... I may have to opt out of that, since I cannot conceive how to tie spawns to a zone ID of an instance (yet). As you know, when we build content the spawn is assigned a spawn_id in the range of the zone (zone_id * 10000). If instances have "virtual zone IDs" I have yet to figure out how to assign spawns to it... and in frustration to get content moving, I may have to settle for just having `file` inserted repeatedly in the zones table for now

zone_file.jpg

This makes my eyes bleed, but for now, we just may not have a choice. Problem, and what brought this up right now, is if we change things later, all the spawn ID's are going to get messed up.

[Sylva1n]

I hope what I'm about to try explaining will make sence...

What if the current zones, and zone spawns were a "model" or "spawn list", and every zones were treated as an instance?

solo instance = 1 player
heroic instance = 6 players (one group)
epic x2 instance = 12 players (2 groups)
epic x4 instance = 24 players (4 groups)
shared dungeons = 100 players
outdoor zones = 100 players

1 db table (active zones) containing active zones with an id for each zone, owners if any, and expiration timer if needed, a current player count and max player amount.
when the max player amount is met, server creates a second instance.

say I zone to QC, and nobody was there before me, server creates something like:

id: 1 zone_id: [QC zone_id] owner: NULL exp_timer: NULL cur_player_count: 1 max_player_count: 100

when everyone leaves that instance the line is deleted, if we get 101 players zoning to QC, a second line is created with id: 2

ok should probably have a second table for zone owners, but could it be a solution?

edit: still would need a temporary db_table to hold infos on what spawn is up and wich is dead for each zones...

[John Adams]

I see where you're going with that, and it's a pretty good idea. It still leaves us with the problem of, how to associate spawns (logically) with a given zone? Ie., Murrar Shar will always (and only) be in Queen's Colony. Currently, QC is zone_id 253, and Murrar's spawn_id is thus 2530000 - the first spawn in that zone (for example).

If we have no idea what the zone_id is, we have no idea what the spawn_id is - and while spawn_id's can be sequential from 1 - 1,000,000 if we chose, this is not how our parser or editors are designed to think - and massive rewrites of our DB editor would be required (nearly everything is laid out in 'ranges' so data is easy to recognize).

That's why the further I got into this zone-revamp thing, the less enthused I was about it. However, I'm still interested in hearing ideas about instancing, and how we might handle those spawns.

Let me ponder that a while. Interesting concept, Syl.

[Sylva1n]

what I meant was, you leave you spawns as they are right now, we just change the way the zones work... when a zone is created it just load the entire spawn range from the id range and populate an living_or_dead table for that current "instance" of the given zone...

qc id is 253 you say

instances_table:
say id: 1 zone_id: 253 (create a current spawns table with every spawns associated with zone 253 and instance_id 1) owner: NULL.... etc.

and since we already have a spawns table just use their IDs to identify them.

so we would just edit the populating of the model or "blueprint" of each zones (just 1 model for any given amount of instances there might be for that zone)

[John Adams]

/bump for Jabantiz, who made an amazing discovery recently.

I screwed around with the instance code and discovered it is mostly functional, a few tweaks and every thing was working rather well. I always thought the code was incomplete and not functional but it turns out it is damn near complete just a few issues here and there.

Yeah talking with Image (the original writer of Instances), he made it sound like it was almost complete but I never saw a way to test it, and by the time he left and LE left, Scatman offered to gut Image's code and start over. But (sadly) Scatman has little time for Emu anymore so if what's there works, let's use it.

We can always "revamp" later. I should rename this project to EQ2Revampulator.net

READ ME

All I did to test it was set instance_type = 1 for forest ruins entry in the zones table then zoned into forest ruins from willow wood (/zone should take you to the base zone that all instances are copied from if I read the code right so had to go there the old fashion way). 2 chars went to there own instances I zoned client 1 back to willow wood, grouped the chars and zoned client 1 back to forest ruins and poped in the same instance as client 2, disbanded zoned client 1 out and back again and went back to the original instance assigned for client 1.

Was some issues with FK in the DB pointing to the wrong thing and the code trying to set it to 2 diffrent table id's, that should all work now though. If respawn on mobs is set to 0 in an instance then once they are dead they will not respawn, even after the zone shuts down or server restart. The code is a lot further along then I ever thought.

...continuing in public...

Holy crap, all that is in there already? I had no idea. I was looking for some command that just zoned me into an instance. Wow. In my defense, it's been years (like, 3 at least) since I looked at Image's work. I cannot believe instancing has been THAT functional all this time.

So what about zones that have varying populations related to the players level? We have to set up 1 actual zone (parent) for each one, then it can be instanced to separate groups/raids from there? Can you think of anything that's missing, config-wise?

Always wanted to add something like LDoN for EQ2 which I always thought would randomly spawn a zone full of mobs, but adjust their levels to match the avg of the player or group entering.

[Jabantiz]

I am still trying to learn all this code (while working on 2 other parts of the emu) so far what I can tell is we have a very solid starting point. From my tests it was working great never got mixed up into other instances, always returned to my instance when not in a group, if in a group and some one is already in the instance I zoned into that instance. I did a few code updates as well as a table update so when a mob was killed in an instance the console wouldn't spit out mysql errors. Also as stated above when a mob is killed in an instance and has a respawn time of 0 it stays dead in that instance even through zone shut down/server restart.

If I am reading the code right then /zone will take you to the base zone, going to the zone by normal means will put you in an instance. There is a "/zone instance id" command already that will zone you into the instance with the given id so admins can go into another players instance for whatever reason.

In the zones table all you have to do is set instance_type to turn that zone into an instance, from the server code

Code: Select all

enum Instance_Type {
	NONE,
GROUP_LOCKOUT_INSTANCE,
GROUP_PERSIST_INSTANCE,
RAID_PERSIST_INSTANCE,
SOLO_LOCKOUT_INSTANCE,
SOLO_PERSIST_INSTANCE,
TRADESKILL_INSTANCE, // allows anyone to enter, server searches for the first instance that is available
PUBLIC_INSTANCE // same as tradeskill, except dead spawns are tracked
};

So 0 = none 1 = group lockout 2 = group persist and so on. There is still work to be done to get the diffrent types of instances to work how they are suppose to but like I said we are at a very good starting point thanks to image.

So what about zones that have varying populations related to the players level?

No clue yet, we could make a lua function to use in a zone script to cycle through all the mobs to update there level/stats based on the player's level. Not all instances scale, and those that do may only scale so much that is why I suggest lua (and I suck with db work ), either way have not put much (any) thought into it yet.

We have to set up 1 actual zone (parent) for each one, then it can be instanced to separate groups/raids from there?

That is exactly how it works right now.

Can you think of anything that's missing, config-wise?

I honestly never paid much attention to how instances worked on live. I think we have everything we need but some one that knows exactly how all types of instances work would have to verify that. We have DB fields for the timers but none are currently used. I plan to get the instance window (alt + z by default) working to help me debug the timers when I get to them. Again any one else can feel free to work on this code.

PS - Huge thanks to Xinux and Zcoretri for finding the packets responsible for the instance window.

[alfa]

In the zones table all you have to do is set instance_type to turn that zone into an instance, from the server code

Code: Select all

enum Instance_Type {
	NONE,
GROUP_LOCKOUT_INSTANCE,
GROUP_PERSIST_INSTANCE,
RAID_PERSIST_INSTANCE,
SOLO_LOCKOUT_INSTANCE,
SOLO_PERSIST_INSTANCE,
TRADESKILL_INSTANCE, // allows anyone to enter, server searches for the first instance that is available
PUBLIC_INSTANCE // same as tradeskill, except dead spawns are tracked
};

I'm think there missing RAID_LOCKOUT_INSTANCE, like group or solo, some raid zone (especally <= T5) are not persistent, when you zone out, you have to wait for the end of timer lock.

For info, some instance need a specific flag to be toogle (like for CoD, you have to speak to a goblin to switch instance you want to zone to). It can be also an equiped item (like for draggon instance in DoF where you need scale for enter.

For instance who are "level adaptive" it ever take biggest level in group for set it.

For timer, you have imporant things, first the lock timer is enabled when you kill the first named mob, not trash (some tweak for raind instance, it enabled at first epic kill so if trash is epic, you timed), second things, for persistant instance you have two times, persist time = time you can go in instance to continue it (the most longer), reset time = time you can reset timer to have a fresh instance a zone in. On live you cannot be flaged more time on the same instance, if you have enabled a zone timer, each time you zone in (solo, group, raid) you go in YOUR instance. If another player have another instance flagged it tell you player are not ready and they have to reset their timer or leave group. A single player can get a full raid flagged on this own instance if he zone first. Players get same timer as player who previously flagged instance (used on live for raiding same zone two times in the same night :p)

And a N.B. A long time ago LE (or another dev, maybe image) talk about having a temp instance spawn table for persistant instance, when player zone in copy rows in temps, on kill delete it, so you don't have to store it in memory, and if the server down / reboot you can get instance at the point you left.
I think trash mob can have a zone timer to save it, and name should be instant, it avoid player to cheat for multiple kill named :p

[John Adams]

First off, let me give a huge THANKS to Jabantiz for taking the time to research this code. We have had this code for 3 years, and none of us could be bothered to review it So, thank you. Again.

Secondly, after hearing Jab's reports, a retro-thank you to Image for this effort. I guess he ditched EQ2Emu too quickly afterwards to really let us (noobs at the time) know it was THIS ready to use!

And a N.B. A long time ago LE (or another dev, maybe image) talk about having a temp instance spawn table for persistant instance, when player zone in copy rows in temps, on kill delete it, so you don't have to store it in memory, and if the server down / reboot you can get instance at the point you left.
I think trash mob can have a zone timer to save it, and name should be instant, it avoid player to cheat for multiple kill named :p

Pretty sure that's what instance_spawn_removed was for, though we had gone back and forth on the best way to do this. I think the solution was to populate these Instance tables with what they need, and remove them as they are defeated, in the instance with no-respawn potential. However, since we're mostly starting from scratch on this design, it can be anything we want or need it to be.


And another question - for zones like Tradeskill Workshops - basically 1 generic zone with differing names (per major city) - I always thought those were "instances" that people shared. Not sure of current SOE implementation, but I seem to remember them being labeled as instances. Either way, being a data miser that I am, I'm almost hopeful that all those zones based on the same "filename" can be set up as instances, eventually. We'd just have to figure a way to name them.

But for now, our bloated Zones tables are fine.

Jabantiz, don't feel you have to add yet-another-system to your list right now. We're just talkin here, so we can get to this later if you'd rather.

[Jabantiz]

Jabantiz, don't feel you have to add yet-another-system to your list right now. We're just talkin here, so we can get to this later if you'd rather.

Any thing to get me away from tradeskills lol. I don't mind working on this, gives me something to escape to when tradeskills give me problems (wich is almost always now).

And another question - for zones like Tradeskill Workshops - basically 1 generic zone with differing names (per major city) - I always thought those were "instances" that people shared. Not sure of current SOE implementation, but I seem to remember them being labeled as instances. Either way, being a data miser that I am, I'm almost hopeful that all those zones based on the same "filename" can be set up as instances, eventually. We'd just have to figure a way to name them.

They were instances, each village in the city, and the city itself, had its own version all were the same zone file though.


As for how it works now is the base zone is essentially loaded as a new instance, when a mob is kill an entry is put into instance_spawn_removed. When you go back and the instance starts up it will load the base zone but skip any spawns listed in instance_spawn_removed. I think this is a good way to do it as you can store less data this way and once a instance is loacked and can't be re entered the data in this table can be purged for the locked instance.

[alfa]

when a mob is kill an entry is put into instance_spawn_removed...I think this is a good way to do it as you can store less data this way...

A greate idea, mutch better than copy all spawns and delete after :p

[John Adams]

As for how it works now is the base zone is essentially loaded as a new instance, when a mob is kill an entry is put into instance_spawn_removed. When you go back and the instance starts up it will load the base zone but skip any spawns listed in instance_spawn_removed. I think this is a good way to do it as you can store less data this way and once a instance is loacked and can't be re entered the data in this table can be purged for the locked instance.

There is only one problem I see immediately with the instance_spawn_remove table; it is based on spawn_id, not spawn_location_placement.id. Meaning, if I zone into an instance that has 10 "a rat" spawns with an id of 1000, and I kill 1 rat, spawn_id 1000 gets put into the table and none of the other 9 placements will ever spawn should I reload the instance, for whatever reason.

Correct?

[John Adams]

In the zones table all you have to do is set instance_type to turn that zone into an instance, from the server code

Code: Select all

enum Instance_Type {
	NONE,
GROUP_LOCKOUT_INSTANCE,
GROUP_PERSIST_INSTANCE,
RAID_PERSIST_INSTANCE,
SOLO_LOCKOUT_INSTANCE,
SOLO_PERSIST_INSTANCE,
TRADESKILL_INSTANCE, // allows anyone to enter, server searches for the first instance that is available
PUBLIC_INSTANCE // same as tradeskill, except dead spawns are tracked
};

Also, Jab - can I / should I turn these into an ENUM in the database Zones table as well? While 0-7 makes sense to us, it would be easier if it were spelled out.

Code: Select all

ALTER TABLE `zones` CHANGE `instance_type` `instance_type` ENUM ('NONE', 'GROUP_LOCKOUT_INSTANCE', 'GROUP_PERSIST_INSTANCE', 'RAID_PERSIST_INSTANCE', 'SOLO_LOCKOUT_INSTANCE', 'SOLO_PERSIST_INSTANCE', 'TRADESKILL_INSTANCE', 'PUBLIC_INSTANCE') DEFAULT 'NONE' NOT NULL ;

And, which one of these types would I use for Zone::Max_Players rules? PUBLIC_INSTANCE sounds close, but not sure about the dead spawn tracking. Thoughts?


Edit: Oh, and how about Guild zones?