Harvesting Pull: Design

Discussions of the design and development of in-game content.

Moderator: Team Members

User avatar
Eradani
Posts: 192
Joined: Wed May 05, 2010 6:25 am
Location: Saskatchewan

Re: Harvesting Pull: Design

Post by Eradani » Wed Jun 16, 2010 1:00 pm

John Adams wrote:You have harvested 1000000000000000000 times? Jeepers, you must love to harvest!
OMG it must have been at least 1000000000000000000 times. First I lvl'd all 9 crafters to 60 harvesting, then they put in those tradeskill quests from the grandmaster, so i had to harvest through all those up to 70, then i came back to find out about the harvest T1-T8 quest, so i had to to that again too!!!

I never had track, so I used to have to find Qs by harvesting and pretending to not pay attention. hehe
my client version: 12682L, 2016/06/06
last one that will run on XP cause i'm just a stick-in-the-mud

User avatar
Eradani
Posts: 192
Joined: Wed May 05, 2010 6:25 am
Location: Saskatchewan

Re: Harvesting Pull: Design

Post by Eradani » Thu Jun 17, 2010 7:59 pm

was fooling around with Mysql Workbench last night and this is what i was thinking for database structure concerning harvesting. snooping around today and i found something a little different for ground spawns (they contained more information than i thought they might). also, i have to figure out how the loot tables are setup as this might help with the last step of deciding what to drop when i get, for instance, 3 common from a bush.

Image

- spawn.name would be like "corrupted arbor" (or w/e from graveyard)
- as far as i know, the spawn.name is unique to a node type and tier, usually also a zone
my client version: 12682L, 2016/06/06
last one that will run on XP cause i'm just a stick-in-the-mud

User avatar
John Adams
Retired
Posts: 9684
Joined: Thu Jul 26, 2007 6:27 am
EQ2Emu Server: EQ2Emulator Test Center
Characters: John
Location: Arizona
Contact:

Re: Harvesting Pull: Design

Post by John Adams » Sun Jun 20, 2010 1:29 pm

Nice work, Eradani. I think you got the right idea, but (as a DB guy) I am not seeing the need for so many individual tables.

Did you read this post? I had re-designed our groundspawn* tables a while ago moving towards the more SOE-like style of pushing the right item(s) during a harvest. Maybe I am way off base, but it was a long time ago as well. Who knows if it's even valid anymore.

User avatar
Eradani
Posts: 192
Joined: Wed May 05, 2010 6:25 am
Location: Saskatchewan

Re: Harvesting Pull: Design

Post by Eradani » Sun Jun 20, 2010 9:04 pm

The harvesting system currently in place really has nothing to do with how harvesting works. I hate stepping on toes, but I'm gonna have to if you want harvesting to work like it does on live.
LethalEncounter wrote:OK, These are my revised tables:

Code: Select all

CREATE TABLE `groundspawn_items` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `groundspawn_entry_id` int(10) unsigned NOT NULL default '1',
  `item_id` int(10) unsigned NOT NULL default '0',
  `num_items` tinyint(3) unsigned NOT NULL default '1',
  `min_score` smallint(5) unsigned NOT NULL default '1',
  `min_rare_level` tinyint(3) unsigned NOT NULL default '0',
  `rare_item` tinyint(3) unsigned NOT NULL default '0',
  `triggers_bonus_check` tinyint(3) unsigned NOT NULL default '0',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `ItemIdx` (`groundspawn_entry_id`,`item_id`,`num_items`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

CREATE TABLE `spawn_ground` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `spawn_id` int(10) unsigned NOT NULL default '0',
  `number_harvests` tinyint(3) unsigned NOT NULL default '3',
  `num_attempts_per_harvest` tinyint(3) unsigned NOT NULL default '1',
  `groundspawn_entry_id` int(10) unsigned NOT NULL default '0',
  PRIMARY KEY  (`id`),
  KEY `SpawnIdx` (`spawn_id`),
  KEY `GroundIdx` (`groundspawn_entry_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

CREATE TABLE `groundspawn_bonus` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `groundspawn_entry_id` int(10) unsigned NOT NULL default '1',
  `item_id` int(10) unsigned NOT NULL default '0',
  `num_items` tinyint(3) unsigned NOT NULL default '1',
  `percentage` tinyint(3) unsigned NOT NULL default '100',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `id` (`id`),
  UNIQUE KEY `GroundSpawnBonusIdx` (`item_id`,`num_items`,`groundspawn_entry_id`,`percentage`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
The function Source\World\GroundSpawn.cpp: "void GroundSpawn::ProcessHarvest(Client* client)" is full of rather convuluted fudges towards some "liveish" kind of harvesting distribution.

I just don't know what to say except - this just isn't how eq2 harvesting (or collecting for that matter) works at all.


Explanations for the tables used in my diagram above:

- Spawn: subset of the spawn information that I need
- Toon Skills: subset of toon information that I need
- Nodes: lookup table for node type and tier based on spawn.name
- Skill Type: I see that you've put this into the spawn_ground table.
- Skill Requirements: used in step 4 below. Would be easy to put this in a .h file, except the small amount of DB space would allow tuning on the fly. Doesn't matter to me if it's dynamic (DB) or static (.h).
- Pull Type: used in step 8 below. A horizontal version of a loot table designed this way to save a little space somewhere and make it easier to visualize and, therefore, maintain.

How these relate to the current DB schema:

I can see the database structure with MySQL Workbench, but still don't have Apache running so can't see any data. I also can't get (win)mysqladmin.exe to run and assume the problem is related to that localport:80 problem.

Assumptions:
- the current Spawn table is 1 row for every NPC, widget, sign, shiney, node, ... that gets spawned.
- the Spawn table points at the various Spawn_widgets, Spawn_signs, Spawn_objects, Spawn_loot, and Spawn_ground for specific details
- the Spawn_scripts table is also pointed to by the Spawn table even though, atm, there's no index indicating this.

Here's a pic of selected tables from the current database schema. It's too wide to fit here, so here's the
link if you want to DL it. (and I lol'd at the NewIndex also)
Image

- my Spawn table: corresponds to current Spawn.name + Spawn_ground, however I really don't get the num_attempts_per_harvest field at all.
- my Toon Skills table: corresponds to current Character_skills and Skills tables.
- my Nodes table: this information doesn't seem to be anywhere. The node type is extrapolated from the harvesting skill used in ProcessHarvest(), but this won't work as Mining works on both Ore and Rock node types. Also, in ProcessHarvest(), collecting is turned into gathering and this is not right at all. Collecting is a separate skill for things such as shinies and is binary (you have it or you don't) as opposed to skill level based.
- Skill Type: a small table for getting the skill used. I see this information in the Spawn_ground table and this may indeed be the place for it. I was thinking it wouldn't be good to accidentally use fishing skill on a rock node, however with quest based nodes, you might use collecting or gathering on rock nodes. (now that I mention quest gathering - you ALWAYS get a skillup when gathering for a quest. grr, more complications)
- Skill Requirements and Pull Type: some of this information is currently stored in the Groundspawn_items table. However, these 2 tables plus ProcessHarvest() are so contradictory concerning eq2 harvesting that I would like to just scrap them. You really don't need this information for every node (or Spawn.name) - only for each node type per tier.
my client version: 12682L, 2016/06/06
last one that will run on XP cause i'm just a stick-in-the-mud

User avatar
Eradani
Posts: 192
Joined: Wed May 05, 2010 6:25 am
Location: Saskatchewan

Re: Harvesting Pull: Design

Post by Eradani » Sun Jun 20, 2010 9:09 pm

Algorithm for ONE Harvesting Pull: (based on numerous Official posts about how harvesting works)

1. given the node.name (ie: corrupted arbor)
2. determine this is T1, Wood, using Foresting skill
3. get toon's current and maximum Foresting skill (these are modified skills that include any buffs (or debuffs))

4. for each Harvesting Tier there is a row in a table (db or memory) that states:
- the base skill needed to harvest a node
- the skill that the node trivializes at
- the skill that you need for a chance at a rare

5. if your skills aren't maxed AND your currentSkill < node.trivial, you get a chance at a fail or a skillup (obviously can't get skillup if you fail)
- if Fail, then exit algorithm

6. if your current skill >= node.bonus, generate random(node.base .. toon.currentSkill)
- if this number >= node.bonus, then you get a pull from the bonus table, otherwise, you get a pull from the base table

7. for each Tier + Node Type there are 2 table rows (base and bonus) in a Pull Type table. This table has columns "1 common", "3 common", "5 common", "1 imbue", "1 rare", "10 common + 1 rare" with the chance for each pull type.

8. generate random(1 .. 1000) allowing for 100% with 1 decimal place. Get the pull type column.

9. (not coded in .js yet) decide the item to be returned. If it's a common, there may be up to 4 choices. If it's an imbue, there's one choice per tier. If it's a rare, there's 0, 1, or 2 choices.

10. return item(s) harvested and decrement node.harvestsRemaining. If harvestsRemaining == 0, delete node.

Notes:
As of the Thu, June 17, 2010, I hadn't gone through our database so I was just making up what I needed for harvesting. I've spent the last couple of days trying to reconcile my algorithm with what we currently have.

1. My node.name is currently spawn.name. I'm just using node as an internal object to store the attributes that I need for harvesting a node.

2. This node.name uniquely identifies the tier of the node, the node type. Each node.name may appear in several zones, ie: bush nodes.

3. As far as I know, harvesting skills only get buffs from items (tools and the gear from Mara). I'm sure there's no harvesting debuffs, but if we allow "modified skill" to include them, like it does for defense for instance, that opens up some nice mean and nasty options.

4. The table I'm using atm is:

Code: Select all

base    =  0, 20,  90, 140, 190, 240, 290, 340, 390
bonus   = 39, 89, 139, 189, 239, 289, 339, 389, 439
trivial = 45, 95, 145, 195, 245, 295, 400, 450, 550
I am fairly sure it's coded on live and not DB based. However, if it were DB based, changes would be easier to check and not have to wait for a "hotfix" recompile.

5. Fails: Various official quotes state that you can only get a fail if your skill isn't maxed AND your current skill is under the trivial level.
Zcoretri wrote:

Code: Select all

One, Three ... Many?
Harvesting changes hit the live servers with LU24 (June 14)... on any harvest where your current skill is above the maximum skill raise cap for the area, you will not fail on a harvest.
6. This is your chance at getting a pull from the bonus table based on how much over the node.bonus level your skills are. Previously, this was at the (T+1)1 level, but now is at the T8 level.

7. I really tried to make it only for each Tier OR only for each Node Type, but reading over Domino's posts has convinced me it must be for each Tier + Node Type.

Using Domino's sample data for T4 root nodes base table, I made a bonus table with slightly higher chances for everything but "1 common". From there, I made another table for Fish, Ore, and Rock nodes because they have 0, 2, 2 rares. Then you have to add an exception for T9 Ore as loam is gone the way of the dodo. Then there's exceptions for T1 as there's no imbue and that 1% has to go somewhere. I ended up with: (255 is default)

Code: Select all

var PullTable = [
//              Type    Tier   Table          a     b     c    d    e    f
new _PullTable("fish",  1,    "base",     [ 70.7, 20.2,  9.1, 0.0, 0.0, 0.0 ]),
new _PullTable("fish",  1,    "bonus",    [ 56.1, 30.3, 13.6, 0.0, 0.0, 0.0 ]),
new _PullTable("bush",  1,    "base",     [ 70.0, 20.0,  8.0, 0.0, 1.2, 0.8 ]),
new _PullTable("bush",  1,    "bonus",    [ 55.0, 30.0, 12.0, 0.0, 1.7, 1.3 ]),
new _PullTable("root",  1,    "base",     [ 70.0, 20.0,  8.0, 0.0, 1.2, 0.8 ]),
new _PullTable("root",  1,    "bonus",    [ 55.0, 30.0, 12.0, 0.0, 1.7, 1.3 ]),
new _PullTable("wood",  1,    "base",     [ 70.0, 20.0,  8.0, 0.0, 1.2, 0.8 ]),
new _PullTable("wood",  1,    "bonus",    [ 55.0, 30.0, 12.0, 0.0, 1.7, 1.3 ]),
new _PullTable("den",   1,    "base",     [ 70.0, 20.0,  8.0, 0.0, 1.2, 0.8 ]),
new _PullTable("den",   1,    "bonus",    [ 55.0, 30.0, 12.0, 0.0, 1.7, 1.3 ]),
new _PullTable("ore",   1,    "base",     [ 69.0, 20.0,  8.0, 0.0, 1.9, 1.1 ]),
new _PullTable("ore",   1,    "bonus",    [ 54.9, 30.0, 12.0, 0.0, 1.8, 1.3 ]),
new _PullTable("rock",  1,    "base",     [ 69.0, 20.0,  8.0, 0.0, 1.9, 1.1 ]),
new _PullTable("rock",  1,    "bonus",    [ 54.9, 30.0, 12.0, 0.0, 1.8, 1.3 ]),

new _PullTable("ore",   9,    "base",     [ 70.0, 20.0,  8.0, 1.0, 0.7, 0.3 ]),
new _PullTable("ore",   9,    "bonus",    [ 54.9, 30.0, 12.0, 1.5, 1.1, 0.5 ]),
new _PullTable("fish",  255,  "base",     [ 70.7, 20.2,  8.1, 1.0, 0.0, 0.0 ]),
new _PullTable("fish",  255,  "bonus",    [ 56.1, 30.3, 12.1, 1.5, 0.0, 0.0 ]),
new _PullTable("ore",   255,  "base",     [ 69.0, 20.0,  8.0, 1.0, 1.4, 0.6 ]),
new _PullTable("ore",   255,  "bonus",    [ 53.5, 30.0, 12.0, 1.5, 2.1, 0.9 ]),
new _PullTable("rock",  255,  "base",     [ 69.0, 20.0,  8.0, 1.0, 1.4, 0.6 ]),
new _PullTable("rock",  255,  "bonus",    [ 53.5, 30.0, 12.0, 1.5, 2.1, 0.9 ]),
new _PullTable(255,     255,  "base",     [ 70.0, 20.0,  8.0, 1.0, 0.7, 0.3 ]), // domino's example data
new _PullTable(255,     255,  "bonus",    [ 54.9, 30.0, 12.0, 1.5, 1.1, 0.5 ])
];
where column a = 1 common, b = 3 common, c = 5 common, d = 1 imbue, e = 1 rare, f = 10 common + 1 rare.

Now we know that T1 has a high rare drop rate and as you increase in tier, the rare drop rate goes down. I haven't done anything with making this adjustment yet, but I did write a small program to take the above data and make rows for every Type + Tier + Table, leading to 126 rows.

As to where this data is stored: DB or internal - I believe that this table is included at compile time as one time when Domino was very new, she made a "tiny" error that cause T2 (at least) rares to be more common than commons and this change could not be fixed for a month till the next update cycle.

The one idea I've had recently, is making a DB table with 6 rows only: T1 + (0, 1, 2 rares) + Table (base, bonus). This could be converted, at run time, to an internal table with a declining rare drop rate based on a formula. One consequence of this, I'm fairly sure, is that 1 decimal place won't be enough. Somewhere, I saw Domino state that rare drop rates would increase by no more than 0.22%. I'm sure she isn't confused and meaning 22%.

8. Data given in an official example for T4 root nodes base table:
1 common = 700, 3 common = 200, 5 common = 80, 1 imbue = 10, 1 rare = 7, 10 common + 1 rare = 3

9. We now have Node Type, Tier, and 1 of columnName[a..f] (above). We need to translate that into the actual item that's returned. I've hesitated making any code for this yet as I wanted to see how loot tables were implemented here. What I'd been thinking is column d is easy as there's only one imbue for the whole tier, column a..c would need some loot table with chance for that common, column e is basically the same kind of loot table. Column f would need to get a common from the loot table and then get the rare for that node type, except in the case of ore and rocks, where iron goes with blackened iron and not alkaline loam.

Then, there's this disturbing quote from Domino "if you get a single item harvest from a trapping node, you have twice the chance of it being a pelt than a meat. For 3 and 5 item harvests they're equal chances". This implies that each pull type column has it's own loot table for 7(type) * 9(tier) * 6(columns) = 378 loot tables, each having a possible 4 entries. IDK but this seems like "customizable overkill" to me.
my client version: 12682L, 2016/06/06
last one that will run on XP cause i'm just a stick-in-the-mud

User avatar
Eradani
Posts: 192
Joined: Wed May 05, 2010 6:25 am
Location: Saskatchewan

Re: Harvesting Pull: Design

Post by Eradani » Sun Jun 20, 2010 9:10 pm

So, atm I'm lost on determining the exact item(s) returned, but I've coded everything up to that point in javascript for testing.

This works for the IE Dom, specifically in IE6. It does not work in the newest upgrade of Firefox and I don't know what else it doesn't work in and frankly I don't care. I figured I could spend many hours on browser compatibility or just nvm and work on harvesting. Just FYI, I'm using frames cause, well, I like frames.

Here's the links for each file if you want to look at them.
- index.html
- 5 harvesting test.html
- 5 messages.html
- library.js
- pullTable.js
my client version: 12682L, 2016/06/06
last one that will run on XP cause i'm just a stick-in-the-mud

User avatar
John Adams
Retired
Posts: 9684
Joined: Thu Jul 26, 2007 6:27 am
EQ2Emu Server: EQ2Emulator Test Center
Characters: John
Location: Arizona
Contact:

Re: Harvesting Pull: Design

Post by John Adams » Mon Jun 21, 2010 7:13 am

Well, you certainly know more about harvesting than me... and regardless what I say, sounds like I'm wrong so - being an open source community project, take the code, change it to be how you think it should be and submit your changes.

I always thought it would be much simpler than that, but I guess not.

User avatar
Zcoretri
Team Member
Posts: 1642
Joined: Fri Jul 27, 2007 12:55 pm
Location: SoCal

Re: Harvesting Pull: Design

Post by Zcoretri » Mon Jun 21, 2010 8:33 am

WOW, haha...that epic post rivals one of yours JA :mrgreen:
Looking forward for this implemetation of harvesting.

User avatar
Eradani
Posts: 192
Joined: Wed May 05, 2010 6:25 am
Location: Saskatchewan

Re: Harvesting Pull: Design

Post by Eradani » Mon Jun 21, 2010 9:10 am

John Adams wrote:Well, you certainly know more about harvesting than me... and regardless what I say, sounds like I'm wrong so - being an open source community project, take the code, change it to be how you think it should be and submit your changes.

I always thought it would be much simpler than that, but I guess not.
Don't say it that way "regardless what I say, sounds like I'm wrong". Cause there's 28327498 other things that you know that I don't. But I have harvested and been interested in harvesting and read about harvesting all the time I've played eq2.

In the meantime, since this site is either going away or in some terrible limbo, Ima going to make a trial account and harvest for 2 weeks. Any chance for some src code for the collector so I can parse my pulls?
my client version: 12682L, 2016/06/06
last one that will run on XP cause i'm just a stick-in-the-mud

User avatar
Zcoretri
Team Member
Posts: 1642
Joined: Fri Jul 27, 2007 12:55 pm
Location: SoCal

Re: Harvesting Pull: Design

Post by Zcoretri » Mon Jun 21, 2010 1:50 pm

The collector will not help you in that regard. If you want to track your harvesting results, I would use ACT (Advanced Combat Tracker). I believe it has a add-on that will do want you want.

I will dig up the URL and post here.

You can get ACT from here http://advancedcombattracker.com/

A link to the plug-in here: Harvest Tracker

User avatar
Eradani
Posts: 192
Joined: Wed May 05, 2010 6:25 am
Location: Saskatchewan

Re: Harvesting Pull: Design

Post by Eradani » Mon Jun 21, 2010 4:11 pm

Zcoretri wrote:The collector will not help you in that regard. If you want to track your harvesting results, I would use ACT (Advanced Combat Tracker). I believe it has a add-on that will do want you want.

I will dig up the URL and post here.

You can get ACT from here http://advancedcombattracker.com/

A link to the plug-in here: Harvest Tracker
ya, i think you're right about te collector not doing what i want. ACT, while nice, doesn't quite do what i want either though. It was a good try though, thx anyways.
my client version: 12682L, 2016/06/06
last one that will run on XP cause i'm just a stick-in-the-mud

User avatar
Eradani
Posts: 192
Joined: Wed May 05, 2010 6:25 am
Location: Saskatchewan

Re: Harvesting Pull: Design

Post by Eradani » Tue Jul 13, 2010 6:43 pm

got my new hard drive, so i can start reinstalling windows so that everything works again, hopefully. then i can start translating everything i've been doing into sql / php / w/e else is more appropriate than javascript.

i've got my log parser working now for harvesting, just working on analyzing the data.
my client version: 12682L, 2016/06/06
last one that will run on XP cause i'm just a stick-in-the-mud

User avatar
John Adams
Retired
Posts: 9684
Joined: Thu Jul 26, 2007 6:27 am
EQ2Emu Server: EQ2Emulator Test Center
Characters: John
Location: Arizona
Contact:

Re: Harvesting Pull: Design

Post by John Adams » Wed Jul 20, 2011 1:44 pm

If Eradani ever comes back and reads this ;)

I have completely re-written ProcessHarvest() since the original code. It uses much more configurable percentages per our previous discussions, however it is still in development. Example, I do no calcs using bonus gear or anything right now, since Items are not really fleshed out anyway. I also do not use Loot tables, because the loot system needs a re-do anyway. However, in all my own testing, results match Live pretty accurately. I wish more people were interested in testing, as Eradani did... because my view of Harvesting is not as granular.

When I have more time, I will re-re-read all of Eradani's posts here and check out the samples, and attempt to beef up ProcessHarvest() v3.0 when i can soak up all this info.
John Adams
EQ2Emulator - Project Ghost
"Everything should work now, except the stuff that doesn't" ~Xinux

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests