Looking back at performance

[Gangrenous]

Unfortunately I did lose some code through the years. I managed to get back almost everything but did lose a performance change I made. Not a bad thing, I know exactly how I tackled the issue so looking at it with fresh eyes will back it better this iteration. One thing I was looking at this morning was how the processes are spawned. Am I correct that the server spawns a process for every core it finds? I noticed on Debian I see 8 processes and I think that VM has 8 cores dedicated to it. My goal is to reduce cpu usage on my server, I have some code within zoneserver that checks for assist but only does it up to a rule range. If my memory servers correctly do the clients have knowledge of spawns in the entire zone or does CheckSpawnRange only store spawns within their range? I am just looking closer to see if there is any trimming that can be done in performance for large zones.

[Ememjr]

well fir of all why would you want to "reduce performance on my server"

is not the goal to make things run better and faster ie Increase preformance

just hounding you

but yes right now all spawns in a zone are sent to the zone,

I do remember a few years back you had changed something using Antonica as a test and only sent spawns to the client that were within (insert your range here) and you said that helped immensly

Jab is looking into something like that for the rewrite

this works as long as if you are tracking a looking for certain NPC's they could still be found server side, and then the X or dot or whatever is put on the map

if you get it figured out ill be happy to test in antonica as well, since i still have over 5000 total spawns in the zone on mine

[Gangrenous]

Change that to "reduce cpu usage"

[Gangrenous]

if you get it figured out ill be happy to test in antonica as well, since i still have over 5000 total spawns in the zone on mine

Are you running Linux or Windows? With just Antonica loaded, what are you getting on usage for that core?

I am just questioning a few things. Our performance as far as CPU goes is impacted by tons of checks for things that are not even close to being in range of the client. If we are at 0,0,0 is the client really worried about what is going on past maybe 1000 from their range? I think we are doing attack checks for NPC's that are close to each other clear across the zone. Sure if two NPC's are of opposing factions we want them to attack each other but not if there is no one to witness the attack. Much like the analogy "If a tree falls in a forest and no one is around to hear it, does it make a sound?" If no one can see or be concerned of the attack, why should it matter.

[Ememjr]

ill have to check tonight, i am runnings windows, but it does take a while to load antonica whereas most other zones are quick

[Gangrenous]

You can understand my concern I hope. As things are added and a zone can use 20-50% of a core, get 10 players in different large zones, we got a problem.

[Ememjr]

not sure if this helps

i am in debug mode runnign from VS2017, this one also running on same machine as my client all local except MySQL

tonight ill compare with my VM

but idle i was showing 15 threads and average of 2.17 cpu usage
after logging in and zoneing to antonica i now sitting at 5.02 average cpu usage
eq2 taking 15%
most of my 8 cores are betwen 20-30 percnet excpept #7 seems to be pinged at 100%

[Ememjr]

now on my vm i only have 4 cores

idle .2 of my cpu and 9 threads for the world
login to frost fang sitting idle now taking 11 threads and .21
zoned to antonica jumped up to .44 and 13 threads
once ully zoned in and standing idle 1.18 cpu usage and 13 threads

what i really need is another login to see what that does

[Ememjr]

so far based on what i am seeing on my end there doesnt seem to be a huge cpu burden yet, so adding 5000 spawns doesnt take much processer but it takes time, and to me time is important

dont want users sitting 3 minutes to zone

my antonica show 5461 NPC 's
305 Objects
50 widgets
261 signs
1792 ground spawns
355 spawn groups
18 spawn group associations
71 spawn group chances

[Gangrenous]

3 minutes to zone is ridiculous.

[Ememjr]

that will be resolved with your proximity thing on spawns going to client

[Gangrenous]

that will be resolved with your proximity thing on spawns going to client

Possibly.

[Ememjr]

wow only antonica, 2 minutes 45 secs from time server screen shows spawn count till the next step listeed, sending login info
20 secs to zone to qeynos

[Gangrenous]

This is preliminary of course. I would never release this to anyone until I test it thoroughly, I do not want to put out crappy work. This was 7203 total spawns in Antonica. With a radius of 300 around my character I have around 200 active spawns, that may be a little high. I think 100-120 radius is fine. I had reduced my attack checks to 1000 instead of 2000, I like that more responsive because of using assist. So what I have done is create a new vector called active_spawn_list and ever 2 seconds it clears its list and adds them back if there is a distance of distance of whatever, currently around 300. Doing all of this reduced my cpu usage from an average of 16-14% to a pretty kickass average of 4-6%. I am sure there is way more areas to increase performance as we go along. Of course this is also depending on how many characters are in the zone, they will very much have a radius around them of what mobs are moving and having attack checks.

In the screenshot you will see Antonica taking about 2% of a core, all processes together about 6%. I would be curious what you are seeing with that many in a zone, please report back.

My load time for eq2world was maybe 1 second, zoning maybe 1-2 seconds. A small thing would be if a player suddenly jumped to another part of the zone there could be up to a 2 second delay until the spawns appeared. Hell I experienced that in all kinds of games and it does not bother me. The delay could be adjusted for the admin as a rule, that is where I put all of my tweaked settings.

eq.JPG

I think the biggest culprit is we are taking every spawn and parsing a list, then checking that NPC against another iterating list for aggro. So as you can imagine this cumulative performance loss as you add more spawns. So if you have 8000 spawns and are check each one against the entire zone that is 64,000,000 checks I guess, I suck at this type of thing. If you reduce it down to where you have 200 around you then that is only 40,000 checks. Basically where it is calling this line...

Code: Select all

if (aggroCheck)
    ProcessAggroChecks(spawn);

I am only doing that and movement for active spawns within a radius. If you look at ProcessAggroChecks you will see how it is check each spawn against the entire zone.

That is just what I observed, I could be wrong.

[tyrbo]

I've never taken three minutes to zone. There is an issue with zone start up that started happening with the switch to the MariaDB libraries that I was experiencing while still running MySQL for the server. With MySQL as the server, it was taking several minutes to load largely populated zones. The second I swapped to MariaDB for the server, the load time for a large zone to start up dropped down to maybe ten seconds. After a zone is loaded, zone times are very quick.

As for the threads, iirc there's a few threads for reading/writing the actual packets, and then there's two or three threads for each loaded zone. Most of the threads have delays as well, so they shouldn't sit there eating up all your processing time.

We've had probably two groups worth of people on Stitch at a time. Can't say we had much lag aside from some that are more design related than anything else (like things that interact with the database).

CheckSpawnRange in the current code base iterates over the spawns and does two things currently. 1) It calculates the distance between the player and the spawn and saves that in a map for later reference. 2) For mobs that are in range, it checks for aggro. The map allows us to iterate over a small subset of the spawns based on their distance. Limiting the active spawns based on player range is a good idea though, I remember we talked about that before on Discord.

Anyway, your load time issue may ultimately be a result of the server software you're running. It was for me. I suffered through long load times that came out of nowhere for quite a long time before by chance I loaded up MariaDB for one of my dev servers instead of MySQL.

edit: I glanced over ProcessAggroChecks. I'm not super familiar with that code, but it looks like it's also only working with a subset of spawns for each spawn.