First by dead client I mean the world stops responding but the client never disconnects, you could sit in there forever running around the zone with out loosing connection. I am sure all us devs have seen this at least once.

Now what it actually is, a zone server crash/deadlock.

While trying to reproduce a bug Foof brought to my attention I ran across this, again. I was able to determine the zone had crapped out on me and the reason I was never disconnected was because the ping was still keeping me alive, the ping is maintained by the world server and not the zone server like all the visible stuff is so when the zone crapped out every thing stopped working but the connection never times out.

I am not sure how the zone was dying without giving any errors and without actually shutting down. I did have a debugger attached so I was able to pause the server until the client disconnected and then resumed and while the world server registered the client timed out the zone server refused to let the client go, finally I typed exit in the console, sadly the stubborn mostly dead zone refused to give in and shut down.

Debugger showed it stuck trying to get the players target, made some changes to how we get a target and how we clear it when a spawn is removed. I really doubt this will fix the dead client issue, however I did not encounter it again in my attempt to reproduce the bug when I ran into it so quickly the first time so who knows maybe it will.

If any of you run across this again can you please post detailed info on what you were doing in game, and if at all possible reproduce what I did with the debugger, pause until the client times out, resume and wait a bit then type exit in the console, wait a bit then pause again and see where it is stuck.

This used to happen to me a lot in the old Queen's Colony zone, since that was where I was doing most of my testing back in the day. Exact symptoms, there are probably a dozen posts by me about this issue already.

Hopefully it happens to you rampantly, so it can be resolved Is there any additional "heartbeats" you can put between World and Zone where World will dominate over the zones as it should, and maybe force a zone shutdown? Or is the problem that Zone won't even hear of it? I know it's probably a thread, not a process, so killing it brutally may not work. I believe terminating threads brute-force is not a good idea either.

Thoughts?

I considered something like this, have the zone send a message to the world, if the world doesn't get a message kill the thread, will need to maintain a list of pointers to all zone threads and all threads a zone starts itself, forcibly killing a thread won't clear the memory it allocated though so will have a massive memory leak every time we have to manually kill it, so just going to leave it to hopefully get a proper fix.

I have seen this before but not for a long time, only instance I was able to capture it with a debugger too, though that info didn't seem to helpful. After the changes I was not able to get this again but it may have just been bad luck I got it the first time.