Crashes / Disconnects

[LethalEncounter]

I had the same thing happen to me that happened to John previously. Logged in and world wasn't responding to me, but I could watch you guys chat.

Oops, I forgot to /unshun you...


j/k


I think I fixed the cause of that deadlock. I'll leave my server up all night tonight if someone wants to try to crash it, feel free I should probably do that more often so that I can fix these issues before they become as big as they were this time.

[Scatman]

It's on!! =)

[LethalEncounter]

I had a crash last night caused by a combat bug and then another this morning caused by a deadlock while processing player items. Both have been fixed. I'll leave it running all day today and fix any errors that occur.

[John Adams]

Thank you LE for helping us get to the bottom of these nightmares. I know it's frustrating, but all good in the end.

[LethalEncounter]

No problem, it wouldn't have been a problem if I had fixed them whenever they started happening. Instead they all compounded into the mess it is now. See the problem is usually not a single lock that causes it. That would be simple to fix. The issue is that there are multiple threads doing everything and sometimes the two threads want to use the save resource. Well the locks prevent them from doing this which prevents lots of random errors. The deadlock issue comes into play whenever thread A has a lock on one resource and then requests a resource that thread B is using. If thread B requests the resource that thread A is using before releasing the resource it is using, then a deadlock occurs because both threads are waiting for the other to finish. This sounds simple enough to fix, but there are so many complex procedures going on in the background that the issue really is hard to track down.

[John Adams]

Aye, I am very familiar with CriticalSections and Deadlocks. They were a part of my everyday life as a QA guy with the extremely complex software our company used to develop (before it got gobbled up by Quest Software and all our hard work tossed to the weeds, f**kers).

I am ready to implement any new code you might want to test on Tess, if you feel you have enough sorted out to make a difference. Any crashe I see at the moment are usually a variation of a previously reported crash... mostly, SpawnCache(). So, gimme what ya got. And we'll keep crashing your server as long as you need.

[LethalEncounter]

OK, could you try to crash my server one last time? I'll post the code shortly after. I just want to fix any simple problems if they occur.

[John Adams]

You know I'd love to do that. =) brt

[John Adams]

Soon as I came back over to you after i killed a hawk, the chat stopped working. I heard Defender Bristlebrow aggro something, tho he didn't move from his spot. So the world must have thought things were still moving - but then after 30s, nothing.

[LethalEncounter]

Ahh this was was an easy fix, a deadlock caused by another zone shutting down.

[LethalEncounter]

btw to answer your question, it usually uses 0 to 1% of the CPU. It only goes above that whenever someone is zoning in and it is processing all the spawns for them. I suppose it would also go up if you were fighting a crap load of spawns.

[John Adams]

Ahh this was was an easy fix, a deadlock caused by another zone shutting down.

Oh yeah, I meant to tell you about that one last night. I saw a zone shut down after I left it, and the world crashed =) But I had enough of crash talk last night and watched a slasher movie instead.


As for CPU usage... I will take some stats to show you what I am talking about. Idle, of course the world uses very little ram or cpu. But when people are moving around, and of course especially at /reload (which only seems to happen when Scatman is online), the CPU goes through the roof even in Release build. Now this may be a deficiency in the database, which I will look into improving. Maybe we can improve load times with more indexing.

Code generally is quiet, though. Just curious about those spikes.

[Image]

I am starting to rip out mutexes on my code, the lock-ups are getting annoying

[John Adams]

That sounds painful, Image.

LE, just as you were shutting down, I looked over to my Jadams client and saw that his console hadn't received any of our chats since before I did my /speed 1000 test. Not sure if that's something else to look into, or wait til it happens again.

[LethalEncounter]

Might have been a desync problem. We definitely didnt have any deadlocks that time.

I think this solves most of the problems we were having. One issue that both John and I saw were that the guards seemed to stop attacking after a while. Not sure what is causing that, but I'll look into it later.

Hey John, I checked in some parser changes as well, but I'm not finished with them so don't use the new code just yet unless you want to debug some stuff.