EQ2TC crashing when players log off

[John Adams]

Jababtiz, there is still something wrong with LD code, I'm afraid. Going to put the world back into Debug and see if it's simply a logwrite gone bad (usually is). No real details, since this was Release mode, but here's the crash card -

Code: Select all

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	EQ2World_x64.exe
  Application Version:	0.0.0.0
  Application Timestamp:	501ec614
  Fault Module Name:	StackHash_1dc2
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	00000000
  Exception Code:	c0000005
  Exception Offset:	0000000000000000
  OS Version:	6.1.7601.2.1.0.256.1
  Locale ID:	1033
  Additional Information 1:	1dc2
  Additional Information 2:	1dc22fb1de37d348f27e54dbb5278e7d
  Additional Information 3:	cbc5
  Additional Information 4:	cbc5ec6970b2af35927ad67117ca57e2

The player was booted due to desync or client crash:

07:42:56 D World: Timeout up!, state=
07:42:56 D Zone: Client is disconnecting in ZoneServer::ClientProcess (camping = false)
07:42:56 D Zone: Sending login equipment appearance updates...
07:42:56 D Zone: Calling clients.Remove(client)...
07:42:56 D Zone: Removing client 'Zenthrose' (127) due to LD/Exit...
07:42:56 I Zone: Scheduling client 'Zenthrose' for removal.
07:42:56 D Player: Toggling Character OFFLINE!
07:42:56 D CClient: Client Disconnect...

And 2 minutes later (LD Timer) World crashed. All that was running between player and zone shut down was NPC combat in QC.

2012-08-07_eq2world_2364.log

[Jabantiz]

If it was exactly when to LD timer was up that sounds like the problem with the client being removed before the spawn was, if that is the case this bug just keeps creeping its way back in. I will look into it and see if I can finally squash this bug.

[Jabantiz]

I looked into this today and after extensive logging It always seemed to crash at my DelayedSpawnRemoval() but no matter what I did it would still crash (crash does not happen all the time, seems random), so I took out the try/catch in ZoneServer::Process() and it ALWAYS reports the crash at ZoneServer::CombatProcess(). (CombatProcess and DelayedSpawnRemoval() run on diffrent threads)

Code: Select all

bool ZoneServer::CombatProcess() {
	bool ret = true;
	MutexList<Spawn*>::iterator itr = spawn_list.begin();
	while (itr.Next()) {
		if (itr->value->IsEntity()) { // ALWAYS crashes here whenever I can get it to crash
			if (!combat->Process((Entity*)itr->value)) {
				ret = false;
				break;
			}
		}
	}
	return ret;
}

DelayedSpawnRemoval() modifies spawn_list (via RemoveSpawn()), it looks like the list is modified while that loop is running causing the crash (in other words MutexList failing again). To test this I will have to replace MutexList with a vector<> and manual locks, sadly spawn_list is heavily used so it will take some time to replace it all, not to mention testing for possible deadlocks. I will try to work on this tomorrow.

[John Adams]

Good detective work, Jab. If you need help, and the replace is simple enough just time consuming, I can take some of the load. Give me a file to work in, and tell me what to do.

[Jabantiz]

It should all be in 1 file (zoneserver.cpp) and I will have to read the code to determine the lock to put in place as well as making sure it doesn't call a function that tries to lock the list again. If you want to do it feel free to, if not I will start it in a little while.

[John Adams]

Nah, you're the expert. It sounded like a global, huge task so I offered to assist. If it's just a few locks, I think you can manage

[Jabantiz]

And that has not solved the problem, other changes I have made have prevented the world from crashing but the zone now crashes in spawn loop and dead locks, a new version of the zone will start up when some on tries to log in, so it is some progress in the right direction just not enough yet.

[John Adams]

Sounds like it's getting worse, but you accidentally invented Instancing.

Nice.

[John Adams]

Just got another crash (thank you Alfa, for never /camp'ing properly ) and since i had try/catch commented out, I have some info.

Exactly 2 mins after LD timer starts, we die in a zone exception - this is likely what Jabantiz already knows, the spawn being deleted is already gone, so of course we crash. Problem is, the if(spawn) passes, so why is it crashing on the if(spawn->IsEntity())? At least that's where debugger ended up, which I think is incorrect... because why would I be in ~ZoneServer() on a zone not set to shut down for 3 more minutes? Stupid debugging.

Unhandled exception at 0x006adba6 in EQ2World__Debug.exe: 0xC0000005: Access violation reading location 0xfeeeff0e.

In ZoneServer::~ZoneServer() line 185 is RemoveLocationProximities(). This is where debugging seems to think the crash was (DeleteData).

Code: Select all

>	EQ2World__Debug.exe!ZoneServer::DeleteData(bool boot_clients)  Line 265 + 0xa bytes	C++
 	EQ2World__Debug.exe!ZoneServer::~ZoneServer()  Line 185	C++
 	EQ2World__Debug.exe!ZoneServer::`scalar deleting destructor'()  + 0x16 bytes	C++
 	EQ2World__Debug.exe!ZoneLoop(void * tmp)  Line 4357 + 0x22 bytes	C++
 	EQ2World__Debug.exe!_callthreadstart()  Line 259 + 0xf bytes	C
 	EQ2World__Debug.exe!_threadstart(void * ptd)  Line 243	C

Edit: I have lowered LD Timer (rule) to 100ms, just to see what happens next LD.

[alfa]

Always a pleasure, I have also notified a lot of lag (not due to me) especially when I was fighting (casting spell)

[John Adams]

The world is running in DEBUG with extensive logging enabled to try and catch exactly where issues arise. It is also hosted on a Windows 2003 x86 machine, since no one plays on it anymore but you or devs. If there were 10+ players regularly, I would host it on x64 and faster processors.

But the lag you are experiencing is either DEBUG (running from VS2010) or the combat changes and something not threading properly. We'll look into duplicating your issues.

Can you describe exactly what is happening? You swing, but do not hit for 5s? Need specifics so I can compare to logging.

[alfa]

First, hotbar icon color is long to change (when you can cast or you are too far away)
If you just cast one spell it seem to be ok, but if you start to chain spell, it seem to getting worse. Alot of combat anim start to miss... Ping is OK, connection too

[John Adams]

Ping is OK, connection too

Yeah, then this is a code thing, not a network thing probably related to DEBUG + Logging. Hopefully we'll figure out the remaining problems and run a fast server again soon. If more players were interested, it would be a no-brainer

[Jabantiz]

As stated here this may have been fixed with some combat/zone server changes I made. I ran the tests I was always able to reproduce this with and had no crash (did the test multiple times and tried other ways). Hopefully this new change will start pushing us back in the right direction and we will never see this crash again.