Page 1 of 1
EQ2TC crashing when players log off
Posted: Tue Aug 07, 2012 12:02 pm
by John Adams
Jababtiz, there is still something wrong with LD code, I'm afraid. Going to put the world back into Debug and see if it's simply a logwrite gone bad (usually is). No real details, since this was Release mode, but here's the crash card -
Code: Select all
Problem signature:
Problem Event Name: APPCRASH
Application Name: EQ2World_x64.exe
Application Version: 0.0.0.0
Application Timestamp: 501ec614
Fault Module Name: StackHash_1dc2
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 00000000
Exception Code: c0000005
Exception Offset: 0000000000000000
OS Version: 6.1.7601.2.1.0.256.1
Locale ID: 1033
Additional Information 1: 1dc2
Additional Information 2: 1dc22fb1de37d348f27e54dbb5278e7d
Additional Information 3: cbc5
Additional Information 4: cbc5ec6970b2af35927ad67117ca57e2
The player was booted due to desync or client crash:
07:42:56 D World: Timeout up!, state=
07:42:56 D Zone: Client is disconnecting in ZoneServer::ClientProcess (camping = false)
07:42:56 D Zone: Sending login equipment appearance updates...
07:42:56 D Zone: Calling clients.Remove(client)...
07:42:56 D Zone: Removing client 'Zenthrose' (127) due to LD/Exit...
07:42:56 I Zone: Scheduling client 'Zenthrose' for removal.
07:42:56 D Player: Toggling Character OFFLINE!
07:42:56 D CClient: Client Disconnect...
And 2 minutes later (LD Timer) World crashed. All that was running between player and zone shut down was NPC combat in QC.
2012-08-07_eq2world_2364.log
Re: EQ2TC crashing when players log off
Posted: Tue Aug 07, 2012 3:02 pm
by Jabantiz
If it was exactly when to LD timer was up that sounds like the problem with the client being removed before the spawn was, if that is the case this bug just keeps creeping its way back in. I will look into it and see if I can finally squash this bug.
Re: EQ2TC crashing when players log off
Posted: Wed Aug 08, 2012 12:40 am
by Jabantiz
I looked into this today and after extensive logging It always seemed to crash at my DelayedSpawnRemoval() but no matter what I did it would still crash (crash does not happen all the time, seems random), so I took out the try/catch in ZoneServer::Process() and it ALWAYS reports the crash at ZoneServer::CombatProcess(). (CombatProcess and DelayedSpawnRemoval() run on diffrent threads)
Code: Select all
bool ZoneServer::CombatProcess() {
bool ret = true;
MutexList<Spawn*>::iterator itr = spawn_list.begin();
while (itr.Next()) {
if (itr->value->IsEntity()) { // ALWAYS crashes here whenever I can get it to crash
if (!combat->Process((Entity*)itr->value)) {
ret = false;
break;
}
}
}
return ret;
}
DelayedSpawnRemoval() modifies spawn_list (via RemoveSpawn()), it looks like the list is modified while that loop is running causing the crash (in other words MutexList failing again). To test this I will have to replace MutexList with a vector<> and manual locks, sadly spawn_list is heavily used so it will take some time to replace it all, not to mention testing for possible deadlocks. I will try to work on this tomorrow.
Re: EQ2TC crashing when players log off
Posted: Wed Aug 08, 2012 2:51 pm
by John Adams
Good detective work, Jab. If you need help, and the replace is simple enough just time consuming, I can take some of the load. Give me a file to work in, and tell me what to do.
Re: EQ2TC crashing when players log off
Posted: Wed Aug 08, 2012 3:39 pm
by Jabantiz
It should all be in 1 file (zoneserver.cpp) and I will have to read the code to determine the lock to put in place as well as making sure it doesn't call a function that tries to lock the list again. If you want to do it feel free to, if not I will start it in a little while.
Re: EQ2TC crashing when players log off
Posted: Wed Aug 08, 2012 4:21 pm
by John Adams
Nah, you're the expert. It sounded like a global, huge task so I offered to assist. If it's just a few locks, I think you can manage
Re: EQ2TC crashing when players log off
Posted: Thu Aug 09, 2012 2:50 pm
by Jabantiz
And that has not solved the problem, other changes I have made have prevented the world from crashing but the zone now crashes in spawn loop and dead locks, a new version of the zone will start up when some on tries to log in, so it is some progress in the right direction just not enough yet.
Re: EQ2TC crashing when players log off
Posted: Thu Aug 09, 2012 6:16 pm
by John Adams
Sounds like it's getting worse, but you accidentally invented Instancing.
Nice.
Re: EQ2TC crashing when players log off
Posted: Thu Aug 16, 2012 12:44 pm
by John Adams
Just got another crash (thank you Alfa, for never /camp'ing properly
) and since i had try/catch commented out, I have some info.
Exactly 2 mins after LD timer starts, we die in a zone exception - this is likely what Jabantiz already knows, the spawn being deleted is already gone, so of course we crash. Problem is, the if(spawn) passes, so why is it crashing on the if(spawn->IsEntity())? At least that's where debugger ended up, which I think is incorrect... because why would I be in ~ZoneServer() on a zone not set to shut down for 3 more minutes? Stupid debugging.
Unhandled exception at 0x006adba6 in EQ2World__Debug.exe: 0xC0000005: Access violation reading location 0xfeeeff0e.
In ZoneServer::~ZoneServer() line 185 is
RemoveLocationProximities(). This is where debugging seems to think the crash was (DeleteData).
Code: Select all
> EQ2World__Debug.exe!ZoneServer::DeleteData(bool boot_clients) Line 265 + 0xa bytes C++
EQ2World__Debug.exe!ZoneServer::~ZoneServer() Line 185 C++
EQ2World__Debug.exe!ZoneServer::`scalar deleting destructor'() + 0x16 bytes C++
EQ2World__Debug.exe!ZoneLoop(void * tmp) Line 4357 + 0x22 bytes C++
EQ2World__Debug.exe!_callthreadstart() Line 259 + 0xf bytes C
EQ2World__Debug.exe!_threadstart(void * ptd) Line 243 C
Edit: I have lowered LD Timer (rule) to 100ms, just to see what happens next LD.
Re: EQ2TC crashing when players log off
Posted: Thu Aug 16, 2012 4:01 pm
by alfa
Always a pleasure, I have also notified a lot of lag (not due to me) especially when I was fighting (casting spell)
Re: EQ2TC crashing when players log off
Posted: Fri Aug 17, 2012 4:07 pm
by John Adams
The world is running in DEBUG with extensive logging enabled to try and catch exactly where issues arise. It is also hosted on a Windows 2003 x86 machine, since no one plays on it anymore but you or devs. If there were 10+ players regularly, I would host it on x64 and faster processors.
But the lag you are experiencing is either DEBUG (running from VS2010) or the combat changes and something not threading properly. We'll look into duplicating your issues.
Can you describe exactly what is happening? You swing, but do not hit for 5s? Need specifics so I can compare to logging.
Re: EQ2TC crashing when players log off
Posted: Fri Aug 17, 2012 4:17 pm
by alfa
First, hotbar icon color is long to change (when you can cast or you are too far away)
If you just cast one spell it seem to be ok, but if you start to chain spell, it seem to getting worse. Alot of combat anim start to miss... Ping is OK, connection too
Re: EQ2TC crashing when players log off
Posted: Sat Aug 18, 2012 10:10 am
by John Adams
alfa wrote:Ping is OK, connection too
Yeah, then this is a code thing, not a network thing probably related to DEBUG + Logging. Hopefully we'll figure out the remaining problems and run a fast server again soon. If more players were interested, it would be a no-brainer
Re: EQ2TC crashing when players log off
Posted: Wed Aug 29, 2012 6:03 pm
by Jabantiz
As stated
here this may have been fixed with some combat/zone server changes I made. I ran the tests I was always able to reproduce this with and had no crash (did the test multiple times and tried other ways). Hopefully this new change will start pushing us back in the right direction and we will never see this crash again.