[FIXED?]World Crash - Face Target

[John Adams]

Yet another fatal crash. Get this...

Server had no one online. I enabled "always_loaded" for Frostfang (to speed up load times) and this crash occurred all on it's own after 90 minutes online.

Stack:

0973d8a8()
> EQ2World.exe!Spawn::FaceTarget(Spawn * target) Line 1882 + 0x12 bytes C++
EQ2World.exe!Spawn::MoveToLocation(Spawn * spawn, float distance, bool immediate) Line 1504 C++
EQ2World.exe!Spawn::ProcessMovement() Line 1534 C++
EQ2World.exe!ZoneServer::SpawnProcess() Line 998 C++
EQ2World.exe!SpawnLoop(void * tmp) Line 5074 + 0xa bytes C++
EQ2World.exe!_callthreadstart() Line 259 + 0x6 bytes C
EQ2World.exe!_threadstart(void * ptd) Line 241 + 0x5 bytes C
kernel32.dll!77e6481f()
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]

Code:

Code: Select all

void Spawn::FaceTarget(Spawn* target){
	if(!target)
		return;
	FaceTarget(target->GetX(), target->GetZ());
>>	if(GetHP() > 0 && target->IsPlayer() && !EngagedInCombat()){
		GetZone()->AddHeadingTimer(this);

Console:

08:51:19 D World : Starting factory Reader
08:51:19 D World : Starting factory Writer
08:51:19 I Net : World server listening on: eq2emulator.net:9200
08:51:19 I Net : World server listening on: 192.168.1.35:9200
08:51:19 I Zone : Starting static zones...
08:51:19 I Zone : Loading new Zone 'FrostfangSea'
08:51:19 D Zone : FrostfangSea: Setting up 'Normal' weather
08:51:19 D Thread : Starting console command thread...
08:51:19 I Console : Type 'help' or '?' and press enter for menu options.
08:51:19 D Thread : Starting autoinit loginserver thread...
08:51:19 I World : Connected to LoginServer: eq2emulator.net: 9100
08:51:20 D Login : Looking for Login Appearance Updates...
08:51:20 I Spawn : Loaded for zone 'FrostfangSea' (470):
1838 NPC(s), 1229 Object(s), 59 Widget(s)
81 Sign(s), 1920 Ground Spawn(s), 1828 Spawn Group(s)
1164 Spawn Group Association(s), 0 Spawn Group Chance(s)
09:06:20 D Login : Looking for Login Appearance Updates...
09:21:20 D Login : Looking for Login Appearance Updates...
09:36:20 D Login : Looking for Login Appearance Updates...
09:51:20 D Login : Looking for Login Appearance Updates...
10:06:20 D Login : Looking for Login Appearance Updates...
10:21:20 D Login : Looking for Login Appearance Updates...

This is getting bad guys. What is going on?

[Jabantiz]

Looks like a pointer went invalid after the check somehow, odd.

[John Adams]

This seems to be happening an awful lot lately... pointers going invalid. Remember a while ago, there was that 1 that we couldn't figure out, so I just hacked in a 2nd check?

These aren't being wiped out by other threads, are they? Threads are generally isolated, except these pointer thingies? Seems with all the changes lately, some very bad thing has crept in.

[John Adams]

Looked at this a little this morning, and it seems the FaceTarget() function itself hasn't changed in months. Maybe years. But what did change was me clustering the "Get/Set" functions for things relating to HP and Power in order to clearly see what I needed to duplicate for Savagery and Dissonance.

I didn't think it was possible that the order in which a function lives in the .cpp file has anything whatsoever to do with it's stability. But that's the only change I made. Reverting EQ2TC to 2122, just before my commit, and we'll see if it still crashes when idle.

[John Adams]

I let EQ2TC sit idle with Frostfang loaded, no players, for a couple hours on 2120 (before my code changes) and it still crashes.

This is easily reproducible using Dev server's spawns (and scripts likely) and a little patience. I would appreciate someone fixing this, since I cannot see what's wrong. Since this function hasn't changed in quite a while, and I normally do not pre-load zones like this, who knows how long this crash has existed. Maybe some of the new scripting or LUA functions has exposed it.

[Jabantiz]

I will get the content from the content svn and make a new db to test this out. Have never seen this issue in queen's colony but there is a lot less going on in that zone.

[John Adams]

Thanks.

Maybe some of the new scripting or LUA functions has exposed it.

I suspect it's something new, but cannot figure out how any of the additions this year could affect FaceTarget. FWIW, historically speaking, EQ2TC (when it was Tess) crashed on FaceTarget as well. So much so, I actually commented the code out at one point. But considering no one is logged in, I do not know what is facing what in a running, empty zone.

Maybe that can give you a starting point.

[Jabantiz]

Didn't get as much time to test this tonight as I thought I would but have been unable to reproduce this as of yet. Created a new db and used the content on dev svn, set frost fang to always be loaded and let it sit for 30 mins with no crash, logged on and ran around in frostfang for another 30 mins and still nothing.

I will let my server run most of the day tomorrow with frost fang loaded to see if I can reproduce this.

[John Adams]

Yeah it definitely needs to sit idle, with no client connections, longer than 30 mins. Mine was 2 hours the last time.

[Jabantiz]

After about 2 hours I finally got this crash, from what I can tell it looks like the target pointer is not valid, this come from the movement code in my case where the target was m_followTarget and that also appears to be invalid.

[Jabantiz]

From what I can tell is the follow target dies and the spawn following it holds the pointer long enough that the zone server deletes it leaving m_followTarget as a invalid pointer. Switching code to smart pointer would be a huge undertaking so I added a check that will hopefully clear the follow target before it gets deleted. Running tests on my server to see if it fixes the problem.

[Jabantiz]

I am at just over 5 hours of my server being up with frost fang loaded and no connections with no crash so hopefully this is a fix, going to let it run for another hour or two just to make sure.

[Jabantiz]

committed the code, been 6 and a half hours with no crash, figure the real test will be if it crashes with John. This will also hopefully fix this issue as well.

[John Adams]

Woot. I'll get this on EQ2TC and enable FFS to load and let you know.

Nice sleuth work there.

Now can you tell me just what a "smart pointer" is? I barely understand dumb ones. What work does that entail?

[Jabantiz]

My server made it 11 hours with no crash, don't know if I was lucky or this is a fix.

Will do my best to explain smart pointers. First I want to remind you that a pointer is just a memory address, the problem we have is the spawn pointers are created and destroyed by the zone server. We then use other pointer, like m_followTarget, to point to a spawn object so we now have 2 pointers pointing to the same spawn object, m_followTarget and the pointer controlled by the zone server. If we then delete the object in the zone server the object is destroyed and the pointer the zone server maintains is set to 0 or null, however m_followTarget will still point to the same address causing it to be an invalid pointer.

Smart pointers are a c++ 11 feature that will keep track of stuff like this so when the pointer the zone server has is deleted the object will be maintained in memory until all other pointers to this object are deleted or reassigned to a new object. To use these we would have to change all pointers owned by the zone, such as spawns, to std:shared_ptr and all other pointers that point to something owned by a different system, such as m_followTarget or just the normal target for that matter, will have to be changed to std:weak_ptr. Essentially all pointers in the code will need to be changed and why I say this will be a huge undertaking.


Here is some more info if you are interested
std:shared_ptr
std:weak_ptr