World Crash, /reload spawns

[John Adams]

Since you haven't had a nice, big, fat, ugly crash in a few weeks, here's your opportunity! I was in Timorous Deep on EQ2TC, and saw the spawns movement at speed=3 was too fast, so I changed the scripts, did a /reload spawnscripts, and when it was done, I then did a /reload spawns. Ka-BOOM!

Console:

0: 41 00 0C 00 73 70 61 77 - 6E 73 63 72 69 70 74 73 | A...spawnscripts
11:05:39 D Command: Player 'Guardian' (2), Command: spawnscripts
11:05:39 D LUA: Loaded 1848 SpawnScripts
0: 41 00 06 00 73 70 61 77 - 6E 73 | A...spawns
11:05:45 D Command: Player 'Guardian' (2), Command: spawns
11:06:11 E Zone: Error Processing SpawnLoop, shutting down zone 'TimorousDeep'...
11:06:11 D Zone: Sending login equipment appearance updates...
11:06:11 D Zone: Calling clients.Remove(client)...
11:06:11 D Guilds: Sent guild member to a client.
11:06:11 D Guilds: Guild Member logged out.
11:06:11 D Zone: Removing client 'Guardian' (159) due to some client zoning...
11:06:11 I Zone: Scheduling client 'Guardian' for removal.
11:06:11 D Player: Toggling Character OFFLINE!
11:06:11 D CClient: Client Disconnect...
11:06:11 D Zone: Starting zone shutdown timers...
11:06:11 I Zone: Initiating zone shutdown of 'TimorousDeep'
11:06:42 E Mutex: Possible deadlock attempt by 'ZoneServer::MMasterSpawnLock'!

Stack:

ntdll.dll!7c81a229()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
ntdll.dll!7c87b559()
ntdll.dll!7c854c7f()
ntdll.dll!7c87caec()
ntdll.dll!7c85567a()
ntdll.dll!7c83e448()
ntdll.dll!7c829dc9()
ntdll.dll!7c83e66d()
ntdll.dll!7c83e448()
ntdll.dll!7c829dc9()
ntdll.dll!7c806ef7()
EQ2World__Debug.exe!_unlock(int locknum) Line 375 C
EQ2World__Debug.exe!_free_dbg(void * pUserData, int nBlockUse) Line 1267 + 0xc bytes C++
EQ2World__Debug.exe!operator delete(void * pUserData) Line 54 + 0x10 bytes C++
EQ2World__Debug.exe!std::allocator<std::_Tree_nod<std::_Tmap_traits<QuestStep *,std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::less<QuestStep *>,std::allocator<std::pair<QuestStep * const,std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,0> >::_Node>::deallocate(std::_Tree_nod<std::_Tmap_traits<QuestStep *,std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::less<QuestStep *>,std::allocator<std::pair<QuestStep * const,std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,0> >::_Node * _Ptr, unsigned int __formal) Line 182 + 0x9 bytes C++
EQ2World__Debug.exe!std::_Vector_val<CollectionRewardItem *,std::allocator<CollectionRewardItem *> >::~_Vector_val<CollectionRewardItem *,std::allocator<CollectionRewardItem *> >() Line 458 C++
EQ2World__Debug.exe!std::vector<CollectionRewardItem *,std::allocator<CollectionRewardItem *> >::~vector<CollectionRewardItem *,std::allocator<CollectionRewardItem *> >() Line 706 + 0xf bytes C++
> EQ2World__Debug.exe!Spawn::~Spawn() Line 113 + 0x111 bytes C++
EQ2World__Debug.exe!Entity::~Entity() Line 60 + 0xa5 bytes C++
EQ2World__Debug.exe!NPC::~NPC() Line 112 + 0x2d bytes C++
EQ2World__Debug.exe!NPC::`scalar deleting destructor'() + 0x16 bytes C++
EQ2World__Debug.exe!ZoneServer::DeleteSpawns(bool delete_all) Line 995 + 0x22 bytes C++
EQ2World__Debug.exe!ZoneServer::~ZoneServer() Line 185 C++
EQ2World__Debug.exe!ZoneServer::`scalar deleting destructor'() + 0x16 bytes C++
EQ2World__Debug.exe!ZoneLoop(void * tmp) Line 4384 + 0x22 bytes C++
EQ2World__Debug.exe!_callthreadstart() Line 259 + 0xf bytes C
EQ2World__Debug.exe!_threadstart(void * ptd) Line 243 C
kernel32.dll!77e6482f()

Edit: And of course, trying the exact same steps, it doesn't crash this time. Hopefully you see something obvious in the stack.

[Jabantiz]

I had a random spawn deadlock last night when xinux logged off my server, when the zone went to shut down I got the dead lock. No clue how or why it happened and I was not able to reproduce it. Will try and do some more tests today but it is looking like it will be one of those hard to reproduce crashes.

[John Adams]

Ok, wasn't sure if something glared at you from the stack (since we get so few of them anymore, I was hopeful).

Might be time to start adding TRACE loggers to functions to see if we can catch where something halts? now that we have queuing, it shouldn't be an ugly mess.

(but then again, you have your printf()s )

[Jabantiz]

From what I can tell it looks like this may be an issue with debug servers. I only get a dead lock in tim deep, no crash, but all other zones work fine with /reload spawns. Tim deep also has over 3000 spawns on my server. Using a 64 release compile I can not produce this issue, only with debug compiles.