World Crash - /reload luasystem

[John Adams]

Got a weird crash message this time, "EQ2World__Debug_x64.exe has triggered a breakpoint." I have no breakpoints set, so I am assuming this is something else.

All I did was /reload luasystem after doing a few /reload spells and not getting the results I expected in my spell writing.

Stack:

ntdll.dll!0000000076e340bf() Unknown
ntdll.dll!0000000076e34736() Unknown
ntdll.dll!0000000076e35942() Unknown
ntdll.dll!0000000076e375f4() Unknown
ntdll.dll!0000000076dd1980() Unknown
ntdll.dll!0000000076dc413d() Unknown
ntdll.dll!0000000076e39bd9() Unknown
ntdll.dll!0000000076dddbc0() Unknown
ntdll.dll!0000000076dc413d() Unknown
kernel32.dll!0000000076c7300a() Unknown
> EQ2World__Debug_x64.exe!_free_base(void * pBlock) Line 50 C
EQ2World__Debug_x64.exe!_free_dbg_nolock(void * pUserData, int nBlockUse) Line 1433 C++
EQ2World__Debug_x64.exe!_free_dbg(void * pUserData, int nBlockUse) Line 1265 C++
EQ2World__Debug_x64.exe!free(void * pUserData) Line 50 C++
EQ2World__Debug_x64.exe!luaL_loadbuffer() Unknown
EQ2World__Debug_x64.exe!luaM_realloc_() Unknown
EQ2World__Debug_x64.exe!luaF_freeproto() Unknown
EQ2World__Debug_x64.exe!luaC_separateudata() Unknown
EQ2World__Debug_x64.exe!luaC_freeall() Unknown
EQ2World__Debug_x64.exe!lua_setupvalue() Unknown
EQ2World__Debug_x64.exe!LuaInterface::DestroySpawnScripts() Line 190 C++
EQ2World__Debug_x64.exe!LuaInterface::~LuaInterface() Line 113 C++
EQ2World__Debug_x64.exe!Commands::Process(unsigned int index, EQ2_16BitString * command_parms, Client * client) Line 1746 C++
EQ2World__Debug_x64.exe!Client::HandlePacket(EQApplicationPacket * app) Line 1411 C++
EQ2World__Debug_x64.exe!Client::Process(bool zone_process) Line 2172 C++
EQ2World__Debug_x64.exe!ZoneServer::ClientProcess() Line 2394 C++
EQ2World__Debug_x64.exe!ZoneServer::Process() Line 1148 C++
EQ2World__Debug_x64.exe!ZoneLoop(void * tmp) Line 5054 C++
EQ2World__Debug_x64.exe!_callthreadstart() Line 257 C
EQ2World__Debug_x64.exe!_threadstart(void * ptd) Line 239 C
kernel32.dll!0000000076c6652d() Unknown
ntdll.dll!0000000076d9c521() Unknown

[Jabantiz]

This looks like a crash when trying to unload spawns scripts

EQ2World__Debug_x64.exe!lua_setupvalue() Unknown
EQ2World__Debug_x64.exe!LuaInterface::DestroySpawnScripts() Line 190 C++
EQ2World__Debug_x64.exe!LuaInterface::~LuaInterface() Line 113 C++

Never seen this before, assume it is with all the scripts from content svn, also will doing /reload lua on its own reproduce this or only after /reload spells?

[Jabantiz]

By chance did you do a /reload spells and then /reload luasystem before the spell reload was 100% finished?

[John Adams]

No, definitely waited for /reload spells to finish before doing the /reload luasystem.

Btw, I just tried it in reverse order (luasystem then spells, after luasystem was done) Got a new stack for ya.

> EQ2World__Debug_x64.exe!std::_Tree<std::_Tmap_traits<LocationProximity * __ptr64,bool,std::less<LocationProximity * __ptr64>,std::allocator<std::pair<LocationProximity * __ptr64 const,bool> >,0> >::begin() Line 1211 C++
EQ2World__Debug_x64.exe!Mutex::readlock(const char * function, unsigned int line) Line 192 C++
EQ2World__Debug_x64.exe!LuaInterface::GetSpawnScript(const char * name, bool create_new, bool use) Line 1189 C++
EQ2World__Debug_x64.exe!LuaInterface::RunSpawnScript(std::basic_string<char,std::char_traits<char>,std::allocator<char> > script_name, const char * function_name, Spawn * npc, Spawn * spawn, const char * message) Line 1276 C++
EQ2World__Debug_x64.exe!ZoneServer::CallSpawnScript(Spawn * npc, unsigned char type, Spawn * spawn, const char * message) Line 2106 C++
EQ2World__Debug_x64.exe!ZoneServer::CheckSpawnScriptTimers() Line 3125 C++
EQ2World__Debug_x64.exe!ZoneServer::SpawnProcess() Line 986 C++
EQ2World__Debug_x64.exe!SpawnLoop(void * tmp) Line 5076 C++
EQ2World__Debug_x64.exe!_callthreadstart() Line 257 C
EQ2World__Debug_x64.exe!_threadstart(void * ptd) Line 239 C
kernel32.dll!0000000076fd652d() Unknown
ntdll.dll!000000007710c521() Unknown

[Jabantiz]

This looks like a crash when a spawn script timer expired and went to call the callback function in the lua, perhaps the callback tried to call a spell while spells were still reloading?

I assume this is in a zone that is loaded or on a server with a zone running? Can it be reproduced in an empty zone with no other zones running?

[John Adams]

I am in Frostfang, preloaded zone, casting nothing (NPCs casting? I don't think that's set up yet).

I simply land in zone, wait to be sure I am fully connected (no lag) and type /reload spells, /reload luasystem (after Spells report Done!)

Do you have the current content DB with the gajillion scripts I just created? Looks like you did, if you had the 6300+ spell scripts loading.

[Jabantiz]

Do you have the current content DB with the gajillion scripts I just created? Looks like you did, if you had the 6300+ spell scripts loading.

Yes and while unrelated is this correct?

16:52:33 I LUA: Loading Spawn Scripts...
16:52:34 D LUA: Loaded 73799 SpawnScripts
16:52:34 I LUA: Loading Zone Scripts...
16:52:34 D LUA: Loaded 1 ZoneScript

Seems high, just got the latest dump from content svn.


And not surprisingly I can not reproduce this error on my end, spent 5 mins in frost fang doing /reload spells and /reload luasystem switching the order that I do them and no crashes.

[John Adams]

Hah of course I am blessed.

And yes, the raw data has 73k spawnscripts because I popped them to callout if they are in a spawngroup, so content builders do not /spawn remove 1 them, thus breaking the grouping.

[Jabantiz]

I have not been able to get the first crash, I have however got the second crash for me it crashed in mutex.cpp at void Mutex::readlock(const char* function, int32 line)

Code: Select all

if(function)
		stack_trace[string(function)] = line;   <-- crashed here

All values are valid it just crashes trying to assign the values to that map.

I don't know why this causes a crash in this instance, the only theory I have is that the lua_interface class is used on multiple threads and as it is being used the other thread tries to delete it but that is more of a gut feeling with no proof that is what is going on.

[Jabantiz]

Found a crash related to spawn scripts when doing /reload luasystem, might be related to a couple of the crashes posted here, think I fixed it by setting the flags that we set when doing /reload spawns. Code on Dev SVN.

[John Adams]

Heh, you must be bored; you're getting to all my old bugs

The /reload thing has been buggy for years. Hopefully you have it nailed down.

[Jabantiz]

Sadly the fix didn't work, happens less often now but still happens. Seems to be something with spawn scripts after the reload that screws it up and it is not an instant crash, seems to take a few calls to spawn scripts before it crashes.

Heh, you must be bored; you're getting to all my old bugs

/reload spells was crashing on me because I had an NPC set up to buff itself for testing purposes, then I ran across the crash with /reload luasystem, kind of gone ocd on them and determined to get the reloads working properly. I just hope I figure them out before it drives me nuts.

[Jabantiz]

Think I figured out the /reload luasystem crash, was a combination of a couple things, first

Code: Select all

			safe_delete(lua_interface);
			lua_interface = new LuaInterface();
			if(lua_interface)
				lua_interface->SetSpawnScriptsReloading(true);

Somehow another thread was able to sneak stuff into the map's between the "new LuaInterface()" line and the "SetSpawnScriptsReloading" line, this would cause issues when reloading is set back to false, fixed this issue by calling "lua_interface->DestroySpells()"

The second was harder to track down

Code: Select all

if(spawn_scripts.count(name) > 0){

This line was returning a false positive causing a crash further down in the code, the problem was spawn_scripts uses a string for its key and name is a const char*, not sure why this resulted in a false positive (might have something to do with pointers changing after the reload but not sure) but fixed it by changing "name" to "string(name)"

After these changes I have not had a single crash related to /reload luasystem on its own for the about hour I have been testing it, follow the /reload luasystem with a /reload spells works great too. However if a spell is being cast when /reload luasystem is used it will crash, I am looking into that now.

[Jabantiz]

Committed my code to Dev SVN

It is far more stable then it was, fixed the issue with spells casting and /reload luasystem by having /reload luasystem reload the entire spell process (like /reload spells does) instead of just the lua scripts. I have run into another crash but it is rare and only happens after you toss a /reload spawns into the mix.

[John Adams]

Awesome. I love less crashes. Thanks Jab.