Another crash

[John Adams]

This came out of nowhere. We were in playing with /useability to test spell effects, and suddenly, crash.

Code: Select all

   0: C0 01 00 00                                        | ....
[20090903 22:17:57] [Status] Unhandled command: clearallqueuedabilities

Program received signal SIGILL, Illegal instruction.
[Switching to Thread 0xb1bfeb90 (LWP 28825)]
0x080f4e72 in Client::HandlePacket (this=0x1071da80, app=0x10010f40)
    at client.cpp:1004
1004    client.cpp: No such file or directory.
        in client.cpp
(gdb)
(gdb)
(gdb) bt
#0  0x080f4e72 in Client::HandlePacket (this=0x1071da80, app=0x10010f40)
    at client.cpp:1004
#1  0x080f8537 in Client::Process (this=0x1071da80, zone_process=true)
    at client.cpp:1438
#2  0x0817c148 in ZoneServer::ClientProcess (this=0xe8bcca8)
    at zoneserver.cpp:1854
#3  0x081867c0 in ZoneServer::Process (this=0xe8bcca8) at zoneserver.cpp:868
#4  0x08186b76 in ZoneLoop (tmp=0xe8bcca8) at zoneserver.cpp:3742
#5  0x0042a51f in start_thread (arg=0xb1bfeb90) at pthread_create.c:297
#6  0x0038b04e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

[Donnarien]

And now there's another!

And the blame is mine, i guess..

I can tell what happens in the game that might be the cause to the server crash.

After hours of playing(but i don't think the hours matter), i began running trough the zone "The Cave", activating
alot of "mobs".
They all follow me and take me out.

I respawn.. (and here is now the problem) The mobs are still in the same spot as i died in + a few others scattered around the map/zone(in the path i have been running).
Now.. some of them won't react anymore, some of them will.
I relogged(in the same time cleaning my "cache" + eu setting files), they still act like statues, until you run right in to the "heap", then some woke up, some don't.. they killed me.. and server crashed.
I assume this somehow "overflows" some sort of "stack" or something(guessing wild here).

Well, hope the info is any good.
If you John just terminated the server, you can erase this message.

[John Adams]

Nah, it's good to know what's happening. I too had a problem where i aggro'd a bunch of mobs, ran for a zone (testing the mighty Train on Scatman), and when I zoned out, I was saved. But, when I zoned back in, the NPCs were still standing there. They did nothing until I moved, then they attacked me again. I stood there for probably 2-3 minutes, too. KOS mobs, 5' from me at the zone line.

I have recompiled the Tess world with some changes to the AllowAttack function (they are not correct changes, but trying to determine which part of the if() is causing the crash). I will leave it up you to crash one more time if you will.

[John Adams]

Bah, my hack still throws the crash. I'll try something else.

[John Adams]

I relogged(in the same time cleaning my "cache" + eu setting files), they still act like statues, until you run right in to the "heap", then some woke up, some don't.. they killed me.. and server crashed.

Here's the problem with this tho... when the world crashes, all spawns are of course returned to normal... yet now whenever your toon logs in, the world crashes almost immediately. Can you tell me what's going on? Do you see anything or just never get connected?

[Donnarien]

It hangs at "Created 510 entities"... that's about it i'm afraid.

[John Adams]

Ok, so the problem might be where your toon was "parked". Maybe you are getting aggro before your client finishes loading. We had a similar problem when proximity was introduced, people who camped near a spawn that spoke to you when you came within 'x' feet of it...

I have commented out that entire check, so one more test if you're still up to it. Then I'll move your toon to a safe loc completely.

[John Adams]

Yup, we're dead likely because you logged out where you get aggro. I cannot think of any other reason why. I will move your toon to a safe loc. That entire code was commented out, so now the very next line is throwing a crash. Not that this helps LE any, but here's the new call stack:

Code: Select all

>	EQ2WorldDebug.exe!Spawn::GetHP()  Line 580 + 0x3 bytes	C++
 	EQ2WorldDebug.exe!Combat::AttackAllowed(Entity * attacker=0x0641ca70, Spawn * victim=0xcdcdcdcd, bool calculate_distance=false, bool range_attack=false)  Line 429 + 0x8 bytes	C++
 	EQ2WorldDebug.exe!Combat::Process()  Line 86 + 0x29 bytes	C++
 	EQ2WorldDebug.exe!ZoneServer::Process()  Line 870	C++
 	EQ2WorldDebug.exe!ZoneLoop(void * tmp=0x067f7008)  Line 3742 + 0x8 bytes	C++
 	EQ2WorldDebug.exe!_callthreadstart()  Line 293 + 0xf bytes	C
 	EQ2WorldDebug.exe!_threadstart(void * ptd=0x06808ae0)  Line 277	C
 	kernel32.dll!77e64829() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

That was with

Code: Select all

	if((attacker->IsPlayer() && victim->appearance.attackable == 0) || (attacker->IsPlayer() && victim->IsPlayer()))
		return false;

commented out. Putting it back and moving the player.

[Donnarien]

Oki, understood!!!

[Donnarien]

Yepp, that did the "trick".

[Scatman]

Got this. All I was doing was /spawn details and /spawn and walking around, nothing else.

Code: Select all

Program received signal SIGILL, Illegal instruction.
[Switching to Thread 0xb3bdeb90 (LWP 30367)]
0x080f5612 in Client::HandlePacket (this=0xa31de50, app=0xb1cd2358) at client.cpp:1006
1006    client.cpp: No such file or directory.
        in client.cpp
(gdb) bt
#0  0x080f5612 in Client::HandlePacket (this=0xa31de50, app=0xb1cd2358) at client.cpp:1006
#1  0x080f8cd7 in Client::Process (this=0xa31de50, zone_process=true) at client.cpp:1440
#2  0x0817dc38 in ZoneServer::ClientProcess (this=0xdf5bcd0) at zoneserver.cpp:1854
#3  0x081882b0 in ZoneServer::Process (this=0xdf5bcd0) at zoneserver.cpp:868
#4  0x08188666 in ZoneLoop (tmp=0xdf5bcd0) at zoneserver.cpp:3742
#5  0x0042a51f in start_thread (arg=0xb3bdeb90) at pthread_create.c:297
#6  0x0038b04e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Edit:
3rd time within 10 minutes this exact crash has occurred by using /spawn details and /spawn

Edit 2: (JA Style Color Highlighting)
The problem was NOT /spawn details and /spawn. It was clicking a spawn (which was supposed to be a groundspawn) that had a spawn_npcs record. After I converted these spawns to groundspawns, the crashing stopped.

[John Adams]

Got one from Tess, too.

Code: Select all

>	EQ2WorldDebug.exe!SpellProcess::DeleteCasterSpell(Spawn * target=0x1db3e0e0, LuaSpell * spell=0x1c7bf008, bool lock=false)  Line 280 + 0x13 bytes	C++
 	EQ2WorldDebug.exe!SpellProcess::Process()  Line 103	C++
 	EQ2WorldDebug.exe!ZoneServer::Process()  Line 873	C++
 	EQ2WorldDebug.exe!ZoneLoop(void * tmp=0x19933308)  Line 3742 + 0x8 bytes	C++
 	EQ2WorldDebug.exe!_callthreadstart()  Line 293 + 0xf bytes	C
 	EQ2WorldDebug.exe!_threadstart(void * ptd=0x1da8a228)  Line 277	C
 	kernel32.dll!77e64829() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

Logs do not show anything important.

[John Adams]

This just in:

Code: Select all

>	EQ2WorldDebug.exe!_output_l(_iobuf * stream=0x00ed6d18, const char * format=0x00d17a7c, localeinfo_struct * plocinfo=0x00000000, char * argptr=0x21c6f954)  Line 1646 + 0x1f bytes	C++
 	EQ2WorldDebug.exe!vfprintf_helper(int (_iobuf *, const char *, localeinfo_struct *, char *)* outfn=0x006cd0bb, _iobuf * str=0x00ed6d18, const char * format=0x00d17a5c, localeinfo_struct * plocinfo=0x00000000, char * ap=0x21c6f950)  Line 69 + 0x13 bytes	C
 	EQ2WorldDebug.exe!vfprintf(_iobuf * str=0x00ed6d18, const char * format=0x00d17a5c, char * ap=0x21c6f950)  Line 117 + 0x18 bytes	C
 	EQ2WorldDebug.exe!EQEMuLog::write(EQEMuLog::LogIDs id=Error, const char * fmt=0x00d17a5c, ...)  Line 139 + 0x1b bytes	C++
 	EQ2WorldDebug.exe!Mutex::lock()  Line 115 + 0x1e bytes	C++
 	EQ2WorldDebug.exe!EQStream::EQ2_Compress(EQ2Packet * app=0x1eff6820, unsigned char offset='')  Line 436	C++
 	EQ2WorldDebug.exe!EQStream::PreparePacket(EQ2Packet * app=0x1eff6820, unsigned char offset=0)  Line 533 + 0xe bytes	C++
 	EQ2WorldDebug.exe!EQStream::EQ2QueuePacket(EQ2Packet * app=0x1eff6820, bool attempted_combine=false)  Line 508	C++
 	EQ2WorldDebug.exe!Client::QueuePacket(EQ2Packet * app=0x1eff6820)  Line 185	C++
 	EQ2WorldDebug.exe!Client::PopulateSkillMap()  Line 193	C++
 	EQ2WorldDebug.exe!Client::SendLoginInfo()  Line 204	C++
 	EQ2WorldDebug.exe!Client::Process(bool zone_process=true)  Line 1421	C++
 	EQ2WorldDebug.exe!ZoneServer::ClientProcess()  Line 1854 + 0x18 bytes	C++
 	EQ2WorldDebug.exe!ZoneServer::Process()  Line 869	C++
 	EQ2WorldDebug.exe!ZoneLoop(void * tmp=0x1d168028)  Line 3742 + 0x8 bytes	C++
 	EQ2WorldDebug.exe!_callthreadstart()  Line 293 + 0xf bytes	C
 	EQ2WorldDebug.exe!_threadstart(void * ptd=0x1da60028)  Line 277	C
 	kernel32.dll!77e64829() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

[John Adams]

Strangely, these appear to be happening the moment I sit down to check on the server.

Code: Select all

>	EQ2WorldDebug.exe!std::_Debug_message(const wchar_t * message=0x00c9b070, const wchar_t * file=0x00c9ab30, unsigned int line=986)  Line 23	C++
 	EQ2WorldDebug.exe!std::vector<LuaSpell *,std::allocator<LuaSpell *> >::erase(std::_Vector_iterator<LuaSpell *,std::allocator<LuaSpell *> > _Where=0x1857a508 {caster=0x00000000 target=0x00000000 spell=0x000b0003 ...})  Line 986 + 0x14 bytes	C++
 	EQ2WorldDebug.exe!SpellProcess::DeleteCasterSpell(Spawn * target=0x0f97bf00, LuaSpell * spell=0x1857a508, bool lock=false)  Line 292 + 0x29 bytes	C++
 	EQ2WorldDebug.exe!SpellProcess::DeleteCasterSpell(Spawn * target=0x0f97bf00, Spell * spell=0x06d5d9b0, bool lock=false)  Line 262 + 0x15 bytes	C++
 	EQ2WorldDebug.exe!SpellProcess::RemoveSpellTimersFromSpawn(Spawn * spawn=0x0f97bf00, bool remove_all=true)  Line 730	C++
 	EQ2WorldDebug.exe!ZoneServer::RemoveSpellTimersFromSpawn(Spawn * spawn=0x0f97bf00, bool remove_all=true)  Line 3498	C++
 	EQ2WorldDebug.exe!ZoneServer::RemoveSpawnSupportFunctions(Spawn * spawn=0x0f97bf00)  Line 3531	C++
 	EQ2WorldDebug.exe!ZoneServer::KillSpawn(Spawn * dead=0x0f97bf00, Spawn * killer=0x0f9a7878, bool send_packet=true, unsigned char damage_type=0, unsigned short kill_blow_type=0)  Line 2904	C++
 	EQ2WorldDebug.exe!Combat::KillSpawn(Spawn * dead=0x0f97bf00, Spawn * killer=0x0f9a7878, unsigned char damage_type=0, unsigned short kill_blow_type=0)  Line 134	C++
 	EQ2WorldDebug.exe!Combat::DamageSpawn(Entity * attacker=0x0f9a7878, Spawn * victim=0x0f97bf00, unsigned char type='Á', unsigned char damage_type=0, unsigned int low_damage=35, unsigned int high_damage=35, char * spell_name=0x06d46be8)  Line 164	C++
 	EQ2WorldDebug.exe!Combat::SpellAttack(Entity * attacker=0x0f9a7878, Spawn * victim=0x0f97bf00, float distance=-3.7344861, Spell * spell=0x06d5e218, unsigned char damage_type=0, unsigned int low_damage=35, unsigned int high_damage=0)  Line 312	C++
 	EQ2WorldDebug.exe!EQ2Emu_lua_SpellDamage(lua_State * state=0x06192438)  Line 833 + 0x40 bytes	C++
 	EQ2WorldDebug.exe!_luaD_precall()  + 0x19d bytes	C
 	EQ2WorldDebug.exe!_luaV_execute()  + 0xad8 bytes	C
 	EQ2WorldDebug.exe!_luaD_call()  + 0x43 bytes	C
 	EQ2WorldDebug.exe!_lua_call()  + 0x65 bytes	C
 	EQ2WorldDebug.exe!_luaD_rawrunprotected()  + 0x3a bytes	C
 	EQ2WorldDebug.exe!_luaD_pcall()  + 0x3e bytes	C
 	EQ2WorldDebug.exe!_lua_pcall()  + 0x53 bytes	C
 	EQ2WorldDebug.exe!LuaInterface::CallSpellProcess(LuaSpell * spell=0x19e71280, unsigned char num_parameters='')  Line 386 + 0x15 bytes	C++
 	EQ2WorldDebug.exe!SpellProcess::ProcessSpell(LuaSpell * spell=0x19e71280, bool first_cast=true)  Line 332 + 0x1b bytes	C++
 	EQ2WorldDebug.exe!SpellProcess::CastProcessedSpell(LuaSpell * spell=0x19e71280)  Line 641 + 0xe bytes	C++
 	EQ2WorldDebug.exe!SpellProcess::Process()  Line 148	C++
 	EQ2WorldDebug.exe!ZoneServer::Process()  Line 873	C++
 	EQ2WorldDebug.exe!ZoneLoop(void * tmp=0x1530be60)  Line 3742 + 0x8 bytes	C++
 	EQ2WorldDebug.exe!_callthreadstart()  Line 293 + 0xf bytes	C
 	EQ2WorldDebug.exe!_threadstart(void * ptd=0x149e4d88)  Line 277	C
 	kernel32.dll!77e64829() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

Edit: Got this one again just now. The console showed some deadlocks

Code: Select all

[20090906 12:01:05] [Error] Error, command index of 999 was requested, but max c
ommand is 467
   0: C0 01 00 00                                        | ....
[20090906 12:01:05] [Status] Unhandled command: clearallqueuedabilities
   0: C0 01 00 00                                        | ....
[20090906 12:01:14] [Status] Unhandled command: clearallqueuedabilities
   0: C0 01 00 00                                        | ....
[20090906 12:01:18] [Status] Unhandled command: clearallqueuedabilities
[20090906 12:01:36] [Debug] Creating Opcode 0 Packet!
[20090906 12:01:45] [Debug] Creating Opcode 0 Packet!
[20090906 12:01:50] [Debug] Creating Opcode 0 Packet!
[20090906 12:01:50] [Error] Possible deadlock attempt by 'SpellProcess::MActiveS
pells'!
[20090906 12:02:22] [Error] Possible deadlock attempt by 'SpellProcess::MActiveS
pells'!
Removing connection
[20090906 12:02:53] [Error] Possible deadlock attempt by 'SpellProcess::MActiveS
pells'!
[20090906 12:03:24] [Error] Possible deadlock attempt by 'SpellProcess::MActiveS
pells'!

[John Adams]

Here's something you can add to the "golly, that was interesting" pile.

As per the norm lately, my Qwest DSL decides almost daily to assign me a new IP (which is why we're almost done with them hopefully this weekend!). This is twice now that my router going down has coincided with a devastating crash of the world.

Code: Select all

 	EQ2WorldDebug.exe!deflate(z_stream_s * strm=0x1afaaf60, int flush=2)  Line 567 + 0x1d bytes	C
>	EQ2WorldDebug.exe!EQStream::EQ2_Compress(EQ2Packet * app=0x1ad41598, unsigned char offset='')  Line 441 + 0x10 bytes	C++
 	EQ2WorldDebug.exe!EQStream::PreparePacket(EQ2Packet * app=0x1ad41598, unsigned char offset=0)  Line 533 + 0xe bytes	C++
 	EQ2WorldDebug.exe!EQStream::EQ2QueuePacket(EQ2Packet * app=0x1ad41598, bool attempted_combine=false)  Line 508	C++
 	EQ2WorldDebug.exe!Client::QueuePacket(EQ2Packet * app=0x1ad41598)  Line 185	C++
 	EQ2WorldDebug.exe!Client::PopulateSkillMap()  Line 193	C++
 	EQ2WorldDebug.exe!Client::SendLoginInfo()  Line 204	C++
 	EQ2WorldDebug.exe!Client::Process(bool zone_process=true)  Line 1421	C++
 	EQ2WorldDebug.exe!ZoneServer::ClientProcess()  Line 1854 + 0x18 bytes	C++
 	EQ2WorldDebug.exe!ZoneServer::Process()  Line 869	C++
 	EQ2WorldDebug.exe!ZoneLoop(void * tmp=0x18be3e50)  Line 3742 + 0x8 bytes	C++
 	EQ2WorldDebug.exe!_callthreadstart()  Line 293 + 0xf bytes	C
 	EQ2WorldDebug.exe!_threadstart(void * ptd=0x1ad33028)  Line 277	C
 	kernel32.dll!77e64829() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

But the console screen is what's more fun!

crrraaassshhh!.jpg