Page 11 of 24
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 6:54 pm
by John Adams
DOH! 13 connections, and we finally bit it. Sorry no debug, but this was the last thing I saw in the console:
Code: Select all
[Error] Error in GetCharacterName query 'SELECT name from characters where id=62
9':
0: 07 00 2D 00 62 61 68 20 - 77 6F 6E 74 20 6C 65 74 | ..-.bah wont let
16: 20 6D 65 20 6D 61 6B 65 - 20 61 6E 79 6D 6F 72 65 | me make anymore
32: 2C 20 6D 61 78 20 70 65 - 72 20 49 50 20 69 73 20 | , max per IP is
48: 33 | 3
OP_SatMsg Received 0x00ab
0: 07 00 3E 00 62 79 20 74 - 68 65 20 77 61 79 2C 20 | ..>.by the way,
16: 49 20 73 74 61 72 74 65 - 64 20 61 20 66 72 69 67 | I started a frig
32: 67 69 6E 27 20 6C 65 67 - 65 6E 64 61 72 79 20 74 | gin' legendary t
48: 68 72 65 61 64 20 6F 6E - 20 65 71 32 66 6C 61 6D | hread on eq2flam
64: 65 73 | es
Left Grid 4294967295 and Entered Grid 924281492
I don't think that matters though (the sql error) as I just tonight cleaned out dead characters so it was probably one of those.
btw, for those missing the fun - check out these stats just a bit ago:
tess1.jpg
tess2.jpg
Thank you all for hanging in there and giving this a good kick in the pants.
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 6:56 pm
by LethalEncounter
heh kk, I'm actually surprised at how long it stayed up with that many people on. We are getting there, slowly...
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 7:13 pm
by John Adams
If it crashes again with this many people, I'll switch to debug and see if I can catch it. But yeah, this is quite amazing. 13 clients is Tess's record breaker so far. Can't wait to see 25 connections! Next milestone hehe.
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 7:18 pm
by LethalEncounter
Yah debug will also attempt to determine the lock that caused a deadlock. It will print it to the screen if it thinks it might have occurred.
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 7:38 pm
by John Adams
Entering another hour with no issues, except for a few that dropped due to the desynch issue. We're down to 6 connects, but wayyyyy better than the last 2 weeks.
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 10:01 pm
by John Adams
Just had a crash, so I am running Debug for the night. But we went nearly 3 hours without an incident.
On starting debug mode and people logging back in, I saw this:
Code: Select all
Possible deadlock attempt by 'ZoneServer::MSpawns'!
How much data do you need regarding these?
Update: Oh great, Scatman is back online and we're crashing every 20 seconds. ~evil grin~
Code: Select all
ntdll.dll!7c84afb2()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
ntdll.dll!7c83d281()
> EQ2WorldDebug.exe!Mutex::lock() Line 139 C++
EQ2WorldDebug.exe!EQStream::SetState(EQStreamState state=CLOSING) Line 258 + 0x1a bytes C++
EQ2WorldDebug.exe!ZoneServer::ClientProcess() Line 1278 C++
EQ2WorldDebug.exe!ZoneServer::Process() Line 709 C++
EQ2WorldDebug.exe!ZoneLoop(void * tmp=0x04d07fc0) Line 2723 + 0x8 bytes C++
EQ2WorldDebug.exe!_callthreadstart() Line 293 + 0xf bytes C
EQ2WorldDebug.exe!_threadstart(void * ptd=0x04d1b5f8) Line 277 C
kernel32.dll!77e64829()
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 10:02 pm
by Scatman
Gah, all I did was one /spawn remove 1.
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 10:19 pm
by LethalEncounter
If you are still up I'll check in two crash bugs that I just found.
Re: Crashes / Disconnects
Posted: Mon Jan 19, 2009 10:25 pm
by John Adams
I never sleep. It's up. =)
Shouldn't you be sleeping? heh
Re: Crashes / Disconnects
Posted: Tue Jan 20, 2009 1:27 am
by John Adams
Good Morning, LethalEncounter! Here is your breakfast crash.
Code: Select all
> EQ2WorldDebug.exe!std::operator<<<char,std::char_traits<char>,std::allocator<char> >(std::basic_ostream<char,std::char_traits<char> > & _Ostr={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & _Str=<Bad Ptr>) Line 569 + 0x2e bytes C++
EQ2WorldDebug.exe!Mutex::lock() Line 112 + 0x21 bytes C++
EQ2WorldDebug.exe!EQStream::GetState() Line 257 + 0x1a bytes C++
EQ2WorldDebug.exe!EQStream::CheckActive() Line 270 + 0x14 bytes C++
EQ2WorldDebug.exe!Client::QueuePacket(EQ2Packet * app=0x11c1c9a8) Line 165 + 0xe bytes C++
EQ2WorldDebug.exe!ZoneServer::SendSpawnChanges(Spawn * spawn=0x0a015158, Client * client=0x10c22798, bool override_changes=false) Line 793 C++
EQ2WorldDebug.exe!ZoneServer::SendSpawnChanges(Spawn * spawn=0x0a015158, bool lock=false) Line 805 C++
EQ2WorldDebug.exe!ZoneServer::SendSpawnChanges() Line 849 C++
EQ2WorldDebug.exe!ZoneServer::SpawnProcess() Line 661 C++
EQ2WorldDebug.exe!SpawnLoop(void * tmp=0x09bb4028) Line 2739 + 0x8 bytes C++
EQ2WorldDebug.exe!_callthreadstart() Line 293 + 0xf bytes C
EQ2WorldDebug.exe!_threadstart(void * ptd=0x09bb0d60) Line 277 C
kernel32.dll!77e64829()
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
A few of the last Console lines from Debug:
Code: Select all
Possible deadlock attempt by 'ZoneServer::MSpawns'!
*** Duplicate packet: Expecting Seq=16, but got Seq=15
*** Duplicate packet: Expecting Seq=16, but got Seq=15
0: D4 00 1B 00 50 72 69 73 - 74 69 6E 65 20 46 6F 72 | ....Pristine For
16: 67 65 64 20 45 62 6F 6E - 20 44 65 76 6F 75 74 | ged Ebon Devout
OP_Unknown Received 0x00b2
0: 00 00 | ..
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: 25 00 01 00 30 | %...0
0: 23 00 01 00 31 | #...1
0: D4 00 14 00 50 72 69 73 - 74 69 6E 65 20 46 6F 72 | ....Pristine For
16: 67 65 64 20 45 62 6F 6E | ged Ebon
Left Grid 4294967295 and Entered Grid 1032252765
Left Grid 4294967295 and Entered Grid 1032252765
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
*** Duplicate packet: Expecting Seq=2271, but got Seq=2270
*** Duplicate packet: Expecting Seq=2271, but got Seq=2270
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
Possible deadlock attempt by 'ZoneServer::MSpawns'!
Left Grid 4294967295 and Entered Grid 924281492
0: 29 00 00 00 | )...
0: 29 00 03 00 61 6C 6C | )...all
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
Possible deadlock attempt by 'ZoneServer::MSpawns'!
0: 18 00 17 00 75 6E 65 71 - 75 69 70 20 34 20 35 35 | ....unequip 4 55
16: 37 30 30 20 31 35 30 20 - 30 20 30 | 700 150 0 0
0: 29 00 03 00 61 6C 6C | )...all
OP_BeginTrackingMsg Received 0x0175
0: FD | .
Removing connection
AddAuth: 1113 Key: 1232439786
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
Possible deadlock attempt by 'ZoneServer::MSpawns'!
0: 13 00 00 00 | ....
303682 New client from ip: 75.72.198.7 port: 58584
ZoneAuth: Access Key, 1232439786, Character Name, Klodi, Account ID, 1113, TimeS
tamp, 1232439786
Loading new Zone 'OutpostOverlord'
Loaded 1952 NPC(s), 164 Object(s), 1 Widget(s), 41 Sign(s), 9 GroundSpawn(s).
OP_Unknown Received 0x0026
0: 01 00 00 00 C8 43 | .....C
Left Grid 1032252765 and Entered Grid 2164419443
Left Grid 2164419443 and Entered Grid 1032252765
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
OP_DialogCloseMsg Received 0x0067
0: 6F 72 14 00 | or..
Unhandled command in OP_EntityVerbsVerbMsg: repair
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
Possible deadlock attempt by 'Possible deadlock attempt by 'ZoneServer::MSpawns
Re: Crashes / Disconnects
Posted: Tue Jan 20, 2009 1:35 am
by John Adams
And...
Code: Select all
> EQ2WorldDebug.exe!Client::GetVersion() Line 123 + 0xf bytes C++
EQ2WorldDebug.exe!Client::SimpleMessage(unsigned char color='', const char * message=0x00b29af8) Line 1475 + 0x8 bytes C++
EQ2WorldDebug.exe!SpellProcess::ProcessSpell(ZoneServer * zone=0x0f4ad038, Spell * spell=0x059735b0, Entity * caster=0x1ab896d8, Spawn * target=0x00000000, bool lock=true) Line 502 C++
EQ2WorldDebug.exe!SpellProcess::Process() Line 179 C++
EQ2WorldDebug.exe!main(int argc=1, char * * argv=0x00352c88) Line 262 C++
EQ2WorldDebug.exe!__tmainCRTStartup() Line 318 + 0x19 bytes C
EQ2WorldDebug.exe!mainCRTStartup() Line 187 C
kernel32.dll!77e6f23b()
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
EQ2WorldDebug.exe!std::_Tree<std::_Tmap_traits<unsigned int,std::vector<GroundSpawnItemEntry *,std::allocator<GroundSpawnItemEntry *> >,std::less<unsigned int>,std::allocator<std::pair<unsigned int const ,std::vector<GroundSpawnItemEntry *,std::allocator<GroundSpawnItemEntry *> > > >,0> >::_Rrotate(std::_Tree_nod<std::_Tmap_traits<unsigned int,std::vector<GroundSpawnItemEntry *,std::allocator<GroundSpawnItemEntry *> >,std::less<unsigned int>,std::allocator<std::pair<unsigned int const ,std::vector<GroundSpawnItemEntry *,std::allocator<GroundSpawnItemEntry *> > > >,0> >::_Node * _Wherenode=0x006e0061) Line 1251 + 0x9 bytes C++
EQ2WorldDebug.exe!std::_Tree<std::_Tmap_traits<DataStruct *,void *,std::less<DataStruct *>,std::allocator<std::pair<DataStruct * const,void *> >,0> >::_Insert(bool _Addleft=true, std::_Tree_nod<std::_Tmap_traits<DataStruct *,void *,std::less<DataStruct *>,std::allocator<std::pair<DataStruct * const,void *> >,0> >::_Node * _Wherenode=0x00730065, const std::pair<DataStruct * const,void *> & _Val={...}) Line 1158 + 0x9 bytes C++
EQ2WorldDebug.exe!std::_Tree<std::_Tmap_traits<DataStruct *,void *,std::less<DataStruct *>,std::allocator<std::pair<DataStruct * const,void *> >,0> >::_Insert(bool _Addleft=true, std::_Tree_nod<std::_Tmap_traits<DataStruct *,void *,std::less<DataStruct *>,std::allocator<std::pair<DataStruct * const,void *> >,0> >::_Node * _Wherenode=0x006e006f, const std::pair<DataStruct * const,void *> & _Val=(0x8b001540 {is_set=??? if_not_set=??? if_set=??? ...},0x5e5ffc45)) Line 1158 + 0x9 bytes C++
EQ2WorldDebug.exe!std::_Tree<std::_Tmap_traits<DataStruct *,void *,std::less<DataStruct *>,std::allocator<std::pair<DataStruct * const,void *> >,0> >::_Insert(bool _Addleft=true, std::_Tree_nod<std::_Tmap_traits<DataStruct *,void *,std::less<DataStruct *>,std::allocator<std::pair<DataStruct * const,void *> >,0> >::_Node * _Wherenode=0x00730065, const std::pair<DataStruct * const,void *> & _Val={...}) Line 1160 + 0x9 bytes C++
EQ2WorldDebug.exe!std::_Tree<std::_Tmap_traits<DataStruct *,void *,std::less<DataStruct *>,std::allocator<std::pair<DataStruct * const,void *> >,0> >::_Insert(bool _Addleft=true, std::_Tree_nod<std::_Tmap_traits<DataStruct *,void *,std::less<DataStruct *>,std::allocator<std::pair<DataStruct * const,void *> >,0> >::_Node * _Wherenode=0x02026678, const std::pair<DataStruct * const,void *> & _Val=(0x00000000 {is_set=??? if_not_set=??? if_set=??? ...},0x00000000)) Line 1158 + 0x9 bytes C++
02026675()
Code: Select all
Unhandled command: clearallqueuedabilities
0: 25 00 01 00 30 | %...0
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: 22 00 05 00 33 30 30 30 - 30 | "...30000
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
Possible deadlock attempt by 'ZoneServer::MSpawns'!
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: 22 00 05 00 33 30 30 30 - 30 | "...30000
0: 22 00 05 00 33 30 30 30 - 30 | "...30000
0: 22 00 05 00 33 30 30 30 - 30 | "...30000
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: 22 00 05 00 33 30 30 30 - 30 | "...30000
0: 22 00 05 00 31 30 30 30 - 30 | "...10000
0: 25 00 01 00 31 | %...1
0: 25 00 01 00 31 | %...1
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: 25 00 01 00 30 | %...0
0: 38 00 04 00 4A 6F 68 6E | 8...John
AddAuth: 4049 Key: 1232440360
OP_ReadyToZone: Player Isen zoning to BigBend
477828 New client from ip: 87.154.160.234 port: 1389
ZoneAuth: Access Key, 12324403600, Character Name, : IsenCE , Account ID, 01 404
900 , TimeStamp, 00 1232440360
OP_Unknown | ....Received 0x
0026Unhandled command:
clearallqueuedabilities0
: 01 00 9A 99 54 44 | ....TD
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: 25 00 01 00 30 | %...0
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
AddAuth: 4059 Key: 1232440366
0: CE 01 00 00 | ....
Unhandled command: clearallqueuedabilities
0: 25 00 01 00 30 | %...0
485829 New client from ip: 87.165.221.237 port: 64471
Got signal 11
Re: Crashes / Disconnects
Posted: Wed Jan 21, 2009 2:33 pm
by John Adams
Update: Minimal crashes the last 2 days, with a lot of item and creature summoning, multiple logins, and locking the /reload commands from Scatman's grubby little fingers.
However, the server sits idle now in debug due to some excessive crashes I saw occurring when the server has always_loaded and city_zone configured on 25 zones, and someone logs off. Currently the console is spamming every few minutes:
Possible deadlock attempt by 'ZoneServer::MSpawnScriptTimers'!
Possible deadlock attempt by 'ZoneServer::MSpawnScriptTimers'!
No one is online as I said, but the NPCs in those active, populated zones are going about their lives. Maybe they are causing possible deadlocks.
As for Crashing - we're not done with this yet, just passed the worst part. Thanks again LE. Got a few more to go! (soon as I catch em!)
Re: Crashes / Disconnects
Posted: Wed Jan 21, 2009 3:32 pm
by LethalEncounter
Could you try the version I am about to upload? I think it fixes the issue.
Re: Crashes / Disconnects
Posted: Wed Jan 21, 2009 3:54 pm
by John Adams
Here's a crash, and yes I will grab the new code next.
Code: Select all
> EQ2WorldDebug.exe!ZoneServer::CheckEnemyList(Spawn * spawn=0x047f8798) Line 455 + 0x8 bytes C++
EQ2WorldDebug.exe!ZoneServer::ProcessMovement() Line 1615 + 0x12 bytes C++
EQ2WorldDebug.exe!ZoneServer::SpawnProcess() Line 676 C++
EQ2WorldDebug.exe!SpawnLoop(void * tmp=0x04d4fd38) Line 2761 + 0x8 bytes C++
EQ2WorldDebug.exe!_callthreadstart() Line 293 + 0xf bytes C
EQ2WorldDebug.exe!_threadstart(void * ptd=0x050ad280) Line 277 C
kernel32.dll!77e64829()
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
From the console, it only appears players were chatting. Didn't see any commands.
Re: Crashes / Disconnects
Posted: Wed Jan 21, 2009 4:00 pm
by Scatman
I did a /repop. Jack and I were trying to figure out something wrong with a groundspawn and we ran out of them so i /repoped. Right after was the crash.