Crashes / Disconnects

[Scatman]

I was just tabbed out too googling something, no interaction with the zone from me at all.

[LethalEncounter]

OK, next time I upload changes, I'll change the defaults to not catch crashes. Just be sure to run in development mode so that we can get more information as to what is causing these.

[John Adams]

You bet, thank you. I kinda miss 18 page posts of my call stacks.

[John Adams]

LE. we got a genuine lock up scenario, repeatable every time on my server with my data ( :p ). I have been setting up merchant lists today, and doing /reload items between changes. Everything is fine there, until I perform a /reload spawns. this is where the console ends up, until the world crashes.

Code: Select all

Connected to LoginServer: eq2emulator.net:9100
Login Lock Check (MinStatus: 10):, UserStatus: 255, CharID: 750
AddAuth: 150 Key: 1235852867
279398 New client from ip: 192.168.1.1 port: 2411
ZoneAuth: Access Key, 1235852867, Character Name, John, Account ID, 150, TimeSta
mp, 1235852867
Loading new Zone 'QueensColony'
Loaded 694 NPC(s), 129 Object(s), 0 Widget(s), 38 Sign(s), 127 GroundSpawn(s).
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF DF 01 - 00 00 00 80 FF FF 0B 01  | ................
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF FF FF FF 0B 01  | ................
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF FF FF FF 0B 01  | ................
OP_Unknown Received 0x0026
   0: 01 00 00 00 61 44                                  | ....aD
OP_Unknown Received 0x00b2
   0: 01 01                                              | ..
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 21 00 0E 00 6D 65 72 63 - 68 61 6E 74 20 36 36 37  | !...merchant 667
  16: 32 35                                              | 25
   0: 21 00 0E 00 6D 65 72 63 - 68 61 6E 74 20 36 36 37  | !...merchant 667
  16: 32 35                                              | 25
   0: 59 00 00 00                                        | Y...
Left Grid 4294967295 and Entered Grid 2425072386
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 41 00 05 00 69 74 65 6D - 73                       | A...items
[Status] Reloading items..
[Status]        Loaded 281 Bag(s)
[Status]        Loaded 2072 Food(s)
[Status]        Loaded 14064 Weapon(s)
[Status]        Loaded 32761 Armor/Shield Piece(s)
[Status]        Loaded 0 Skill Item(s)
[Status]        Loaded 9891 Regular Items
[Status]        Loaded 5 Merchant List(s)
[Status]        Loaded 0 Merchant Multiplier(s)
   0: 59 00 00 00                                        | Y...
   0: CC 01 00 00                                        | ....

This was me logging in, and testing my new merchants, changing prices and reloading items.


After that, I ran over to Crestrider and looked at his merchant list, while Zcoretri signed back in.

Code: Select all

Unhandled command: clearallqueuedabilities
   0: 59 00 00 00                                        | Y...
   0: 13 00 00 00                                        | ....
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 13 00 00 00                                        | ....
   0: 59 00 00 00                                        | Y...
Left Grid 2425072386 and Entered Grid 924281492
Left Grid 924281492 and Entered Grid 819993724
Left Grid 819993724 and Entered Grid 815930599
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 59 00 00 00                                        | Y...
Left Grid 815930599 and Entered Grid 819993724
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
Login Lock Check (MinStatus: 10):, UserStatus: 100, CharID: 608
AddAuth: 126 Key: 1235853180
584812 New client from ip: 67.171.21.110 port: 1503
ZoneAuth: Access Key, 1235853180, Character Name, Sakurato, Account ID, 126, Tim
eStamp, 1235853180
OP_ChatFiltersMsg Received 0x013b
   0: 0D 00 FF FF FF FF FF 01 - 00 00 00 80 FF FF 0B 01  | ................
OP_ChatFiltersMsg Received 0x013b
   0: 0D 00 FF FF FF FF FF FF - FF FF BF EF FF FF 0B 01  | ................
OP_ChatFiltersMsg Received 0x013b
   0: 0D 00 FF FF FF FF FF FF - FF FF BF FF FF FF 0B 01  | ................
OP_Unknown Received 0x0026
   0: 01 00 00 00 48 44                                  | ....HD
OP_Unknown Received 0x00b1
   0: 00 00                                              | ..
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
OP_ChatFiltersMsg Received 0x013b
   0: 0D 00 FF FF FF FF FF FF - FF FF BF FF FF FF 0B 01  | ................
   0: 29 00 03 00 61 6C 6C                               | )...all
   0: 0C 00 24 00 68 6F 6C 64 - 20 6F 6E 20 66 6F 72 20  | ..$.hold on for
  16: 61 6E 6F 74 68 65 72 20 - 63 72 61 73 68 2E 20 6D  | another crash. m
  32: 61 79 62 65 2E 20 3B 29                            | aybe. ;)
   0: 41 00 06 00 73 70 61 77 - 6E 73                    | A...spawns
[Status] Loaded 504 entity command lists.
[Status] Loaded 3935 NPC(s).
[Status] Loaded 609 Object(s).
[Status] Loaded 58 GroundSpawn(s).
[Status] Loaded 475 Widget(s).
[Status] Loaded 576 Sign(s).
[Status] Loading ground spawn items..
[Status] Loading loot data..
[Status]        Loaded 1 loot tables.
[Status]        Loaded 1 loot drops.
[Status]        Loaded 1 spawn loot lists.
[Status] Loading Transporter Information..
[Status]        Loaded 24 Teleporter(s)
Error Processing SpawnLoop, shutting down zone...
Exception caught when in ZoneServer::ClientProcess()!
[Status] Using database 'eq2live' at mysql
[Status] CURRENT_WORLD_VERSION:EQ2EMu 0.6.5 Developer Alpha

I did a /reload spawns to change some settings on the "?" harvest nodes, and as I saw before, it gets to Loaded 24 Teleporter(s) and hangs there. This time I let it sit, and the zone crashed, followed by the World.

I am still using the exception-catch code, so no debug.

[John Adams]

If I hadn't said this before, I am starting to think this is all about groundspawns. I wasn't doing jack that time but standing there after harvesting a feather. I was in another window adding stuff to the ? spawns when I heard my client stop making tweety bird noises. Looked, World restarted. No one else was online, and no commands or movement was going on.

Going from bad to worse here! But at least we'll catch it, knowing how to reproduce it.

[LethalEncounter]

I fixed what was likely causing the problem. Whenever you did a /reload spawns (and repop) the client thread was deleting data while it was still in use. We got lucky most of the time, but it was MUCH more likely to occur whenever there were multiple zones running. This could have caused all kinds of issues, and although I'm not sure it caused ALL of the crash issues you have been experiencing, I'm pretty sure it cause quite a few of them.

[John Adams]

Excellent. I like when you DO find something in all my ramblings. Whenever you commit this and the placeholder fix, I'll wait for the next crash then set up the old remote debugger for another round.

Thank you, if I don't say it enough.

[LethalEncounter]

np It is checked in.

[John Adams]

Is the crash catcher code turned off? I just crashed zoning and got the old Error processing spawnloop, shutting down zone.

[LethalEncounter]

Are you running in debug mode? It should only be turned on in release mode (which is what my compiled exe is that I upload).

[John Adams]

Ohh, sorry no. I am in release still. I usually only run debug when I am trapping crashes, since my server is a VM and debug seem to really die on that VM. Sooo sloowwww.

But ok, since it is crashing, I'll debug and see what we come up with.

[Scatman]

That one was my fault (what else is new), don't even post it /sigh

[John Adams]

hahaha ok, if you say so.

[John Adams]

Ahh, here's one little bugger that's been pissing me off. Seems to happen when you delete items and/or move / stack items. Not sure which causes it, since I just deleted 2 vouchers, then stacked a feather in char inventory slots 4 to 2, then opened my bag and dragged a "pile of sticks" onto another pile and immediately got the crash break.

Code: Select all

>	EQ2WorldDebug.exe!strlen(unsigned char * buf=0x2f9c4b4a)  Line 69	Asm
 	EQ2WorldDebug.exe!DBcore::getSafeEscapeString(const char * from_string=0xdddddddd)  Line 312 + 0x9 bytes	C++
 	EQ2WorldDebug.exe!WorldDatabase::SaveItem(unsigned int account_id=150, unsigned int char_id=750, Item * item=0x066ce430, char * type=0x00b89bfc)  Line 516 + 0x5c bytes	C++
 	EQ2WorldDebug.exe!WorldDatabase::SaveItems(Client * client=0x0707d868)  Line 498	C++
 	EQ2WorldDebug.exe!Client::Save()  Line 1857	C++
 	EQ2WorldDebug.exe!ZoneServer::SaveClient(Client * client=0x0707d868)  Line 830	C++
 	EQ2WorldDebug.exe!ZoneServer::SaveClients()  Line 841	C++
 	EQ2WorldDebug.exe!ZoneServer::Process()  Line 788	C++
 	EQ2WorldDebug.exe!ZoneLoop(void * tmp=0x064ab178)  Line 2993 + 0x8 bytes	C++
 	EQ2WorldDebug.exe!_callthreadstart()  Line 293 + 0xf bytes	C
 	EQ2WorldDebug.exe!_threadstart(void * ptd=0x064a5ae8)  Line 277	C
 	kernel32.dll!77e64829() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

[John Adams]

Yeah this don't look right, but I got it immediately this time dragging the first feather in slot 4 to slot 2.

Code: Select all

 	ntdll.dll!7c82bc32() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
>	EQ2WorldDebug.exe!_unlock(int locknum=4128768)  Line 376	C
 	EQ2WorldDebug.exe!_free_dbg(void * pUserData=0x0fa8fa48, int nBlockUse=112290472)  Line 1199 + 0x7 bytes	C++
 	EQ2WorldDebug.exe!_free_dbg(void * pUserData=0x0fa8fa48, int nBlockUse=112290472)  Line 1196 + 0xc bytes	C++
 	00000008()	
 	ntdll.dll!7c82a4e1() 	
 	ntdll.dll!7c82a0fc() 	
 	EQ2WorldDebug.exe!std::_Container_base::_Orphan_all()  Line 200 + 0x8 bytes	C++
 	EQ2WorldDebug.exe!_Mtxunlock(_RTL_CRITICAL_SECTION * _Mtx=0x00d786b8)  Line 54	C
 	EQ2WorldDebug.exe!std::_Lockit::~_Lockit()  Line 69 + 0x14 bytes	C++