Crashes / Disconnects

[LethalEncounter]

That is strange, I can't find anything wrong with that section of code that would explain that error message. It could have been caused by another section of code and it just died there :/

[John Adams]

Here is the evilest of crashes right now. Literally every /reload spawn brings the server spewing out Got Signal 11's and the world never ends, you have to kill it.

Code: Select all

Combined Packet!
Left Grid 4294967295 and Entered Grid 1275535964
   0: 41 00 06 00 73 70 61 77 - 6E 73                    | A...spawns
[Status] Loaded 506 entity command lists.
[Status] Loaded 6420 NPC(s).
[Status] Loaded 1262 Object(s).
[Status] Loaded 92 GroundSpawn(s).
[Status] Loaded 780 Widget(s).
[Status] Loaded 1062 Sign(s).
[Status] Loading ground spawn items..
[Status] Loading loot data..
[Status]        Loaded 6 loot tables.
[Status]        Loaded 8 loot drops.
[Status]        Loaded 5 spawn loot lists.
[Status] Loading Transporter Information..
[Status]        Loaded 24 Teleporter(s)
Loaded 74 NPC(s), 90 Object(s), 19 Widget(s), 60 Sign(s), 14 GroundSpawn(s).

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1289208944 (LWP 18918)]
ZoneServer::CheckLocationProximity (this=0xcb77ee8) at zoneserver.cpp:2806
2806                                            float x = prox->x;
Missing separate debuginfos, use: debuginfo-install e2fsprogs.i386 gcc.i386 glibc.i686 keyutils.i386 krb5.i386 libselinux.i386 lua.i386 mysql.i386 openssl.i686 zlib.i386
(gdb) bt
#0  ZoneServer::CheckLocationProximity (this=0xcb77ee8) at zoneserver.cpp:2806
#1  0x08181245 in ZoneServer::SpawnProcess (this=0xcb77ee8)
    at zoneserver.cpp:726
#2  0x0818135a in SpawnLoop (tmp=0xcb77ee8) at zoneserver.cpp:3060
#3  0x00ced50b in start_thread () from /lib/libpthread.so.0
#4  0x00c2eb2e in clone () from /lib/libc.so.6
(gdb)

Well, this is one of the evilest.


This was a moment later after a fresh start, simply typing /repop

Code: Select all

   0: 52 00 00 00                                        | R...
Loaded 74 NPC(s), 90 Object(s), 19 Widget(s), 60 Sign(s), 14 GroundSpawn(s).

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1289139312 (LWP 18964)]
ZoneServer::CheckLocationProximity (this=0xc8a40c0) at zoneserver.cpp:2806
2806                                            float x = prox->x;
(gdb) bt
#0  ZoneServer::CheckLocationProximity (this=0xc8a40c0) at zoneserver.cpp:2806
#1  0x08181245 in ZoneServer::SpawnProcess (this=0xc8a40c0)
    at zoneserver.cpp:726
#2  0x0818135a in SpawnLoop (tmp=0xc8a40c0) at zoneserver.cpp:3060
#3  0x00ced50b in start_thread () from /lib/libpthread.so.0
#4  0x00c2eb2e in clone () from /lib/libc.so.6

I blame bad data. Just not sure where it is. LE you are welcome to use my Linux server any time.

[Scatman]

Fixed =|

[John Adams]

Oooh, a Scatman bug. Bravo! Thanks Scat.

[John Adams]

LE, TessEQ2 ran literally for 2 weeks straight without a crash, but a lot of failed or dropped connections. I (personally) need about 3-4 connection attempts before I succesfully land in the game, but not 100% of the time. That is, closing the locked up "Loading UI resources" client and starting with cleared caches and what-not.

The World stays up, but what I see are repeated lines of

Code: Select all

Combined Packet!
Combined Packet!
Combined Packet!
Combined Packet!
Combined Packet!

over and over, about one every second, until I kill the client. I do not see any data around it that leads me to think something else is wrong. I also notice many players are simply getting dropped while doing normal operations - usually idenfitied by this in the console:

Code: Select all

Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
OP_BeginTrackingMsg Received 0x0175
Zone: Removing client from ip:69.76.186.203 port:5998
Removing connection
Combined Packet!
Combined App Packet!

Or newly created characters that end up trying to connect and failing.

Code: Select all

Name check on: Name_01
Resp: 1
AddAuth: xxxx Key: 1241237725
10996986 New client from ip: 222.70.xxx.xxx port: 2486
Adding new client...
10997017 New client from ip: 222.70.xxx.xxx port: 2486
Combined Packet!
World: Removing client from ip:222.70.xxx.xxx port:2486
Combined Packet!

Not sure if this means anything to you, just reporting some odd behaviors.

[John Adams]

Getting a world crash (linux) using current dev build hailing one of our NPCs in Forest Ruins. Poko Zing, responds, but the console start spitting out "Got signal 11". Went into debug to catch it, and here's the backtrace.

Code: Select all

Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
Left Grid 4294967295 and Entered Grid 208740015

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb3ca2b90 (LWP 13942)]
0x080d44f6 in Client::DisplayConversation (this=0xf9335c8, npc=0xed4f8e0,
    conversations=0xb1d065c8,
    text=0xb1d063dc "What do you need and want and why did you come to me for it?", mp3=0x0, key1=0, key2=0) at client.cpp:2710
2710                            if(conversations->at(i).function.length() > 0)
Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.4-4.fc10.i386 glibc-2.9-3.i686 keyutils-libs-1.2-3.fc9.i386 krb5-libs-1.6.3-18.fc10.i386 libgcc-4.3.2-7.i386 libselinux-2.0.78-1.fc10.i386 libstdc++-4.3.2-7.i386 lua-5.1.4-1.fc10.i386 mysql-libs-5.0.77-1.fc10.i386 openssl-0.9.8g-12.fc10.i686 zlib-1.2.3-18.fc9.i386
(gdb) bt
#0  0x080d44f6 in Client::DisplayConversation (this=0xf9335c8, npc=0xed4f8e0,
    conversations=0xb1d065c8,
    text=0xb1d063dc "What do you need and want and why did you come to me for it?", mp3=0x0, key1=0, key2=0) at client.cpp:2710
#1  0x081aaa6a in EQ2Emu_lua_StartConversation (state=0xed51250)
    at LuaFunctions.cpp:315
#2  0x0090dc57 in ?? () from /usr/lib/liblua-5.1.so
#3  0x00919240 in ?? () from /usr/lib/liblua-5.1.so
#4  0x0090e218 in ?? () from /usr/lib/liblua-5.1.so
#5  0x00909061 in ?? () from /usr/lib/liblua-5.1.so
#6  0x0090d7e3 in ?? () from /usr/lib/liblua-5.1.so
#7  0x0090d848 in ?? () from /usr/lib/liblua-5.1.so
#8  0x00908e78 in lua_pcall () from /usr/lib/liblua-5.1.so
#9  0x081adb02 in LuaInterface::CallSpawnScript (this=0x8228cb8,
    state=0xed51250, num_parameters=0 '\0') at LuaInterface.cpp:396
#10 0x081b2bad in LuaInterface::RunSpawnScript (this=0x8228cb8, script_name=
        {static npos = 4294967295, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xb1d36064 "SpawnScripts/ForestRuins/PokoZing.lua"}},
    function_name=0x81e00a6 "hailed", npc=0xed4f8e0, spawn=0xc30ed18,
    message=0x0) at LuaInterface.cpp:930
#11 0x08149a44 in ZoneServer::CallSpawnScript (this=0xc3148c0, npc=0xed4f8e0,
    type=<value optimized out>, spawn=0xc30ed18, message=0x0)
---Type <return> to continue, or q <return> to quit---
    at zoneserver.cpp:1136
#12 0x080a44fb in Commands::Process (this=0x820f4dc, index=19,
    command_parms=0xb3ca211c, client=0xf9335c8) at Commands.cpp:745
#13 0x080e33e4 in Client::HandlePacket (this=0xf9335c8, app=0x116d5ce8)
    at client.cpp:941
#14 0x080e4077 in Client::Process (this=0xf9335c8, zone_process=true)
    at client.cpp:1373
#15 0x0814b4ff in ZoneServer::ClientProcess (this=0xc3148c0)
    at zoneserver.cpp:1400
#16 0x0815cf85 in ZoneServer::Process (this=0xc3148c0) at zoneserver.cpp:792
#17 0x0815d237 in ZoneLoop (tmp=0xc3148c0) at zoneserver.cpp:3056
#18 0x0042a51f in start_thread () from /lib/libpthread.so.0
#19 0x0038b04e in clone () from /lib/libc.so.6
(gdb)

Scatman fixed what was likely causing the problem. Here's the script (paraphrased).

Code: Select all

function hailed(NPC, Spawn)
	FaceTarget(NPC, Spawn)
	conversation = CreateConversation()
	
	WhatDoYouNeed(NPC, Spawn, conversation)
end


function WhatDoYouNeed(NPC, Spawn, conversation)
	PlayFlavor(NPC, "voiceover/english/tutorial_revamp/poko_zing/qey_adv02_ruins/quests/poko/poko_neezer000.mp3", "", "", 1600190149, 86494412, Spawn)
	StartConversation(conversation, "Is something wrong?", "OnlyEverything")
	StartConversation(conversation, NPC, Spawn, "What do you need and want and why did you come to me for it?")
end

Note the problem, we think, was the double StartConversation(). But this shouldn't crash the world, maybe throw a LUA error instead.

[John Adams]

Today's crash-a-thon brought to you by Scatman's Script-o-matic! Dunno what we had going on here. Chatting in OOC, a few reloads.

Crash.

Code: Select all

   0: 0C 00 05 00 2C 2C 2C 2C - 3F                       | ....,,,,?
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 36 00 00 00                                        | 6...
OP_Unknown Received 0x01d2
   0: 00 01 CC 00 00 00 01 CC - 00 00 00 00 00 00 00     | ...............
   0: CC 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 36 00 00 00                                        | 6...

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb14ffb90 (LWP 20581)]
Mutex::lock (this=0x4) at ../common/Mutex.cpp:118
118             if(name.length() > 0){
(gdb) bt
#0  Mutex::lock (this=0x4) at ../common/Mutex.cpp:118
#1  0x081beca0 in Quest::CheckQuestChatUpdate (this=0x0, id=0, update=false)
    at Quests.cpp:562
#2  0x08134425 in Player::CheckQuestFlag (this=0xc3884f8, spawn=0xc3884f8)
    at Player.cpp:2020
#3  0x0814a106 in ZoneServer::SendQuestUpdates (this=0xebca278,
    client=0x122c61d0, spawn=0xc3884f8) at zoneserver.cpp:2264
#4  0x081a88be in EQ2Emu_lua_AddQuestStepChat (state=0xbfa9bd0)
    at LuaFunctions.cpp:1535
#5  0x0090dc57 in ?? () from /usr/lib/liblua-5.1.so
#6  0x00919240 in ?? () from /usr/lib/liblua-5.1.so
#7  0x0090e218 in ?? () from /usr/lib/liblua-5.1.so
#8  0x00909061 in ?? () from /usr/lib/liblua-5.1.so
#9  0x0090d7e3 in ?? () from /usr/lib/liblua-5.1.so
#10 0x0090d848 in ?? () from /usr/lib/liblua-5.1.so
#11 0x00908e78 in lua_pcall () from /usr/lib/liblua-5.1.so
#12 0x081b1b9b in LuaInterface::CallQuestFunction (this=0x8228cb8,
    quest=0xb1d01658, function=0xbfb2434 "Step3_Complete_GotPoison",
    player=0xc3884f8, step_id=4294967295) at LuaInterface.cpp:279
#13 0x080d5198 in Client::SendQuestUpdate (this=0x122c61d0, quest=0xb1d01658)
    at client.cpp:2505
#14 0x080d56f8 in Client::CheckPlayerQuestsKillUpdate (this=0x122c61d0,
    spawn=0x14d5e468) at client.cpp:2352
---Type <return> to continue, or q <return> to quit---
#15 0x08159bd1 in ZoneServer::KillSpawn (this=0xebca278, dead=0x14d5e468,
    killer=0xc3884f8, send_packet=true) at zoneserver.cpp:2386
#16 0x080ce9e5 in Combat::KillSpawn (this=0xebca494, dead=0x14d5e468,
    killer=0xc3884f8) at Combat.cpp:126
#17 0x0809e8f6 in Commands::Process (this=0x820f4dc, index=54,
    command_parms=0xb14ff13c, client=0x122c61d0) at Commands.cpp:970
#18 0x080e382c in Client::HandlePacket (this=0x122c61d0, app=0x1260f638)
    at client.cpp:987
#19 0x080e4077 in Client::Process (this=0x122c61d0, zone_process=true)
    at client.cpp:1373
#20 0x0814b4ff in ZoneServer::ClientProcess (this=0xebca278)
    at zoneserver.cpp:1400
#21 0x0815cf85 in ZoneServer::Process (this=0xebca278) at zoneserver.cpp:792
#22 0x0815d237 in ZoneLoop (tmp=0xebca278) at zoneserver.cpp:3056
#23 0x0042a51f in start_thread () from /lib/libpthread.so.0
#24 0x0038b04e in clone () from /lib/libc.so.6

[Scatman]

Yeah the first one got fixed from removing the duplicated StartConversations on the same conversation.
The second one was caused by me using a check on the wrong quest ID

[LethalEncounter]

Yeah the first one got fixed from removing the duplicated StartConversations on the same conversation.
The second one was caused by me using a check on the wrong quest ID

John is right. I need to fix both of these as even if the even messes up it is not desirable to crash the world.

Which function was it you were calling for the second one?

[Scatman]

I was using QuestIsComplete(Player, 104). The quest ID I meant to use was 204. The weird thing is, quest 104 is a valid quest ID, but I didn't have the quest, nor did I have it completed.

[John Adams]

There might be an issue with the newest code (that being our Stats timer stuff, Scatman). TessEQ2 was running fine for days, and I just took it down, upgraded to some changes I made to Stats (added the initial defines) and I think this was the first time Tess has seen Stats - since she just pulled down the table "Statistics".

Now, I see 2 players online, 2 zones active, with no one really there (that I can see). Just this in the console:

Code: Select all

Left Grid 4294967295 and Entered Grid 1701767597
   0: 29 00 03 00 61 6C 6C                               | )...all
Error Processing SpawnLoop, shutting down zone...
Possible deadlock attempt by 'ZoneServer::MMasterSpawnLock'!
Shutting Down Zone 'TimorousDeep'
Removing connection
AddAuth: 4614 Key: 1242243340
AddAuth: 4614 Key: 1242243346
Shutting Down Zone 'oakmyst'
AddAuth: 4614 Key: 1242244232
AddAuth: 4614 Key: 1242244247
Delete char request: 4614 2073

Going to revert for now until we can validate if Stats code is locking up the World. Shouldn't, but meh... there ya go.

[John Adams]

LE, here's a sample of starting the world, logging in, testing a few merchant lists, doing a /reload spawns, then simply logging off. I hadn't noticed this til this morning, so apparently it didn't happen right away.

Code: Select all

[Status]        Loaded 2 Merchant List(s)
[Status] Loading Spawn Script Information..
[Status] Loading Zone Script Information..
[Status] Loading EQ time of day..
<Rogean>: There is only Everquest and Food, no Sleep.
Starting factory Reader
Starting factory Writer
World server listening on: tesseq.servegame.com:9300
Starting static zones...
Connected to LoginServer: eq2emulator.net:9100
Login Lock Check (MinStatus: 100):, UserStatus: 255, CharID: 1
AddAuth: 150 Key: 1242454481
24225 New client from ip: 192.168.1.1 port: 1185
Adding new client...
24229 New client from ip: 192.168.1.1 port: 1185
World: Removing client from ip:192.168.1.1 port:1185
Combined Packet!
Combined App Packet!
Removing connection
Login Lock Check (MinStatus: 100):, UserStatus: 255, CharID: 1
AddAuth: 150 Key: 1242454562
101604 New client from ip: 192.168.1.1 port: 1197
Adding new client...
Combined Packet!
Combined App Packet!

Above, you'll notice server startup, first connect was a dud, locked my client at connecting to loginserver. I had to kill it, and connect again... second time I got through fine. Log continues:

Code: Select all

ZoneAuth: Access Key, 1242454562, Character Name, Johnrat, Account ID, 150, Time                                                                                                                               Stamp, 1242454562
Loading new Zone 'QueensColony'
Loaded 687 NPC(s), 121 Object(s), 0 Widget(s), 4 Sign(s), 135 GroundSpawn(s).
OP_QuestJournalSetVisibleMsg Received 0x006c
   0: 01 00 00 00 8E 00 00 00 - 00                       | .........
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF DF 01 - 00 00 00 00 FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_Unknown Received 0x0026
   0: 01 00 00 00 61 44                                  | ....aD
Combined Packet!
Combined Packet!
OP_Unknown Received 0x00b2
   0: 01 01                                              | ..
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 59 00 00 00                                        | Y...
Left Grid 4294967295 and Entered Grid 924281492
   0: 27 00 03 00 31 30 30                               | '...100
Left Grid 924281492 and Entered Grid 819993724
Left Grid 819993724 and Entered Grid 815930599
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 59 00 00 00                                        | Y...
Left Grid 815930599 and Entered Grid 819993724
Left Grid 819993724 and Entered Grid 924281492
Left Grid 924281492 and Entered Grid 2425072386
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 27 01 00 00                                        | '...
Unhandled command: afk
   0: 27 01 00 00                                        | '...
Unhandled command: afk
   0: 41 00 05 00 69 74 65 6D - 73                       | A...items
[Status] Reloading items..
[Status]        Loaded 282 Bag(s)
[Status]        Loaded 2072 Food(s)
[Status]        Loaded 14064 Weapon(s)
[Status]        Loaded 32761 Armor/Shield Piece(s)
[Status]        Loaded 0 Skill Item(s)
[Status]        Loaded 9906 Regular Items
[Status]        Loaded 78 Merchant Inventory Item(s)
[Status]        Loaded 4 Merchant List(s)
   0: 14 00 03 00 32 37 38                               | ....278
Zone access check for OutpostOverlord
Grabbing zone requirements for OutpostOverlord
Zone Requirements: 0 status to override negative status requirement, 0 status to                                                                                                                                override level requirement.
Loading new Zone 'OutpostOverlord'
AddAuth: 150 Key: 1242455146
Loaded 1597 NPC(s), 169 Object(s), OP_ReadyToZone: Player Johnrat zoning to Outp                                                                                                                               ostOverlord
1 Widget(s), 41 Sign(s), 126 GroundSpawn(s).
681098 New client from ip: 192.168.1.1 port: 1197
Adding new client...
Combined Packet!
ZoneAuth: Access Key, 1242455146, Character Name, Johnrat, Account ID, 150, Time                                                                                                                               Stamp, 1242455146
OP_Unknown Received 0x0026
   0: 01 00 00 00 61 44                                  | ....aD
Zone: Removing client from ip:192.168.1.1 port:1197
OP_Unknown Received 0x00b2
   0: 01 01                                              | ..
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
Left Grid 4294967295 and Entered Grid 924281492
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
Combined Packet!
Shutting Down Zone 'QueensColony'
   0: 41 00 06 00 73 70 61 77 - 6E 73                    | A...spawns
[Status] Loaded 525 entity command lists.
[Status] Loaded 7183 NPC(s).
[Status] Loaded 1450 Object(s).
[Status] Loaded 98 GroundSpawn(s).
[Status] Loaded 796 Widget(s).
[Status] Loaded 1111 Sign(s).
[Status] Loading ground spawn items..
[Status] Loading loot data..
[Status]        Loaded 6 loot tables.
[Status]        Loaded 8 loot drops.
[Status]        Loaded 4 spawn loot lists.
[Status] Loading Transporter Information..
[Status]        Loaded 25 Teleporter(s)
Loaded 1597 NPC(s), 169 Object(s), 1 Widget(s), 41 Sign(s), 126 GroundSpawn(s).
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 59 00 00 00                                        | Y...
Left Grid 924281492 and Entered Grid 819993724
Left Grid 819993724 and Entered Grid 815930599
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: C3 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 59 00 00 00                                        | Y...
OP_BeginTrackingMsg Received 0x0175
Removing connection
Timeout waiting for read locks: ZoneServer::MClients
Possible deadlock attempt by 'ZoneServer::MClients'!
Zone: Removing client from ip:192.168.1.1 port:1197

This one shows me reloading items, zoning, and reloading spawns, then logging off after testing a couple of merchant lists. After logoff, I did not see the deadlock notice until this morning - so it didn't show up immediately, but with no timestamps on log entries, i cannot confirm exactly when that happened.

Server is still responsive to connections, though no right-clicking on NPCs seems to work. slash commands do not work. NPC movement loops appear to still be working.

[John Adams]

LE, we got this old bugger back as well. Not sure what caused it, as I was sleeping.

locked-up.jpg

Console looks like this (end sample):

Code: Select all

OP_BeginTrackingMsg Received 0x0175
   0: 14 00 03 00 32 37 31                               | ....271
Zone access check for VeeshansPeak
Grabbing zone requirements for VeeshansPeak
Zone Requirements: 0 status to override negative status requirement, 0 status to
 override level requirement.
Loading new Zone 'VeeshansPeak'
Loaded 0 NPC(s), 0 Object(s), 0 Widget(s), 0 Sign(s), 0 GroundSpawn(s).
AddAuth: 2571 Key: 1243067617
OP_ReadyToZone: Player Dreadral zoning to VeeshansPeak
Zone: Removing client from ip:97.82.58.161 port:53868
16933347 New client from ip: 97.82.58.161 port: 53868
Adding new client...
Combined Packet!
ZoneAuth: Access Key, 1243067617, Character Name, Dreadral, Account ID, 2571, Ti
meStamp, 1243067617
Combined Packet!
OP_Unknown Received 0x0026
   0: 01 00 00 80 22 44                                  | ...."D
OP_Unknown Received 0x00b2
   0: 00 00                                              | ..
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
Removing connection
OP_BeginTrackingMsg Received 0x0175
Removing connection
Zone: Removing client from ip:66.157.98.95 port:59373
Shutting Down Zone 'antonica'
Zone: Removing client from ip:97.82.58.161 port:53868
Shutting Down Zone 'TrialBoFen'
AddAuth: 4484 Key: 1243067787
17103476 New client from ip: 83.227.162.130 port: 1523
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243067787, Character Name, Geka, Account ID, 4484, TimeSt
amp, 1243067787
OP_QuestJournalSetVisibleMsg Received 0x006c
   0: 07 00 00 00 75 00 00 00 - 7E 00 00 00 7F 00 00 00  | ....u...~.......
  16: 81 00 00 00 80 00 00 00 - 7B 00 00 00 7A 00 00 00  | ........{...z...
  32: 00                                                 | .
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF 01 - 00 00 00 80 FF FF 0B     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF FF FF FF 0B     | ...............
OP_Unknown Received 0x0026
   0: 01 00 00 00 48 44                                  | ....HD
Combined Packet!
Combined Packet!
OP_Unknown Received 0x01d2
   0: 00 00 01 76 00 00 00 00 - 00 00 00                 | ...v.......
Combined Packet!
Combined Packet!
OP_Unknown Received 0x00b2
   0: 00 00                                              | ..
Shutting Down Zone '0VillageShin: '
C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
Left Grid 4294967295 and Entered Grid 1758985534
Shutting Down Zone 'MysticLake'
Shutting Down Zone 'Commonlands'
Shutting Down Zone 'SepulcherZanFi'
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF FF FF FF 0B     | ...............
   0: 14 00 03 00 32 30 33                               | ....203
   0: 14 00 03 00 32 30 33                               | ....203
   0: 14 00 00 00                                        | ....
Shutting Down Zone 'ForsakenCity'
Shutting Down Zone 'XuxlaiosRoost'
Shutting Down Zone 'SplitpawDen'
   0: 14 00 00 00                                        | ....
   0: 14 00 03 00 32 30 34                               | ....204
Zone access check for ScarabHive
Grabbing zone requirements for ScarabHive
Zone Requirements: 0 status to override negative status requirement, 0 status to
 override level requirement.
Loading new Zone 'ScarabHive'
Loaded 0 NPC(s), 0 Object(s), 0 Widget(s), AddAuth: 4484 Key: 1243067921
0 Sign(s), 0 GroundSpawn(s).
OP_ReadyToZone: Player Geka zoning to ScarabHive
Zone: Removing client from ip:83.227.162.130 port:1523
17238824 New client from ip: 83.227.162.130 port: 1523
Adding new client...
17238870 New client from ip: 83.227.162.130 port: 1523
Combined Packet!
Combined App Packet!
World: Removing client from ip:83.227.162.130 port:1523
Shutting Down Zone 'GukStrongholdRaid'
Removing connection
Shutting Down Zone 'qey_guildhall_tier4'
Shutting Down Zone 'VeeshansPeak'
Shutting Down Zone 'ScarabHive'
Shutting Down Zone 'CryptDecrepit'
Name check on: Alliz
Resp: 1
AddAuth: 4660 Key: 1243076387
25704212 New client from ip: 91.77.202.189 port: 1271
Adding new client...
25704259 New client from ip: 91.77.202.189 port: 1271
Combined Packet!
Combined App Packet!
World: Removing client from ip:91.77.202.189 port:1271
Removing connection
Name check on: Archi
Resp: 1
AddAuth: 4660 Key: 1243076608
25925108 New client from ip: 91.77.202.189 port: 1314
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243076608, Character Name, Archi, Account ID, 4660, TimeS
tamp, 1243076608
Loading new Zone 'QueensColony'
Loaded 694 NPC(s), 129 Object(s), 0 Widget(s), 38 Sign(s), 127 GroundSpawn(s).
Exception while running 'QueensColony'
Shutting Down Zone 'QueensColony'
Removing connection
AddAuth: 3316 Key: 1243077845
27163406 New client from ip: 86.211.192.213 port: 50744
Adding new client...
Combined Packet!
ZoneAuth: Access Key, 1243077845, Character Name, Hyert, Account ID, 3316, TimeS
tamp, 1243077845
Loading new Zone 'QueensColony'
Loaded 694 NPC(s), 129 Object(s), 0 Widget(s), 38 Sign(s), 127 GroundSpawn(s).
Exception while running 'QueensColony'
Shutting Down Zone 'QueensColony'
Removing connection
Delete char request: 4660 2165
Delete char request: 4660 2166
Name check on: Allize
Resp: 1
AddAuth: 4660 Key: 1243082878
32194753 New client from ip: 91.77.202.189 port: 1453
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243082878, Character Name, Allize, Account ID, 4660, Time
Stamp, 1243082878
Loading new Zone 'QueensColony'
Loaded Exception while running 'QueensColony'
Shutting Down Zone 'QueensColony'
Removing connection
AddAuth: 4282 Key: 1243084010
33334376 New client from ip: 88.22.80.111 port: 51499
Adding new client...
Combined Packet!
Combined App Packet!
Removing connection
AddAuth: 4282 Key: 1243084083
33406737 New client from ip: 88.22.80.111 port: 49664
Adding new client...
Combined Packet!
Combined App Packet!
Removing connection
AddAuth: 4282 Key: 1243084570
33894375 New client from ip: 88.22.80.111 port: 61771
Adding new client...
Combined Packet!
Combined App Packet!
Removing connection
Name check on: Felony
Resp: 1
AddAuth: 4660 Key: 1243087114
36430981 New client from ip: 91.77.202.189 port: 1931
Adding new client...
Combined Packet!
Combined App Packet!
Removing connection
AddAuth: 4665 Key: 1243089552
38869434 New client from ip: 97.82.58.161 port: 1083
Adding new client...
Combined Packet!
Combined App Packet!
Name check on: Evrim
Resp: 1
Timeout up!, state=0
Name check on: Evrim
Resp: 1
Timeout up!, state=2
Timeout up!, state=2
Removing connection
Name check on: Evrim
Resp: 1
AddAuth: 4665 Key: 1243089715
39032391 New client from ip: 97.82.58.161 port: 1080
Adding new client...
Combined Packet!
AddAuth: 3316 Key: 1243089736
39055298 New client from ip: 86.211.192.213 port: 57642
Adding client to waiting list...
39055298 New client from ip: 86.211.192.213 port: 57642
Adding new client...
Combined Packet!
Combined App Packet!
Removing connection
Removing connection
Name check on: Assassinate
Resp: 1
AddAuth: 4665 Key: 1243089915
39232380 New client from ip: 97.82.58.161 port: 1095
Adding new client...
Combined Packet!
Combined App Packet!
Delete char request: 2571 2123
Name check on: Deathdread
Resp: 1
Removing connection
Name check on: Deathdread
Resp: 1
Name check on: Deathdread
Resp: 1
AddAuth: 2571 Key: 1243090277
39595093 New client from ip: 97.82.58.161 port: 64463
Adding new client...
Combined Packet!
Removing connection
AddAuth: 4665 Key: 1243090326
39642563 New client from ip: 97.82.58.161 port: 1210
Adding new client...
Combined Packet!
Removing connection
AddAuth: 2571 Key: 1243090405
39722643 New client from ip: 97.82.58.161 port: 54666
Adding new client...
Combined Packet!
Removing connection
AddAuth: 2571 Key: 1243090768
40085574 New client from ip: 97.82.58.161 port: 65030
Adding new client...
Combined Packet!
Removing connection
AddAuth: 2571 Key: 1243091960
41277245 New client from ip: 97.82.58.161 port: 59630
Adding new client...
Combined Packet!
Removing connection
AddAuth: 2571 Key: 1243093355
42673437 New client from ip: 97.82.58.161 port: 65284
Adding new client...
Combined Packet!
Removing connection
AddAuth: 2571 Key: 1243094642
43959611 New client from ip: 97.82.58.161 port: 57884
Adding new client...
Combined Packet!
Removing connection
AddAuth: 4665 Key: 1243094985
44302854 New client from ip: 97.82.58.161 port: 3245
Adding new client...
Combined Packet!
Removing connection
AddAuth: 150 Key: 1243095638
44961746 New client from ip: 192.168.1.1 port: 2995
Adding new client...
Combined Packet!
Combined App Packet!

When I tried logging in (last connection), I am stuck in teh client at "Logging into zone server".

[John Adams]

Dev server (linux) just went apeshit on us. I zoned into Antonica, then started running up the hill towards the lighthouse. I noticed no spawns were popping in as I moved around, so Zcoretri tried to zone into Antonica with me. While he was zoning, chat wouldn't show, and it took him a long time to zone. As he popped into Antonica, I saw this on the console:

Code: Select all

AddAuth: 126 Key: 1243101881
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
OP_ReadyToZone: Player Zcoretri zoning to Antonica
Zone: Removing client from ip:72.220.103.15 port:1418
2341337 New client from ip: 72.220.103.15 port: 1418
Adding new client...
Combined Packet!
ZoneAuth: Access Key, 1243101881, Character Name, Zcoretri, Account ID, 126, Tim                                                                                                                               eStamp, 1243101881
Read Timeout waiting for write lock: ZoneServer::MConnectedClients
Timeout waiting for read locks: ZoneServer::MClients
Possible deadlock attempt by 'ZoneServer::MConnectedClients'!
   0: 0C 00 14 00 61 77 65 73 - 6F 6D 65 20 6D 61 6E 2C  | ....awesome man,
  16: 20 74 68 61 6E 6B 73 21                            |  thanks!
Possible deadlock attempt by 'ZoneServer::MClients'!
Combined Packet!
OP_Unknown Received 0x0026
   0: 01 00 00 00 FA 43                                  | .....C
Shutting Down Zone 'NorthQeynos'
Combined Packet!
OP_Unknown Received 0x00b2
   0: 00 00                                              | ..
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
   0: 0C 00 1B 00 74 6F 6F 6B - 20 61 20 6C 69 74 74 6C  | ....took a littl
  16: 65 20 77 68 69 6C 65 20 - 74 6F 20 7A 6F 6E 65     | e while to zone
Left Grid 4294967295 and Entered Grid 1689359714
   0: 27 00 02 00 34 38                                  | '...48
   0: 0C 00 1B 00 79 61 68 2C - 20 6A 75 73 74 20 63 61  | ....yah, just ca
  16: 75 67 68 74 20 61 20 64 - 65 61 64 6C 6F 63 6B     | ught a deadlock

Specifically, the

Code: Select all

Read Timeout waiting for write lock: ZoneServer::MConnectedClients
Timeout waiting for read locks: ZoneServer::MClients
Possible deadlock attempt by 'ZoneServer::MConnectedClients'!

my text popped into chat the moment the lock released. Then, a moment later, I tried /reload spawns to see if the mobs would appear, but /reload locked up the server. Never saw the reload happen, and all chatting ceased, no commands, the usual problem I've been reporting the last week or two.

Hope this helps. I'll put server into debug, and run around today to see what I can catch.

[John Adams]

While not a "crash", I'll just stick this in here that /reload spawns does not appear to be working at all, once you run into an area where spawns are not .. spawning

Confirmed, /reload spawns does work if I am in an area that is spawned properly, before moving off (at /speed 100) to an area not spawning.

Also:
When I was in gdb mode, I ctrl-c the world and did a BT just to see what it would say.

Code: Select all

   0: 0C 00 48 00 61 63 74 75 - 61 6C 79 2C 20 6C 65 74  | ..H.actualy, let
  16: 20 6D 65 20 72 65 73 74 - 61 72 74 20 69 74 2E 20  |  me restart it.
  32: 49 20 6E 65 65 64 20 74 - 6F 20 74 65 73 74 20 69  | I need to test i
  48: 66 20 2F 72 65 6C 6F 61 - 64 20 73 70 61 77 6E 73  | f /reload spawns
  64: 20 77 6F 72 6B 73 20 65 - 76 65 72 2E              |  works ever.
   0: 0C 00 1A 00 6F 72 20 6A - 75 73 74 20 69 6E 20 74  | ....or just in t
  16: 68 69 73 20 63 6F 6E 64 - 69 74 69 6F 6E 2E        | his condition.
   0: 0C 00 04 00 62 72 62 21                            | ....brb!
   0: 0C 00 05 00 6F 6B 69 65 - 73                       | ....okies
^C
Program received signal SIGINT, Interrupt.
0x004f7416 in __kernel_vsyscall ()
Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.4-5.fc10.i386 openssl-0.9.8g-13.fc10.i686
(gdb) bt
#0  0x004f7416 in __kernel_vsyscall ()
#1  0x00346456 in nanosleep () from /lib/libc.so.6
#2  0x00383d6c in usleep (useconds=0) at ../sysdeps/unix/sysv/linux/usleep.c:33
#3  0x080982fb in Sleep (x=10) at ../common/unix.cpp:26
#4  0x0819104e in World::ProcessReloadSpawns (this=0x821aa00) at World.cpp:1239
#5  0x08194705 in World::Process (this=0x821aa00) at World.cpp:217
#6  0x080f1041 in main () at net.cpp:274

EDIT: on the No Spawning issue, I have just ruled out /speed as a factor. Even running normal speed, once I exit the initial spawn radius my client received, I get no new spawns.

Going to the Windows server now to see if it behaves the same, though we do not have this load of data on Tess.