Crashes / Disconnects

[John Adams]

The development teams will be overjoyed to learn TessEQ2 (public) had a crash last night - and this time, I cannot blame Scatman, Zcoretri or Xan for reloads or anything! Woot... now I have one person to blame: LE!! hehe j/k

Unfortunately, I have no debug info but I will start running the server in debug this week to see if I can catch it. From the looks of the console around the time of the crash, there was a lot of those "combined packet!" warnings, then a crash.

Code: Select all

   0: 22 00 05 00 31 30 30 30 - 30                       | "...10000
Combined Packet!
   0: 22 00 05 00 34 30 30 30 - 30                       | "...40000
   0: 22 00 05 00 34 30 30 30 - 30                       | "...40000
   0: 22 00 05 00 31 30 30 30 - 30                       | "...10000
Combined Packet!
Combined Packet!
Combined Packet!
   0: 22 00 05 00 31 30 30 30 - 30                       | "...10000
   0: 22 00 05 00 34 30 30 30 - 30                       | "...40000
   0: CB 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
Combined Packet!
   0: CB 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: CB 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: CB 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
Combined Packet!
OP_Unknown Received 0x01d2
   0: 00 02 6C 00 00 00 69 00 - 00 00 01 6A 00 00 00 00  | ..l...i....j....
  16: 00 00 00 02                                        | ....
Combined Packet!
   0: 55 00 00 00                                        | U...
   0: 4C 00 03 00 31 30 39                               | L...109
OP_Unknown Received 0x01d2
   0: 00 02 6C 00 00 00 69 00 - 00 00 02 6D 00 00 00 6A  | ..l...i....m...j
  16: 00 00 00 00 00 00 00 00                            | ........
OP_Unknown Received 0x01d2
   0: 00 02 6C 00 00 00 69 00 - 00 00 02 6A 00 00 00 6D  | ..l...i....j...m
  16: 00 00 00 00 00 00 00 32                            | .......2
   0: F9 00 03 00 31 30 36                               | ....106
Unhandled command: share_quest
[Status] Using database 'eq2live' at mysqlx64.mmoemulators.local

[John Adams]

Awright, Mr LE. I am now convinced of 2 things...

1) Whenever I close my browser to EQ2Emulator.net, my server crashes - so I have to reopen it and come to this epic thread (think I'll just bookmark it )

2) Something is wrong with /reload spawns (and something else).

My server is LOCKED. I am the only one on it. I am standing perfectly still. I go to the DB and delete a few spawn records. I execute a /reload spawns and go about my business. Without having moved (you can see my auto-afk go up after the reload), and without any logins or other activity besides the one active zone, Queen's Colony crashes. And it was not Queen's Colony that I removed anything from.

Code: Select all

   0: 41 00 06 00 73 70 61 77 - 6E 73                    | A...spawns
[Status] Loaded 506 entity command lists.
Combined Packet!
[Status] Loaded 2447 NPC(s).
[Status] Loaded 386 Object(s).
[Status] Loaded 83 GroundSpawn(s).
[Status] Loaded 302 Widget(s).
[Status] Loaded 359 Sign(s).
[Status] Loading ground spawn items..
[Status] Loading loot data..
[Status] 	Loaded 4 loot tables.
[Status] 	Loaded 8 loot drops.
[Status] 	Loaded 4 spawn loot lists.
[Status] Loading Transporter Information..
[Status] 	Loaded 24 Teleporter(s)
Loaded 694 NPC(s), 129 Object(s), 0 Widget(s), 38 Sign(s), 127 GroundSpawn(s).
Combined Packet!
   0: 75 01 00 00                                        | u...
Unhandled command: afk
Exception while running 'QueensColony'
Shutting Down Zone 'QueensColony'
Possible deadlock attempt by 'Combat::MFighters'!
Removing connection

Now recall, this is the zone where I have gobs and defenders aggro'ing each other, so perhaps that was what the Combat::MFighters is referring to. I think elsewhere in this zone is more hatred going on, betwee gobs and skellies.

Regardless: I deleted nothing from Queen's Colony. It just happens to be the zone I am standing in, not moving. That's all the info I can think of right now. Hope it helps!


Edit: I did these same steps a few more times, even with a few more zones running, but could not get it to crash again. Who knows, man. It's a mystery.

Edit2: I take that back. After I logged out (did a /camp Johnrat over to the dev server), TessEQ2 reports 0 zones loaded, 1 client connected. So either the /camp confused it, or there is some odd hangup happening after I disconnected. Been about 15 mins tho, and no crash - just the lingering client count.

[John Adams]

Linux server crashed after what looked like some spell casting.

Code: Select all

(gdb) bt
#0  0x0808c3bd in Mutex::lock (this=0x1a2c) at ../common/Mutex.cpp:117
#1  0x0814e5cc in Player::ModifySpellStatus (this=0x0, spell=0xfd2fe98, value=66, all_spells=false, recast=0) at Player.cpp:936
#2  0x081c09e5 in SpellProcess::UnlockSpell (this=0x8242520, client=0xc53e180, spell=0xfd2fe98) at SpellProcess.cpp:367
#3  0x081c36f8 in SpellProcess::Process (this=0x8242520) at SpellProcess.cpp:153
#4  0x080fdc4a in main () at net.cpp:274
(gdb)

[Xanibunib]

I did a /useability 2 to try and summon a horse while you were on and it crashed... other than that the only thing i have changed is... I created a new quest script in the db to reserve a quest ID # but that shouldnt have made it crash...

[Xanibunib]

So i tired to summon another mount today, I got the +speed but no model so i tried the /test blah blah blah and it seems like the server deadlocked again. I followed your instructions

Debugging / Backtrace
I wanted to post this separate from the word explosion above. If you guys are on Dev and encounter something that is crashing the server constantly, please log in and run the world in DEBUG and get a BackTrace for LE - otherwise we won't know what's going on.

Simply fixing a bad script or data is not the solution. The world should not TERMINATE, EVER. That's the goal.

- Log in using your account.
- chdir ~/servers/dev.
- First, stop the persistent world with ./eq2emu stop.
- Type: gdb eq2world and hit enter.
- Once prompted, type "run" or just "r" and hit enter. You see the world starting up, and connect to the login server.
- After a crash, the debugger will be sitting at a prompt again. Type "bt" and hit enter
- Copy the contents of the screen starting where you typed BT through the end, and paste it to the Crashes/Disconnects thread. BE DESCRIPTIVE if you can about what you, or others, were doing at the time of the crash. The more steps you remember, the better the chance we can reproduce it and kill the bug.

Questions?

and I got this from the server...

Code: Select all

(gdb) r
Starting program: /home/eq2emu/servers/dev/eq2world
[Thread debugging using libthread_db enabled]
[New Thread -1209083584 (LWP 3893)]
[Status] Using database 'eq2dev' at mysqlx64.mmoemulators.local
[Status] CURRENT_WORLD_VERSION:EQ2EMu 0.6.5 Developer Alpha
[Status] ./LoginServer.ini read.
[Status] Connecting to Update Server..
[New Thread -1209087088 (LWP 3896)]
[Status]        Connected to Update Server: updates.eq2emulator.net:9103
[Status]        Checking for updates.
[Status]        Server is already up-to-date.
[Status] Loading opcodes..
[Status] Loading structs..
[Status] Loading command list..
[Status] Loaded 506 entity command lists.
[Status] Loaded 3907 NPC(s).
[Status] Loaded 625 Object(s).
[Status] Loaded 86 GroundSpawn(s).
[Status] Loaded 467 Widget(s).
[Status] Loaded 567 Sign(s).
[Status] Loading ground spawn items..
[Status] Loading loot data..
[Status]        Loaded 4 loot tables.
[Status]        Loaded 8 loot drops.
[Status]        Loaded 4 spawn loot lists.
[Status] Loading Transporter Information..
[Status]        Loaded 24 Teleporter(s)
[Status] Loaded global variables list..
[Status] Loaded visual states list..
[Status] Loading World Time..
[Status] Loading items.. (this might take a few minutes)
[Status]        Loaded 282 Bag(s)
[Status]        Loaded 2072 Food(s)
[Status]        Loaded 14064 Weapon(s)
[Status]        Loaded 32761 Armor/Shield Piece(s)
[Status]        Loaded 0 Skill Item(s)
[Status]        Loaded 9906 Regular Items
[Status] Loading spells..
[Status]        Loaded 3956 Spell(s)
[Status] Loading LUA spells..
[Status]        Loaded 24 LUA Spell(s)
[Status] Loading factions..
[Status]        Loaded 206 Factions
[Status] Loading skills..
[Status]        Loaded 129 Skills
[Status] Loading Quests..
[Status]        Loaded 153 Quest(s)
[Status] Loading Merchant Information..
[Status]        Loaded 206 Merchant List(s)
[Status]        Loaded 0 Merchant Multiplier(s)
[Status] Loading Spawn Script Information..
[Status] Loading Zone Script Information..
[Status] Loading EQ time of day..
Please stand behind the yellow line before I start chopping off heads.
Failed to open port 9300.
[Thread -1209087088 (LWP 3896) exited]

Program exited with code 01.
Missing separate debuginfos, use: debuginfo-install e2fsprogs.i386 gcc.i386 glibc.i686 keyutils.i386 krb5.i386 libselinux.i386 lua.i386 mysql.i386 openssl.i686 zlib.i386

After that it went back to gdb and if i tried to do "bt" i got this...

Code: Select all

(gdb) bt
No stack.

It looks like the debug didnt work? Or... i dunno, didn't look like much of anything useful for finding out whats happening

[John Adams]

You have to stop the current running world before starting a debug session. The "Failed to open port 9300." is a dead give-away that something else already has that port locked... and that would only be eq2world on that server.

Did you ./eq2emu stop before gdb eq2world? If not, that's likely why it looked funky.

[Xanibunib]

Indeed i did stop the world first...

[John Adams]

You sure it really stopped? When I tried stopping it, I got an error. Permission Denied.

[Xanibunib]

Hmm i'll try it again if you really want me to lol

[John Adams]

Crash it all you like! That's what it's there for.

Well, sorta.

[Scatman]

while (true)
/reload spawns

[Xanibunib]

BOOM! there it goes....

[Scatman]

Yeah you were calling SpawnSet, passing yourself into the function trying to make yourself attackable = 1

[Xanibunib]

haha, ok fixxed that.

[John Adams]

Yeah you were calling SpawnSet, passing yourself into the function trying to make yourself attackable = 1

Nubs.