/spawn remove 1 bug

[John Adams]

LE, as Scatman reported and I have confirmed, /spawn remove 1 does remove the spawn from the DB, and from the zone, but when it's respawn timer expires, the spawn re-appears.

If you do a /repop, of course this does not happen. I think the spawn needs to be removed from the list of current spawns for respawn too.

[LethalEncounter]

Should be fixed.

[John Adams]

Got a bug with this, I think. Or, incredible coincidence

Code: Select all

Unhandled command: clearallqueuedabilities
   0: 01 00 08 00 72 65 6D 6F - 76 65 20 31              | ....remove 1
Timeout waiting for read locks: ZoneServer::MSpawns
Possible deadlock attempt by 'ZoneServer::MSpawns'!
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities

Eventually we recovered, at least I think we did. I am still there, but Scat is off in his own little world not responding.

However, after the second time it happened, I had to restart the world.

[Scatman]

I'm locked up waiting to see if I recover. I recovered the first time and everything was super fast again. Once I typed /spawn remove 1, instant lock up again. Waiting...

[Scatman]

Never recovered. Had to /q

[John Adams]

Well, like so many of my bugs at work, soon as I try to get a dev to look at it, everything works perfectly

Scat is my witness though, it really was way screwed up. But, I think it was something other than /spawn remove 1. Here's what really happened...

Scat went into Outpost, and per another report I posted, the amount of "sea life" we had scripted to move, literally killed the World. It was so busy processing movement, no commands would work. Things would appear locked up - like /spawn combine commands taking 5 minutes to respond.

I removed all the spawnscripts from the swimming sea life... I did a /reload spawnscripts, then a /reload spawns (global). Then, zoned into Outpost and swam around - discovering at that point it was definitely the spawnscripts that caused the slowness.

Scat then joined me in the Outpost, and tried removing some of the excess sea life. Everytime he /spawn remove 1, we locked up (the above posts).

Restarted the world, and we tried removing spawns with only 1 client in the zone - and everything worked. But, we discovered it was the restart of the world that fixed things, not the lack of 2 clients.



Long story short, something weird happened with /reload scripts, /reload spawns, and /spawn remove 1.

[John Adams]

Scat, here is what I am seeing right now while you are struggling to get back online.

Code: Select all

Unhandled command: clearallqueuedabilities
   0: 01 00 08 00 72 65 6D 6F - 76 65 20 31              | ....remove 1
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
Removing connection
Login Lock Check (MinStatus: 100):, UserStatus: 200, CharID: 9
AddAuth: 25 Key: 1243738278
3203483 New client from ip: 68.45.201.93 port: 4675
Adding new client...
Combined Packet!
ZoneAuth: Access Key, 1243738278, Character Name, Scatman, Account ID, 25, TimeS                       tamp, 1243738278
Timeout waiting for read locks: ZoneServer::MConnectedClients
Possible deadlock attempt by 'ZoneServer::MConnectedClients'!
Timeout waiting for read locks: ZoneServer::MClients
Read Timeout waiting for write lock: ZoneServer::MClients
Possible deadlock attempt by 'ZoneServer::MClients'!
Login Lock Check (MinStatus: 100):, UserStatus: 200, CharID: 9
AddAuth: 25 Key: 1243738379
3304889 New client from ip: 68.45.201.93 port: 4681
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243738379, Character Name, Scatman, Account ID, 25, TimeS                       tamp, 1243738379
Timeout waiting for read locks: ZoneServer::MConnectedClients

I seem to still be connected fine. Just cannot type any commands.

[Scatman]

Ya I'm watching it too. Was just about to post it. Where it says 'removing connection', I waited a good 5 minutes before that (while no commands would work) and finally exited manually out of the client.

Edit:
Completely stuck at log-in screen with the server backlogged message trying to get in.

[LethalEncounter]

If you want I can look at the NPC movement code and see if I can improve it (probably can). From what you describe it sounds like that possible deadlock was caused by the excess spawn movement and not really anything else. It says possible deadlock because it just gives up after 20 seconds, so if the server can't grab it for 20 seconds it thinks it is deadlocked. Could you do me a favor? Upload a copy of the scripts and the spawn id's of the ones that used it, or something along those lines. Just something I can use to easily find any bottlenecks in the system.

[Scatman]

I believe the deadlock happened once when we had the movement scripts in place and I never even got a chance to spawn remove 1, then it happened once when the spawns weren't moving and after I did a /spawn remove 1. Regardless, the movement with all those spawns were causing issues so I'll see if the scripts and IDs are already uploaded, they are pretty old. If you swim through the water in Outpost of Overlord and can't see past the wall of fish, then you know you have the right content It's one script that gets assigned to all the a fishs' spawn_ids which is a universal movement script.

[John Adams]

The TessEQ2-DB on SVN has these spawns. The ID's I know about (that I disabled last night) are:

2780011 (needlefang)
2780073 (sandstalker)
2780015 (sawtooth shark)
2780010 (shoal glider)
2780008 (triggerfish)
2780013 (triggerfish)
2780078 (vicious sandstalker)

The script that was assigned is in the Generic folder, "movement_circle_small.lua"

Code: Select all

function spawn(NPC)
	choice = math.random(1, 4)
	if choice == 1 then
		clockwise1(NPC)
	elseif choice == 2 then
		clockwise2(NPC)
	elseif choice == 3 then
		counter_clockwise1(NPC)
	elseif choice == 4 then
		counter_clockwise2(NPC)
	end
end

function respawn(NPC)
	spawn(NPC)
end

function clockwise1(NPC)
	x = GetX(NPC)
	y = GetY(NPC)
	z = GetZ(NPC)
	MovementLoopAddLocation(NPC, x + 7 , y, z - 8 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x - 5 , y, z - 10, 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x - 10, y, z + 9 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x + 5 , y, z + 8 , 2, math.random(5, 15))
end

function clockwise2(NPC)
	x = GetX(NPC)
 	y = GetY(NPC)
	z = GetZ(NPC)
	MovementLoopAddLocation(NPC, x + 2 , y, z - 8 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x - 7 , y, z - 5 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x     , y, z + 6 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x + 9 , y, z + 1 , 2, math.random(5, 15))
end

function counter_clockwise1(NPC)
	x = GetX(NPC)
	y = GetY(NPC)
	z = GetZ(NPC)
	MovementLoopAddLocation(NPC, x - 7 , y, z + 8 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x + 5 , y, z + 10, 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x + 10, y, z - 9 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x - 5 , y, z - 8 , 2, math.random(5, 15))
end

function counter_clockwise2(NPC)
	x = GetX(NPC)
	y = GetY(NPC)
	z = GetZ(NPC)
	MovementLoopAddLocation(NPC, x - 2 , y, z + 8 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x + 7 , y, z + 5 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x     , y, z - 6 , 2, math.random(5, 15))
	MovementLoopAddLocation(NPC, x - 9 , y, z - 1 , 2, math.random(5, 15))
end

(and yes, it has no script header )

The initial problem with this zone (which had nothing to do with /spawn remove 1) was that with all those hundreds of extra mobs having this same movement script assigned, the zone was just so overloaded process-wise, we could not even hail, right-click or /spawn combine {radius}, let along save or remove anything >>in the area of the traders outpost<< Getting out of that area, I believe everything worked ok.

Advertisement: Outpost of the Overlord still has 200% too many spawns, we're just waiting for some additional Spawn support before we can finish it.

Soon as I removed those 7 spawn_id's spawnscripts and restarted the world, everything was smooth again - so that's the cause, and honestly, I do not blame World for this. We just didn't clean the zone first


Where World is getting the blame is now that this is resolved, we still run into these deadlock/timer issues using /spawn remove 1 (or so we think, could be something entirely different, and just coincidence).


If I am running Debug on Linux, can I break the code when this is happening and backtrace to see what is stuck? Any way to know what it's stuck on? Maybe it'll help save you setup time.

[LethalEncounter]

Awesome thanks, I'll take a look at it after I'm doing testing the movement code.