Deadlock and Exceptions

[John Adams]

Since the crashes/disconnects thread is just out of control, I'll put some new stuff here so it floats to the top.

LE, using 709 build, I am seeing consistent "possible deadlocks" trying to log in.

Code: Select all

OP_Unknown Received 0x0026
   0: 01 00 00 00 7A 44                                  | ....zD
Combined Packet!
Combined Packet!
OP_Unknown Received 0x01d2
   0: 00 00 01 0A 00 00 00 00 - 00 00 00                 | ...........
OP_Unknown Received 0x00b2
   0: 00 00                                              | ..
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
Left Grid 4294967295 and Entered Grid 924281492
Exception while running 'QueensColony'
Shutting Down Zone 'QueensColony'
Possible deadlock attempt by 'Combat::MFighters'!
AddAuth: 150 Key: 1243469492
15021900 New client from ip: 192.168.1.1 port: 1714
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243469492, Character Name, John, Account ID, 150, TimeSta

I saw Zexis log in, as I was selecting my char. I see the console filling with AddAuth, for other accounts than mine, and I got a server rejected my login - but surprisingly, got logged into my server. But a moment later as I was starting this post, I got disconnected.

Here's the whole console:

Code: Select all

Starting EQ2World logging . . .
[Status] Using database 'eq2live' at mysqlx64.mmoemulators.local
[Status] CURRENT_WORLD_VERSION:EQ2EMu 0.6.5 Developer Alpha
[Status] ./LoginServer.ini read.
[Status] Connecting to Update Server..
[Status]        Connected to Update Server: updates.eq2emulator.net:9103
[Status]        Checking for updates.
[Status]        Found Updates.
[Status]        Processing Updates.
[Error] Login reports that the latest version of the 'spawn' is 65019, you are u
sing 65012.  The query that will run to update your table is:
ALTER TABLE `spawn` ADD COLUMN `size_offset` TINYINT(3) DEFAULT '0' NOT NULL AFT
ER `size`



[Status]        Attempting to update database table 'spawn' ... SUCCESS!
[Status]        Table Update complete.
[Status]        Server is already up-to-date.
[Status] Loading opcodes..
[Status] Loading structs..
[Status] Loading command list..
[Status] Clearing player online flags..
[Status] Loaded 506 entity command lists.
[Status] Loaded 2383 NPC(s).
[Status] Loaded 318 Object(s).
[Status] Loaded 83 GroundSpawn(s).
[Status] Loaded 213 Widget(s).
[Status] Loaded 345 Sign(s).
[Status] Loading ground spawn items..
[Status] Loading loot data..
[Status]        Loaded 4 loot tables.
[Status]        Loaded 8 loot drops.
[Status]        Loaded 4 spawn loot lists.
[Status] Loading Transporter Information..
[Status]        Loaded 24 Teleporter(s)
[Status] Loaded global variables list..
[Status] Loaded visual states list..
[Status] Loading World Time..
[Status] Loading items.. (this might take a few minutes)
[Status]        Loaded 282 Bag(s)
[Status]        Loaded 2072 Food(s)
[Status]        Loaded 14064 Weapon(s)
[Status]        Loaded 32761 Armor/Shield Piece(s)
[Status]        Loaded 0 Skill Item(s)
[Status]        Loaded 9906 Regular Items
[Status] Loading spells..
[Status]        Loaded 3956 Spell(s)
[Status] Loading LUA spells..
[Status]        Loaded 24 LUA Spell(s)
[Status] Loading factions..
[Status]        Loaded 206 Factions
[Status] Loading skills..
[Status]        Loaded 129 Skills
[Status] Loading Quests..
[Status]        Loaded 142 Quest(s)
[Status] Loading Merchant Information..
[Status]        Loaded 0 Merchant Inventory Item(s)
[Status]        Loaded 0 Merchant List(s)
[Status] Loading Spawn Script Information..
[Status] Loading Zone Script Information..
[Status] Loading EQ time of day..
I'm counting how many times you've opened this...
Starting factory Reader
Starting factory Writer
World server listening on: tesseq.servegame.com:9200
World server listening on: 192.168.1.33:9200
Starting static zones...
Connected to LoginServer: eq2emulator.net:9100
67.11.231.175:9100: TCPConnection::RecvData(): Error: 10053
Connected to LoginServer: eq2emulator.net:9100
67.11.231.175:9100: TCPConnection::RecvData(): Error: 10053
Connected to LoginServer: eq2emulator.net:9100
67.11.231.175:9100: TCPConnection::RecvData(): Error: 10053
Connected to LoginServer: eq2emulator.net:9100
67.11.231.175:9100: TCPConnection::RecvData(): Error: 10053
Connected to LoginServer: eq2emulator.net:9100
AddAuth: 4487 Key: 1243459737
5263478 New client from ip: 65.78.105.32 port: 53956
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243459737, Character Name, Unaexeam, Account ID, 4487, Ti
meStamp, 1243459737
Loading new Zone 'QueensColony'
Loaded 694 NPC(s), 129 Object(s), 0 Widget(s), 38 Sign(s), 127 GroundSpawn(s).
OP_QuestJournalSetVisibleMsg Received 0x006c
   0: 05 00 00 00 06 00 00 00 - 07 00 00 00 09 00 00 00  | ................
  16: 1E 00 00 00 03 00 00 00 - 00                       | .........
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF 01 - 00 00 00 80 FF FF 0B     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF FF FF FF 0B     | ...............
OP_Unknown Received 0x0026
   0: 01 00 00 00 7A 44                                  | ....zD
Combined Packet!
Exception while running 'QueensColony'
Possible deadlock attempt by 'Combat::MFighters'!
Shutting Down Zone 'QueensColony'
Removing connection
AddAuth: 4487 Key: 1243460232
AddAuth: 4487 Key: 1243460245
AddAuth: 4487 Key: 1243460284
AddAuth: 4487 Key: 1243460318
AddAuth: 4487 Key: 1243460505
AddAuth: 4487 Key: 1243460543
AddAuth: 4487 Key: 1243460547
AddAuth: 4487 Key: 1243460806
67.11.231.175:9100: TCPConnection::RecvData(): Error: 10053
Error: LoginServer::Connect: 'TCPConnection::Connect(): connect() failed. Error:
 10061'
Connected to LoginServer: eq2emulator.net:9100
67.11.231.175:9100: TCPConnection::RecvData(): Error: 10053
Connected to LoginServer: eq2emulator.net:9100
AddAuth: 69 Key: 1243469414
14938726 New client from ip: 68.9.149.81 port: 51395
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243469414, Character Name, Zexis, Account ID, 69, TimeSta
mp, 1243469414
Loading new Zone 'QueensColony'
Loaded 694 NPC(s), 129 Object(s), 0 Widget(s), 38 Sign(s), 127 GroundSpawn(s).
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF 01 - 00 00 00 00 FC FF 3F     | ..............?
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 7F     | ...............
OP_Unknown Received 0x0026
   0: 01 00 00 00 7A 44                                  | ....zD
Combined Packet!
Combined Packet!
OP_Unknown Received 0x01d2
   0: 00 00 01 0A 00 00 00 00 - 00 00 00                 | ...........
OP_Unknown Received 0x00b2
   0: 00 00                                              | ..
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
Left Grid 4294967295 and Entered Grid 924281492
Exception while running 'QueensColony'
Shutting Down Zone 'QueensColony'
Possible deadlock attempt by 'Combat::MFighters'!
AddAuth: 150 Key: 1243469492
15021900 New client from ip: 192.168.1.1 port: 1714
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243469492, Character Name, John, Account ID, 150, TimeSta
mp, 1243469492
Loading new Zone 'QueensColony'
Loaded 694 NPC(s), 129 Object(s), 0 Widget(s), 38 Sign(s), 127 GroundSpawn(s).
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF DF 01 - 00 00 00 00 FC FF 5F     | .............._
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 5F     | .............._
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 5F     | .............._
OP_Unknown Received 0x0026
   0: 01 00 00 00 61 44                                  | ....aD
Removing connection
AddAuth: 69 Key: 1243469506
AddAuth: 69 Key: 1243469513
Combined Packet!
Combined Packet!
OP_Unknown Received 0x00b2
   0: 01 01                                              | ..
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
AddAuth: 69 Key: 1243469529
Exception while running 'QueensColony'
AddAuth: 69 Key: 1243469541
Shutting Down Zone 'QueensColony'
Possible deadlock attempt by 'Combat::MFighters'!
AddAuth: 69 Key: 1243469575
Removing connection
AddAuth: 69 Key: 1243469621
AddAuth: 69 Key: 1243469642

btw, restarted my world, and I got logged in fine but almost immediately threw an exception:

Code: Select all

Starting EQ2World logging . . .
[Status] Using database 'eq2live' at mysqlx64.mmoemulators.local
[Status] CURRENT_WORLD_VERSION:EQ2EMu 0.6.5 Developer Alpha
[Status] ./LoginServer.ini read.
[Status] Connecting to Update Server..
[Status]        Connected to Update Server: updates.eq2emulator.net:9103
[Status]        Checking for updates.
[Status]        Server is already up-to-date.
[Status] Loading opcodes..
[Status] Loading structs..
[Status] Loading command list..
[Status] Clearing player online flags..
[Status] Loaded 506 entity command lists.
[Status] Loaded 2383 NPC(s).
[Status] Loaded 318 Object(s).
[Status] Loaded 83 GroundSpawn(s).
[Status] Loaded 213 Widget(s).
[Status] Loaded 345 Sign(s).
[Status] Loading ground spawn items..
[Status] Loading loot data..
[Status]        Loaded 4 loot tables.
[Status]        Loaded 8 loot drops.
[Status]        Loaded 4 spawn loot lists.
[Status] Loading Transporter Information..
[Status]        Loaded 24 Teleporter(s)
[Status] Loaded global variables list..
[Status] Loaded visual states list..
[Status] Loading World Time..
[Status] Loading items.. (this might take a few minutes)
[Status]        Loaded 282 Bag(s)
[Status]        Loaded 2072 Food(s)
[Status]        Loaded 14064 Weapon(s)
[Status]        Loaded 32761 Armor/Shield Piece(s)
[Status]        Loaded 0 Skill Item(s)
[Status]        Loaded 9906 Regular Items
[Status] Loading spells..
[Status]        Loaded 3956 Spell(s)
[Status] Loading LUA spells..
[Status]        Loaded 24 LUA Spell(s)
[Status] Loading factions..
[Status]        Loaded 206 Factions
[Status] Loading skills..
[Status]        Loaded 129 Skills
[Status] Loading Quests..
[Status]        Loaded 142 Quest(s)
[Status] Loading Merchant Information..
[Status]        Loaded 0 Merchant Inventory Item(s)
[Status]        Loaded 0 Merchant List(s)
[Status] Loading Spawn Script Information..
[Status] Loading Zone Script Information..
[Status] Loading EQ time of day..
I'm counting how many times you've opened this...
Starting factory Reader
Starting factory Writer
World server listening on: tesseq.servegame.com:9200
World server listening on: 192.168.1.33:9200
Starting static zones...
Connected to LoginServer: eq2emulator.net:9100
AddAuth: 150 Key: 1243469797
55876 New client from ip: 192.168.1.1 port: 1714
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243469797, Character Name, John, Account ID, 150, TimeSta
mp, 1243469797
Loading new Zone 'QueensColony'
Loaded 694 NPC(s), 129 Object(s), 0 Widget(s), 38 Sign(s), 127 GroundSpawn(s).
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF DF 01 - 00 00 00 00 FC FF 5F     | .............._
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 5F     | .............._
OP_ChatFiltersMsg Received 0x013d
   0: 0D 00 FF FF FF FF FF FF - FF FF FF 7F FC FF 5F     | .............._
OP_Unknown Received 0x0026
   0: 01 00 00 00 61 44                                  | ....aD
OP_Unknown Received 0x00b2
   0: 01 01                                              | ..
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1
Left Grid 4294967295 and Entered Grid 819993724
Exception while running 'QueensColony'
Shutting Down Zone 'QueensColony'
Possible deadlock attempt by 'Combat::MFighters'!
Removing connection

Sorry. Something's amiss!


Edit: I reverted back to dev SVN 705, and all works well. I'll creep forward and see where it breaks. :/

[John Adams]

Final report:

As much as I would love to blame Scatman for forgetting a paren, the problem occurs starting with rev 707. Just went from 700 - 705, no problems.

Zone starts, throws an exception as soon as the defenders and invaders aggro each other.


Here's a backtrace done on Linux of the problem using rev 709 (yay the problem exists on both Win and Lin!)

Code: Select all

Left Grid 924281492 and Entered Grid 771354606
   0: 3A 00 04 00 39 3A 30 30                            | :...9:00
Possible deadlock attempt by 'Combat::MFighters'!

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb3ca2b90 (LWP 8692)]
0x05fd3862 in std::_Rb_tree_increment (__x=0x5)
    at ../../../../libstdc++-v3/src/tree.cc:68
68              while (__x->_M_left != 0)
(gdb) bt
#0  0x05fd3862 in std::_Rb_tree_increment (__x=0x5)
    at ../../../../libstdc++-v3/src/tree.cc:68
#1  0x080d3446 in std::_Rb_tree_iterator<std::pair<Entity* const, bool> >::operator++ ()
    at /usr/lib/gcc/i386-redhat-linux/4.3.2/../../../../include/c++/4.3.2/bits/stl_tree.h:190
#2  Combat::Process (this=0xea69bb4) at Combat.cpp:46
#3  0x08165b8d in ZoneServer::Process (this=0xea69778) at zoneserver.cpp:790
#4  0x0816601a in ZoneLoop (tmp=0xea69778) at zoneserver.cpp:3103
#5  0x0042a51f in start_thread (arg=0xb3ca2b90) at pthread_create.c:297
#6  0x0038b04e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
(gdb)

I ran up to the defenders in Queens Colony, set the time to 9:00 so I could see the invaders coming, and soon as they got within aggro range, the defenders didn't react (they do on Windows), and the mobs just bumbled around in a circle til the World crashed (Got signal 11 if not in debug).

Hope this helps ID the problem.

[LethalEncounter]

Yah, I see the problem. Apparently my previous crash fix just moved the crash from zone to combat. It will be fixed shortly.

[Image]

We ever thought of building a automated QA system so we can test for deadlocks and such automatically? We could even make an automated client really (I have one for EQ I could probably modify for EQ2)..

[John Adams]

That sounds great, Image - but I QA for a living, and unless someone else does it, I am not so interested in bringing more of my work home

But I agree, it could help LE on the front end if he could see troubles ahead of time. This particular issue is NPC on NPC combat, so you would only have had to start the zone (always_loaded?) to make it happen, I believe.

[ZexisStryfe]

I enjoyed crashing your server 5 times in 20 min

[John Adams]

Whose, LE's or mine? TessEQ2 is still running with no apparent crashes - but I haven't checked the Dev server yet

[Image]

Would be nice if we could at least start building a list of test cases, even if we have the community contribute.

[John Adams]

At the risk of sounding like an arrogant asshole, there is probably not too much you can think of in this arena that I haven't already thought of, my friend

test_plans.jpg

Again. QA is >what I do<, and that includes all aspects of QA/QC automation, test plans, and ridiculous quantities of reporting and statistics... some of my stats requests you yourself poo-poo'd, may I remind you But that's my job, and I get paid well for it.

My question to you is, what good will something like this really do to this project, besides make us look really smart and organized (which I am all for, mind you)? We can barely get 2 people to log into a server to test anything other than putting uber armor on and making themselves level 255... We can't get people to help maintain the Wiki. Hell, it's a struggle to get anyone to post and even appear remotely interested in our project.

In this, I have zero faith that anyone will step up and generate the massive amounts of data needed for thorough "test plans" for our little project -- though if you want to try, I got your back. You should know by now, I am freakishly drawn to organization and trying to look smart

[Wdneq2]

This section here....
My question to you is, what good will something like this really do to this project, besides make us look really smart and organized (which I am all for, mind you)? We can barely get 2 people to log into a server to test anything other than putting uber armor on and making themselves level 255... We can't get people to help maintain the Wiki. Hell, it's a struggle to get anyone to post and even appear remotely interested in our project

Not meaning to sound noobish or like a naughty child looking for direction , but my question is..
What will be of most use , and that i as a player can help with .
I have not created a server , or gained max level or adorned uber armour.
My characters have run around doing quests, interacting with npc's ,scenery, collectibles etc .
I have used the (broker window) itemsearch and equipped a couple of nutso items just to see them (then usually got rid of them) some slightly higher armour (again for looks and seeing if the buff's worked)
I use the /bug command a lot as it was suggested it was better (more practical ) than the forums
Are there specific tasks you need done? More than willing to help , but looking for a bit of guidance ..
Running multiple toons at once , looting same node over and over , camping a specific spawn, zoneing in and out ..logging in and out , repeating/dumping /collecting/doing quests ?

[John Adams]

Make sure you do not put yourself into my generalization, Woody... there have been a few very good, very helpful players who come here, test stuff, and sincerely make the effort to help US make a better server.

You few are an exception... trust me. Without you guys, we'd be doing it all and things would take even longer.

This discussion was not about poor us, but more about what good will it do to "professionalize" our side project... that's the point I am trying to make.

[Image]

I can give you a good example, I recently provided some fixes to the EQEmu source to protect against potential injections, unfortunately my login source code was different than the eqemu base and users were getting dropped out. This got pushed onto PEQ's live server in peak hours (oops..) - thats a lot of pissed off players. We can prevent builds from going out like that.

[John Adams]

Shame on you! and I agree. in cases like that, where the project is jumping with hundreds, maybe even thousands of people interested, I can see it being critical to make sure code is fully tested before it hits a "live" server.

And getting started early is not a bad idea. I'm just sayin, I can't do it. Just not enough time. I tried Testlink (which is a great little free web-based planner), it's just so huge, I collapsed under the weight of the task

If you want to use my Testlink server, I'll make you admin. But I am guessing you have your own wares for stuff like this.

[Wdneq2]

Make sure you do not put yourself into my generalization, Woody... there have been a few very good, very helpful players who come here, test stuff, and sincerely make the effort to help US make a better server.

You few are an exception... trust me. Without you guys, we'd be doing it all and things would take even longer.

This discussion was not about poor us, but more about what good will it do to "professionalize" our side project... that's the point I am trying to make.

Its all good.. i just want to throw it out there that sometimes we as players (well me ;p) just need a little direction ..push shove etc etc
Just putting my hand up ..if there's a task needed , give us a yell, and im more than willing to have a go

[John Adams]

LE, running the newest dev SVN (TessDev 712), the combat issues seem resolved on Queen's Colony, but Scat just had a zone bug out on him. I'll let him add what he was doing..

Code: Select all

Unhandled command: clearallqueuedabilities
   0: 01 00 08 00 72 65 6D 6F - 76 65 20 31              | ....remove 1
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 01 00 0B 00 63 6F 6D 62 - 69 6E 65 20 61 64 64     | ....combine add
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 01 00 0B 00 63 6F 6D 62 - 69 6E 65 20 61 64 64     | ....combine add
   0: C8 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 52 00 00 00                                        | R...
OP_SatMsg Received 0x00ab
   0: 0C 00 0E 00 68 6F 77 27 - 73 20 74 68 65 20 73 69  | ....how's the si
  16: 73 3F                                              | s?
Timeout waiting for read locks: ZoneServer::MClients
Read Timeout waiting for write lock: ZoneServer::MClients
Login Lock Check (MinStatus: 100):, UserStatus: 200, CharID: 9
AddAuth: 25 Key: 1243726629
Possible deadlock attempt by 'ZoneServer::MClients'!
Loaded 107 NPC(s), 22 Object(s), 25 Widget(s), 113 Sign(s), 0 GroundSpawn(s).
Removing connection
Login Lock Check (MinStatus: 100):, UserStatus: 200, CharID: 9
AddAuth: 25 Key: 1243726678
1624845 New client from ip: 68.45.201.93 port: 3629
Adding new client...
Combined Packet!
Combined App Packet!
ZoneAuth: Access Key, 1243726678, Character Name, Scatman, Account ID, 25, TimeS                                                                                                                               tamp, 1243726678
Zone: Removing client from ip:68.45.201.93 port:3604
Combined Packet!

I was in the QC, watching the defenders and invaders fight, die, respawn to make sure that was all working now - and was chatting with Scat who was in Graystone Yard. At one point, he stopped responding. He thought he got desynched, but I did not see normal evidence of this in the console.

Anyway, I also noticed the very first aggro the Defender got on an Invader after compile and bringing up the World, and starting the zone.. one defender aggro'd but then stood there for like 10s before fighting, and saw a few of the invaders piling up like they did before. But, after that one incident, everything was working fine. Even did a /repop.

Just something to keep an eye out for.