[FIXED!!!] CRITICAL Connection Issues

[John Adams]

First, I know this is not just me, because I have monitored EQ2TC and DB Project servers for weeks and see the exact same behavior happening to many, many others. Just no one else has reported it.

I cannot connect to *ANY* world, EQ2TC, DB Project (rarely) or JA Test server perhaps 9 out of 10 tries. When I do finally get into the game, I am desynced and booted 50% of the time and have to start the entire drill over. This is CRITICAL (to me) because I cannot make any headway on Spells or any other tasks on my plate if I cannot connect to my own world to test the changes.

To be clear, this is a First-Connection issue, when the zone is NOT always_loaded, and there is in excess of 30s delay from Play to starting to see the server->client communicating. Symptom:

lag.jpg

Rarely, if I attempt to log right back in, I can get into the game. Most times, I cannot, or client crashes on 2nd attempt. If I do get connected, and restart the world, the whole mess starts all over again.


The log entries show this is happening to MANY others that they get passed the loading character functions and go LD after 2 minutes from the first time they connected. Examples:

11:39:51 D Net: AddAuth: 150 Key: 1377455991
~
11:42:09 D Zone: Removing client 'Hanta' (1) due to LD/Exit...


19:20:14 D Net: AddAuth: 6484 Key: 1377397214
~
19:22:11 D Zone: Removing client 'Heron' (416) due to LD/Exit...


12:28:38 D Net: AddAuth: 6342 Key: 1376940518
~
12:30:27 D Zone: Removing client 'Alfa' (117) due to LD/Exit...

The timing is far too coincidental to be "Just Happens to JA".


I'd like some ideas on how to trace this to see what specific information is being dropped/skipped/timed out that causes these very disruptive connection issues. I don't have to tell you how costly these problems are for building a community, let alone the game itself.


PS: I'm going to set up my own loginserver locally again, and see if it still happens with everything contained on 1 box... if not, then it could very well be something wrong with my setup (which hasn't changed in 3+ years)

[Jabantiz]

I have seen this on my local login server/ world but no where near as often as you describe. What I have seen is the client seems to stop communicating, but still receives info so they can see chat but never send anything back to the server. Server will eventually disconnect the client from lack of communication.

I have no clue where this issue is coming from, not sure what would cause the client to stop communicating.

[xinux]

Yup it happens to me sometimes when i load into a zone that has spawns in it. I don't think it has ever happened to me loading into a empty zone.

[John Adams]

Yes. This is why my theory is it is happening more often now (along with a ton of other surprises) because theFoof's Frostfang zone is an actual Zone finally, with tons of moving parts. Our previous content didn't really do as much, and anything outside QC or OO never got touched anyway hence the problems are only showing up now.

And I don't know if it is just about spawn presence, or spawns with a pulse. Ie., if the zone has 5,000 spawns with no scripting, it is less likely to happen than on a "living zone".

[xinux]

Just did some changes to the world loginreply packet let me know if it helps. Tested with a live account and a DoV account.

[John Adams]

Jabantiz, did you authorize Xinux to start making C++ changes? I'm scared.

[Jabantiz]

Jabantiz, did you authorize Xinux to start making C++ changes? I'm scared.

lol

He posted his changes in irc last night and it was simple packet work, I hope this is an actual fix for this issue.

[xinux]

Doesn't look like it since i still had it happen to me.

[xinux]

Can i have everyone do me a favor go into your ruleset_details and change the following rules to these values.

LinkDeadTimer 1
RemoveDisconnectedClientsTimer 5

Don't think it will effect the 4k ping issue even tho i haven't got it in awhile after changing this but i don't seem to be hanging anymore when i camp and log right back in after adjusting those values.

[John Adams]

Your honor, the prosecution wishes to recall that we have said "Linkdead Timer" for about 6 mos now

EQ2TC has been on a 2000ms LD timer for quite a while. I just changed DB Project to "1" as you suggested, with the other setting too.

I have a desire to rip out LD code entirely, if it turns out to be the problem.

[xinux]

Ok more information to add to the 4k ping issue. The following testing was done in The Ruins zone.

I started noticeing when looking at our log we spam the shit out of the client with OP_EqUpdateGhostCmd while all the character information is still loading which led me to do the following.

i commented out the SendSpawnChanges in the following code so the update packets would never get sent. Completely closed my server and client about 20 times with no 4k ping issues. I then uncommented it changes the +50 to 100 logged in my ping climbed to about 1900 then recovered and was fine then i changed it to 200 and i'm on restart number 10 and no issues so far.

Zoneserver.cpp

Code: Select all

	MMasterSpawnLock.lock();
	if(!LoadingData && !zoneShuttingDown){
		if(!zoneShuttingDown && Timer::GetCurrentTime2() > last_spawn_update){ //check for changed spawns every 50 milliseconds
			last_spawn_update = Timer::GetCurrentTime2()+50;
			SendSpawnChanges();

Not committing anything till someone can tell me if there is a negative/noticeable effect to changing this.

I'm really starting to think this is a packet order problem and/or sending to much information when the client is trying to process other information that is why every now and then the client appears to catch up and your ping goes back down. This is also why it doesn't happen in zones with no spawns in it.

*On number 12 login i hit 1300 ping then recovered and was fine with 200 settings.*
* just did 8 more with no issues*

[John Adams]

I would think the only possible side effect would be spawns warping vs sauntering across the zone paths normally... but .2 seconds is pretty damn fast... if my math is right, .05 seconds seems almost unreasonable.

Maybe we make this a Rule, set it to 50 by default. Then we can change it in our rules and we can play with it without recompiling.


Edit: I've made this a rule but will wait for you to get back to me before committing.

[John Adams]

Xinux, when you were testing the 200ms update timer, did you remember to set your LD back to 120000 and your disconnect back to 100?

With all the stock settings, before this latest code attempt, I am connecting 100% of the time now. Blowing my mind lol... but I am also using the current live client. I will also try my 1199 client, which was all I used before.

[xinux]

Nope left the LD stuff set at 1 and 5 since i still think that it is the LD code when you camp and reenter the world unless you are seeing otherwise.

[John Adams]

I thought you said you completely restarted your world, so that's what I have been doing.