EQ2World.exe crash (rev 150)

[John Adams]

A player is repeatedly crashing my server, so I went ahead and tried to trace their steps and was able to reproduce what seems to be brining my server down.
Zoning from Source of Evil back to Queen's Colony seems to crash it. The spawn_signs data looks fine (widget_to_tutorial_island_good), so I am not sure what's causing the "memory cannot be 'read'" crash. There are no other signs of a problem at the console, either.

[LethalEncounter]

The player is clicking a zone sign and then it is crashing?

[John Adams]

It appears so, which is why right now Source of Evil zone sign back to Queen's Colony is set to "generic" so it stops crashing my server.
I've also taken notice each time the server is in a crashed state (and seeing popups also about deadlocks which have no further info), and it is always coming from 'somewhere' back TO Queen's Colony. I haven't noticed if the colony zone is reloading (after shutting down in other words) each time, but I am watching for common causes now.
Once it was coming from the tradeskill instance - where there is no door, so player had to /zone list queen, then /zone 253, then crashed. Another time it was someone doing the same thing from Willowwood back to the colony. Seems it's the Colony that's doing it. Maybe we'll luck out and this all has to do with the other crash caused by LUA/Proximity

[John Adams]

Well that blows my theory - this time I saw a crash where the player left Queen's Colony to Starcrest Commune, so it is not just tied to zoning into the colony. sorry, that would have made things easier. I am going to attempt to set up debug today to see if it shows anything more. You tell me what you need since I doubt we can make this happen on cue (though exiting Source of Evil did seem to crash it similarly each time).
Data on SVN if needed.

[John Adams]

Also, probably not related, but I have started seeing lots more of this type of text in my console than ever before - even while still on rev 150, used to not happen, now is:

Code: Select all

*** Duplicate packet: Expecting Seq=682, but got Seq=681
*** Duplicate packet: Expecting Seq=682, but got Seq=681
*** Duplicate packet: Expecting Seq=682, but got Seq=681
*** Duplicate packet: Expecting Seq=683, but got Seq=681
*** Duplicate packet: Expecting Seq=683, but got Seq=682
*** Duplicate packet: Expecting Seq=683, but got Seq=681
*** Duplicate packet: Expecting Seq=683, but got Seq=682
*** Duplicate packet: Expecting Seq=683, but got Seq=681
*** Duplicate packet: Expecting Seq=683, but got Seq=682
*** Duplicate packet: Expecting Seq=683, but got Seq=681
*** Duplicate packet: Expecting Seq=683, but got Seq=682
*** Duplicate packet: Expecting Seq=683, but got Seq=681
*** Duplicate packet: Expecting Seq=683, but got Seq=682
*** Duplicate packet: Expecting Seq=684, but got Seq=681
*** Duplicate packet: Expecting Seq=684, but got Seq=682
*** Duplicate packet: Expecting Seq=684, but got Seq=683
*** Duplicate packet: Expecting Seq=684, but got Seq=681
   0: CD 01 00 00                                        | ....

Not sure if this is tied to some action the player is doing, since I cannot see all actions in the console. Just results.

[John Adams]

Example of right after login (sometimes):

Code: Select all

*** Future packet: Expecting Seq=82, but got Seq=84
*** Future packet: Expecting Seq=82, but got Seq=107
*** Future packet: Expecting Seq=82, but got Seq=126
*** Future packet: Expecting Seq=82, but got Seq=145
*** Future packet: Expecting Seq=82, but got Seq=150
*** Future packet: Expecting Seq=82, but got Seq=151
*** Processing Future packet: Seq=84
*** Duplicate packet: Expecting Seq=85, but got Seq=84
*** Processing Future packet: Seq=107
*** Duplicate packet: Expecting Seq=108, but got Seq=107
*** Processing Future packet: Seq=126
*** Duplicate packet: Expecting Seq=127, but got Seq=126
*** Processing Future packet: Seq=145
*** Duplicate packet: Expecting Seq=146, but got Seq=145
*** Processing Future packet: Seq=150
*** Processing Future packet: Seq=151
OP_Unknown Received 0x00b2
   0: 00 00                                              | ..
   0: CD 01 00 00                                        | ....
Unhandled command: clearallqueuedabilities
   0: 25 00 01 00 30                                     | %...0
   0: 23 00 01 00 31                                     | #...1

fwiw, I rarely, if ever, saw this before the last week or so. Which is odd, because not only has the server code not changed, but the data hasn't changed much in a week+ either. This leaves me and my ISP, possibly?

[LethalEncounter]

Those messages would only be seen when running in debug mode, so if you were running the compiled release version you wouldn't have seen them. Those messages for the most part are harmless and are just letting you know that there was some lag and the client needed to catch up. Left alone is should correct itself. Are you running your server on Windows or Linux? If on windows could you run it through the debugger so if it crashes you can catch the callstack? If on Linux, could you run it using gdb to hopefully catch the same? If you can't catch it I'll open my server up and see if I can catch it. Some of these errors can be prevented by removing some of the processing from the main thread. Eventually I'll get around to doing this.

[John Adams]

I am currently running Windows with debug, running from VS2005. Oddly, never saw the crash today at all so far. Switched to the new rev 151 and left it running on normal mode for the night. I usually catch 4-5 crashes a day - so far, nothing. Weird.

[John Adams]

Caught this call stack right after manually using /spawn create to make an NPC, then assigning it a spawn script, reloading the LUA system and typing /repop - crashed. I didn't get a chance to reload the NPC list before the crash, so I'll try to make it happen again.

Code: Select all

>	EQ2World.exe!NPC::Copy()  Line 44 + 0x6 bytes	C++
 	EQ2World.exe!World::GetNewNPC(unsigned int id=260001)  Line 142 + 0x19 bytes	C++
 	EQ2World.exe!ZoneServer::AddNPCSpawn(SpawnGroup * spawngroup=0x10eb6820, SpawnEntry * spawnentry=0x0f97bb60)  Line 909 + 0x11 bytes	C++
 	EQ2World.exe!ZoneServer::ProcessSpawnGroup(SpawnGroup * spawngroup=0x10eb6820, bool respawn=false)  Line 827 + 0x1b bytes	C++
 	EQ2World.exe!ZoneServer::ProcessSpawnGroups()  Line 867	C++
 	EQ2World.exe!ZoneServer::Process()  Line 601	C++
 	EQ2World.exe!ZoneLoop(void * tmp=0x0fd1d010)  Line 2371 + 0x8 bytes	C++
 	EQ2World.exe!_callthreadstart()  Line 293 + 0xf bytes	C
 	EQ2World.exe!_threadstart(void * ptd=0x11006ca0)  Line 277	C
 	kernel32.dll!7c80b713() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

This the kind of thing you're looking for?

[John Adams]

Here's a call stack on Got signal 11. The server was idle for hours after a Timeout up! error, then the first poor sucker to try to connect crashed it.

Code: Select all

>	EQ2World.exe!EQStream::GetrIP()  Line 261 + 0xf bytes	C++
 	EQ2World.exe!Client::Client(EQStream * ieqs=0x00000000)  Line 92 + 0xe bytes	C++
 	EQ2World.exe!main(int argc=1, char * * argv=0x003a3a18)  Line 248 + 0x32 bytes	C++
 	EQ2World.exe!__tmainCRTStartup()  Line 327 + 0x19 bytes	C
 	EQ2World.exe!mainCRTStartup()  Line 196	C
 	kernel32.dll!7c817067() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

Console read:

Code: Select all

AddAuth: 25 Key: 1223581528
45720465 New client from ip: xxx.xxx.xxx.xxx port: 1030
Adding client to waiting list...
Removing client from waiting list...
Got signal 11

I noticed "Adding client to waiting list..." I'd never caught before. Hope this helps.

[John Adams]

Another one, seems to be the same place:

Code: Select all

>	EQ2World.exe!EQStream::GetrIP()  Line 261 + 0xf bytes	C++
 	EQ2World.exe!Client::Client(EQStream * ieqs=0x00000000)  Line 92 + 0xe bytes	C++
 	EQ2World.exe!main(int argc=1, char * * argv=0x003a3a18)  Line 248 + 0x32 bytes	C++
 	EQ2World.exe!__tmainCRTStartup()  Line 327 + 0x19 bytes	C
 	EQ2World.exe!mainCRTStartup()  Line 196	C
 	kernel32.dll!7c817067() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	

Different console this time:

Code: Select all

238 Object(s), 7 Widget(s), 128 Sign(s).
OP_ReadyToZone: Player Gobiin zoning to Castleview
1573654 New client from ip: xxx.xxx.xxx.xxx port: 1653
Adding client to waiting list...
Removing client from waiting list...
Got signal 11

[LethalEncounter]

Awesome work! Those are exactly what I need to hopefully fix both problems.

[LethalEncounter]

Both are fixed and will be available tomorrow.

[John Adams]

Thank god, man. Right now, my server is totally screwed when it crashes because I foolishly installed MSSQL 2005, which stupidly installed some VS2005 interface and threw a debugger config on my otherwise clean box. Now when the world crashes, it does not restart but pauses with a damn "click OK to continue" crappy dialog that I cannot seem to turn off.
Long story short, thank you. Can't wait.

[LethalEncounter]

Np If you have some free time I'll check some pretty massive changes in for you to test. These include all your database changes, model_type instead of race_type, the crash fixes, and a few fixes here and there. If you choose to accept this task, grab the code from private SVN and run the following queries to update your DB:

Code: Select all

ALTER TABLE `character_pet` CHANGE `race_type` `model_type` INT(10) UNSIGNED DEFAULT '0' NOT NULL;
ALTER TABLE `characters` CHANGE `race_type` `model_type` MEDIUMINT(8) UNSIGNED NOT NULL;
ALTER TABLE `characters` CHANGE `soga_race_type` `soga_model_type` MEDIUMINT(8) UNSIGNED NOT NULL;
ALTER TABLE `spawn` CHANGE `race_type` `model_type` MEDIUMINT(8) UNSIGNED DEFAULT '0' NOT NULL;
ALTER TABLE `spawn_npcs` CHANGE `soga_race_type` `soga_model_type` SMALLINT(5) UNSIGNED DEFAULT '0' NOT NULL;
alter table `characters` add column `current_zone_id` int(11) UNSIGNED DEFAULT '253' NOT NULL after `current_zone`;
update characters, zones set current_zone_id=zones.id where zones.name=current_zone
alter table `revive_points` add column `zone_id` int(10) UNSIGNED DEFAULT '12' NOT NULL after `zone_name`, add column `respawn_zone_id` int(10) UNSIGNED DEFAULT '12' NOT NULL after `respawn_zone`;
update revive_points, zones set zone_id=zones.id where zones.description = zone_name;
update revive_points, zones set respawn_zone_id=zones.id where zones.description = respawn_zone;
alter table `spawn_signs` add column `zone_id` int(10) UNSIGNED DEFAULT '0' NOT NULL after `zone_name`;
update spawn_signs, zones set zone_id=zones.id where zones.name=zone_name;
alter table `character_details` change `character_id` `char_id` int(10) UNSIGNED default '0' NOT NULL;
alter table `bugs` add column `zone_id` int(10) UNSIGNED DEFAULT '0' NOT NULL;

I have completed some parser changes, but I want to make sure this works before I check that is so we minimize the parser db updates.