After yesterday's outage, I recompiled all services and now I am witnessing the linux box CPU pegged at 100% for the world server process. It doesn't stay there, but jumps up and then idles again, but it does it fast enough often enough to throw alarms on ESX for excessive CPU usage.
I do not want to have to throttle the VM, so can someone take the current database/scripts and make a Public server attached to LoginServer (ie, not minilogin, not a local machine, but a publicly available one) and troubleshoot what thread is eating up the CPU time? If no one has time to do this, I will limit EQ2Emu's CPU time on the host, which will likely result in poor performance for anyone logging in.

One thing I suspect is loading appearances to send to Login. Seems to explain it's repeated, short-lived spike time.

Thanks.

This is odd, any clue if it was happening before? Also I assume this is just with the server running idle with no connections or zones loaded?

Definitely only started happening recently. Server is also completely idle.

edit: At first I thought it was Login Appearnces, or the big picture packet... but after watching the log for a while, I do not see the server trying to Looking for Login Appearances... at all when the processors are stuck at a good 130% (100% of 1 proc, 30% of another). I'm not a Linux processes expert, but I believe this shows current_world running 7 "threads", maybe? Or processes? anyway, you can see it's just 1 of them that's freaking out.
I can't remember, does EQ2Emu have an option to disable LUA, or NPCs or anything like that, so I can try narrowing it down? I wouldn't think either would run on an idle server, but something is going apeshit.


Here's what the processes look like after a fresh startup under gdb (debug)
Aside
In looking into your Picture thing, here's something else of note. There are 36 character_pictures on World but only 17 on Login. The largest record on Login is 65k, while World has larger pictures. 65k seems suspicious. Maybe you already know this, and I've forgotten our conversations.

With no zone running there should be no lua/npc code running at all. 7 threads seems a bit much for an idle server, but I have honestly never looked into what the server does while idle. And no there is no way to disable LUA or NPC's.

As for the picture code it should only run when there is a connection, relies on a client packet. There is also 2 types of pictures, a head shot and a full body paper doll image, only the head shot is sent to the login server and it is much smaller size wise then the full body.

I saw on the info page there was a single connection to the server is it possible that a zone deadlocked (single thread freaking out) or has it been restarted recently?

There might have been a dead connection holding something open - that actually makes sense. The server Log didn't show any of the usual symptoms, of course.

I restarted EQ2Dev yesterday and looking at the processes now, it's calm. So this likely was a runaway process that ended when I restarted the world. Something you cannot even see unless you are SSH into the box looking at "htop" (process viewer).

Next time you shell in, run It should show you all processes spawned by that user, which is login, patcher and world. You can then press F5 to see it in Tree view, where the individual processes are clearer.

Well this looks like it will be a difficult one to figure out as there were no messages at all, glad a restart cleared it up though.