*Possible* memory leak?

[John Adams]

Ok LE, I have watched our Win32 and Linux binaries pretty closely for the last few days. Here's what I have discovered.

Win32:
At server startup, the memory usage before any connections is 150mb physical, 150mb virtual (avg).
After letting that server sit overnight and after only 1 successful connection and one failed connection, EQ2World.exe process is sitting at 398mb physical, 428mb virtual - 0 zones, 0 clients.
The Win32 binary, even with players online doing stuff, normally sits around 0-10% CPU (see comparison below - important)
When you MINIMIZE the console (to the Windows taskbar), memory stats fall through the floor! Garbage collection, anyone?

Physical Memory drops to 1,300K <-- Kilobyte! woot. Virtual stays the same, but hey - check out that low-memory usage, baby!


Win32 CPU Utilization Tests:
1 login, CPU was around 0%.
2 logins, another zone, CPU is 0%.
3 logins, a 3rd zone, after the CPU settled down, hung out around, yes, 0% steadily


WIn32 Conclusion: Only when lots of spell/script processing is going on do you ever see the Win32 binary using more than 0-1% CPU. Memory slowly creeps higher over time, suggesting a leak.



Linux x86:
Using "top" (thanks Scatman), eq2world binary starts up using 9% of the total system memory (2gb). That is comparible to the Win32, considering we have many more spawns and other data loaded.
RES showed 150m or so when it first started
VIRT showed about 150m or so as well - first startup
Overnight, and currently with 1 player online, RES and VIRT show 276m / 492m
Total system memory free last night was 1,300MB - this morning (8 hours later) it's 919mb

The linux-based eq2world process constantly utilizes between 15-40% of the CPU, no matter how many are connected (up to 3 connections monitored at this time). With 1 connection, it averaged 15%. Soon as a second came online in another zone, it shot up to 30% average and dropped and rised between 15-30%. When the third came on, it did not rise again, stayed at that 30-35%, with now 3 different zones active.


Linux CPU Utilization tests:
1 login, CPU was around 7%.
2 logins, another zone, CPU is 15%.
3 logins, a 3rd zone, after the CPU settled down, hung out around 23-30% steadily


Linux Conclusion: I understand EQ2Emulator is primarily designed for Win32, though Linux OS's should be supported by the inherent nature of the code. I am merely pointing out the obvious differences in how the two binaries run on their respective operating systems.


Scatman pointed out something obvious to me I hadnt' considered. My script generator puts a SetPlayerProximityFunction() in EVERY script, because I was noticing lately, more NPCs react to my presence than do not. The cheering, saluting, shouting something out, waving you over, or "shuddering" in fear. Scat mentioned that with 200+ NPCs in each zone constantly checking for proximity... could be a problem. But you tell me... we need this functionality to match Live (or custom) servers.

One thing the Windows version does not have is the completed work since the Dev team moved over to a Dev server, plus the Dev server has many more zones spawned and thus many more Spawn and Quest scripts implemented. But that does not really excuse these major differences.


Note: While I was typing this, the free memory on the Linux machine has dropped another 100mb -> now at 859mb. I think we're leaky.

[LethalEncounter]

Yah there are probably a few memory leaks still around. A while back I ran a tuner and got rid of the major ones, but there still have been some minor ones left over. Whenever you were doing your test was any GM types using any /reload commands? If even a single one of those had a memory leak it would appear to be very large. I'm not sure I believe that Windows only used 1.3 MB of memory unless it was doing something crazy when you minimized it. 1.3 MB is not much at all
The SetPlayerProximityFunction function by itself probably isn't an issue. The proximity code is only called as needed every 2 seconds and is checked whenever the spawn thread is checking the distances from various other NPCs. What might be causing an issue is the LUA scripts themselves though. I've never done some performance profiling of the LUA scripts, so those might be to blame if there are a lot of them being called back-to-back.

[John Adams]

The thing with minimize, which I am almost positive exists in the C#/.NET world anyway, is the garbage collection stuff that happens when you minimize the apps. My main project at work is in C#, and I see this all the time where the client gobbles up as much memory as the OS will allow, then bogarts it - not letting it go - until you "minimize", thus pushing the process to the background or something technical like that. The resulting effect is, anything not in use, gets disgarded. At least that's my understanding of it.

I think you said you use .NET 2003, so maybe there's something to that? If you don't, then I have no idea... world operates perfectly even after minimizing to 1.3mb. That's some damn efficient coding, my friend

During the tests listed above, it was just me and no /reload - just logging in and /zoning. Xan may have been online during one of the tests (Linux) and likely reloaded at some point, but I can run the tests again to be absolutely sure what I'm seeing.

[John Adams]

Scat and I just rand a simple test on the new Linux server. It's got 2GB ram allocated to the VM, and with world and all the other crap loaded, there was around 1.5GB RAM free.

We started out in Beggars Court, and each of us zoned into another populated zone, repeatedly, starting as many zones as we could stand. Meanwhile, I was watching "top" via the console and saw the RAm getting gobbled up, of course. Not really a lot though. I think we may have gotten around 15 zones running, populated. The server ended up with around 1.2GB free (see? not so bad).

Just as the lead zones started shutting down, I did a /reload spawns - which finished amazingly fast on this new server setup (yay!)

The bad came when those zones started shutting down. The ram never got freed. The eq2world process still held onto it's 690MB physical, 1300MB virtual.

So, I dunno LE. Something you'll probably have to try yourself. I cannot say this behaves the same on Win32... so not sure how much of a priority it is to you.

[Image]

I just looked in the code and saw one, I will clean some up

[LethalEncounter]

KK, I'll take a look at it. I'll be scarce until Sunday as I have family in town for the next couple of days.

[John Adams]

Thanks, Image.

Easter in the Lethal household?

[LethalEncounter]

Something like that

[Image]

Thanks, Image.

Easter in the Lethal household?

Yeah busy for me this weekend too, just to clarify though you are concerned about mem leaks or the cpu usage. Linux reports cpu/mem usage differently than Windows.

I have found a few memleaks but nothing serious, just unhandled pointers in memory really.

[John Adams]

I know Windows and Linux are different, so I was really just reporting the facts to show the drastic differences between the two. Since this is being primarily developed on a Win32x86 platform, I figured it was possible some NIX utilization issues might exist. But if you're saying the constant 30% CPU on an idle binary, and the constantly increasing memory usage until all available memory is exhausted is normal... I'll have to disagree, but accept it hehe. Since I don't know any better

I think we know there's a reload leak, so if you fixed that then that'll probably go far for us since we're constantly /reloading everything... and likely why I see it more on my server.

[Image]

guess this would be a good opportunity to test the profiler build

if you think it is with /reload that gives me a better idea, I was just skimming through the code really.

[John Adams]

Does the Profiler build work on Linux compiles? I know in VS2005 I can just select a different config, but I am not sure how to do that on Linux - nor how to get you the proper results. If I can reproduce the memory gobbling effect on Win32, I can try that build this week and see what happens.

[Image]

Does the Profiler build work on Linux compiles? I know in VS2005 I can just select a different config, but I am not sure how to do that on Linux - nor how to get you the proper results. If I can reproduce the memory gobbling effect on Win32, I can try that build this week and see what happens.

It should work fine on the linux build but you have to add the define for the profiler and you have to put a function call in the methods you want to track for usage. I didn't put them in the code yet because it didn't seem necessary until we needed to start worrying about performance

[John Adams]

Ahh, ok. If you feel like adding them, I'll run some tests with it. But you're right, performance is not really an issue yet... just my questioning the consumption of the ram and cpu on Linux compared to Win32.

Were you able to identify and leaks in the /reload commands? That would probably solve some of our resources problems for now. I am starting to suspect that the more we /reload, the more resources are consumed, until the world explodes with that Got signal 11 (which I don't know what that really means yet). This is guarenteed to happen nightly, hourly, while we're developing on the server.

[Bion]

Signal 11, or officially know as "segmentation fault", means that the program accessed a memory location that was not assigned.

edit:
cool i found something that may be of interest, personally I do not know much about debugging or alot of things really but did find a page that tells you how to use gdb with core dumps to reproduce the state of the fault
it is here