I was noticing intermittent lag on my server, every once in a while, and it worsens with more people on.
Turns out, the SaveClients call was causing a few seconds of lag (or worse, depending on the number of clients connected).

Initially, I tried having threads created to handle the save process, but this didn't really seem to improve much (confusingly). I would have liked to detach the threads, which would have eliminated the lag in all likelihood, but that seemed to run me into memory access errors.

I ended up moving the call from the zone server loop and into the world loop, much like how vitality is updated. This eliminates the zone lag, and I haven't seen anything negative happen as a result yet.

Just as an fyi,

It is likely the poorly constructed inserts over the years that never got optimized. I always meant to go back and fix those (being a Database nazi) but of course, never did.

Another thing I always meant to do was implement a Rule for how often the character data saves. 1 minute seems too often, but I think it's every 5 minutes - and it should be per character not globally (meaning, 100 players online should not all be saving their data at the same time, but on their own timers).

Still, prolly inserts. Some of the datasets are large (items, spells).

Generally, anything that communicates with another service (like a database or HTTP API or something of that sort) should be backgrounded, especially when the saves happen in a thread responsible for processing player movement updates and spell casts and things like that.

So far having the periodic Save happen from the world thread has worked wonders, although a lock needs to be added around hotbars, as changing them while the save happens can cause a crash.

Planning to do the same with saving during log off, and eventually background the loading for characters as well.