Critical System Recovery - 3/13/2017

[John Adams]

The fact this even worked is a miracle beyond explanation.

On March 9, 2017 the network area storage device housing every-single-one of my virtual machines (in an ESX cluster) had a critical issue come up that required a cold reboot of the disk array. Naturally, this wreaks havoc on RUNNING computers whose disks are on said drive array.

When the array came back up, I prayed the VMs survived. Most did; the one (and only) important one that did not -- the WebServer/MySQL machine. Of course. In a panic, I shut the entire environment down and went and had a few beers and slept on it.

With a well-rested mind and less panic, I began searching for "how to fix read-only drive issues in Linux" - only to discover that every fucking moron on the internet is wrong - about every single thing. Every command they suggested we run exacerbated the problem, making matters worse. Until I finally ended up late Saturday evening with a completely destroyed hard disk volume.

Believe me when I say, I did everything possible to restore services - but I was so thoroughly disgusted with this whole thing, I almost pulled the plug and just said "good luck"... until I remembered, I >just< upgraded EQ2Emulator.net to a new forum version - thus had a non-destroyed, relatively current backup

Yes, you lost a few weeks of posting (though not much has been going on here). But as of right now, EQ2Emulator.net is back in business, the Login, Patcher and World databases restored and functional. DB Editor will be offline a bit longer. IRC and SVN never went down, except the first initial panic moments to assess the situation (they are not on the Apache machine, but on a survivor).

This site is currently running on a temporary apache server until I have time to rebuild a new, more redundant webserver cluster. Let me know if anything doesn't work, or you see errors you do not believe should exist. There may be missing images, avatars, but hopefully not too many.

[John Adams]

Aside from a few dozen lost posts, I found only 1 orphaned file. I'll attach it here, so whomever owns it can re-post it if they want.

[Cynnar]

That was mine John, and I'll repost it this weekend.
Hate that there was an issue.
Glad to see that you are able to restore the backup, and we were lucky you were working on a update for us.

If there is anything I can do to help just let me know.

[John Adams]

Thanks Cynnar. The last few days were extraordinarily stressful to say the least With this temp webserver running, I was able to stop worrying about what I could not restore, and reverted the original (crashed) webserver VM to a previous snapshot I made just after setting the OS up (2015). Much to my surprise, it works. I think this means I can push files/data to it and go from there.

This site will likely have another outage or 3 along the way. Rest assured, I have it working here if nothing else.

[Cynnar]

I can only imagine. I went crazy when my nas box crashed. Ended up being the Mobo, and I was able to move everything over to a new/old computer with minimal problems. That was a small problem compared to yours, so yeah taking a stick of dynamite to the entire pile of computer anything would be one of those ideas you would have to fight doing.

That is something good news that you got a snapshot up and running. Maybe things will go smoothly now.

Any ideas on what caused this critical issue, or is it to soon to worry about that?

As for the outages, thanks for the heads up, and when it gets fixed it gets fixed. I'm sure you will let us know when you have things back to as normal as possible.

[John Adams]

Any ideas on what caused this critical issue, or is it to soon to worry about that?

To be perfectly honest, my lack of give-a-shits and laziness was probably 95% of the problem. I had backups running, but (stupidly) saving to the same box filesystem. I meant to backup to another mysql instance, just never got around to it. For 3 years Lesson learned though, if I'm going to keep hosting shit that people actually rely on, a solid backup plan is absolutely a must.

For example, aside from EQ2Emu's recent phpBB data backup for the forum upgrade, the most recent backups I found (to my utter shock) was Aug 2016. That's 3 mos >before< I even set up EVEmulator... so 100% of that site is lost. All the great, informative posts from visitors to help us along, etc. Talk about heartbreaking. Oh well, what can you do? Fortunately, I have the phpBB files and can just make a new DB. But it'll be starting completely from scratch; again.

[John Adams]

Okay, EQ2Emulator.net forums should once again be on their permanent server. Poke around, tell me if stuff is broken.

[Scribble]

Thanks for all thr work john.

[Scribble]

From discord chat.

I see the forums are up, but most links not working, ie anyhting i need to download to get started.

Tried to register and get error.
SQL ERROR [ mysqli ]

Field 'user_kb_permissions' doesn't have a default value [1364]

An SQL error occurred while fetching this page. Please contact the Board Administrator if this problem persists.

[Gangrenous]

Also, Sik San has never been able to access the forum.


http://imgur.com/Y4QhJ8J

[John Adams]

I see the forums are up, but most links not working, ie anyhting i need to download to get started.

Any idea what "most links" are that are not working? I have not added any portal stuff to this upgraded site yet.

Tried to register and get error.
SQL ERROR [ mysqli ]

Field 'user_kb_permissions' doesn't have a default value [1364]

An SQL error occurred while fetching this page. Please contact the Board Administrator if this problem persists.

This has been fixed, thanks for the reminder.

Also, Sik San has never been able to access the forum.

Sik San needs to move to a country that is not banned j/k... this is fixed too, until I get hit with spammers again.

[Ememjr]

I see the forums are up, but most links not working, ie anyhting i need to download to get started.

Any idea what "most links" are that are not working? I have not added any portal stuff to this upgraded site yet.

Tried to register and get error.
SQL ERROR [ mysqli ]

Field 'user_kb_permissions' doesn't have a default value [1364]

An SQL error occurred while fetching this page. Please contact the Board Administrator if this problem persists.

This has been fixed, thanks for the reminder.

Also, Sik San has never been able to access the forum.

Sik San needs to move to a country that is not banned j/k... this is fixed too, until I get hit with spammers again.

thanks got registers, the links not working were mostly pointing to stuff on the wikia. the wikia apparently is working now, and i was able to access the information i was looking for ie, link to svn, and information on running own server. i have finally got the downloads done, and in process of configuring, hoping to have server up and running soon. i am only worried about what client i will use, i have most retail boxed cd's and current live client.

I have been playing eq2 since release and really hopeful tht i can help contribute to this project, most like in the content\quest area and possibly in the development area

thanks for all you have done

[John Adams]

Thanks for the info. The Wiki may have some issues to iron out, so please report any problems here. I will (hopefully) be updating it soon to a newer version that is compatible with the modern forum software too. Expect periodic outages while this is happening.

PS: I also hope to have Server Status and Account Management modules sorted out asap, as accounts cannot be made without them. If you need a login account and world account created asap for your world, let me or Jab know and we can manually enter your account info for now.

[Gangrenous]

Will they be on the side as before?

[John Adams]

Will they be on the side as before?

That is what Jabantiz suggested, yes. Not wrapping the forum, just the landing page (portal) as before. Similar to how I did VGO's portal:

portal.jpg

Not fancy, but it works.