Page 1 of 2
Unplanned Outage - 2013.11.06
Posted: Wed Nov 06, 2013 7:34 am
by John Adams
The main server for all my services has lost a hard drive from the mirror, so I am turning off non-essential services until I can get a replacement. Web, IRC and EQ2Emu services have been migrated to a backup server and should remain functional, but maybe a little slower.
Let me know if you experience any major problems.
Re: Unplanned Outage - 2013.11.06
Posted: Wed Nov 06, 2013 7:52 am
by Zcoretri
It does seem things are a little slower to pull up...Thanks for the heads up JA

Re: Unplanned Outage - 2013.11.06
Posted: Fri Nov 08, 2013 4:41 pm
by John Adams
Sorry about IRC, some channels became unregistered again and corrupted the DB somehow

so Services wouldn't start. Bleah... and Xinux and his war-dialing reconnector kept grabbing the channel for himself so I couldn't re-register

So I had to block IRC for a moment. You'll probably need to reconnect.
Meanwhile, the VMG/EQ2/VGO NOC is getting a much needed upgrade. I pulled the trigger today on parts for building a new server, since the PowerEdge 2900 is 7 years old, and it has been giving me trouble lately.
Not that you care, but I like hardware

- The 2900 is a dual quad core 1.8GHz box with 24gb ram and a raid1 500gb drives... fairly simple, but has run upwards of 15 VMs for me for years.
It will now be replaced (or clustered with) this:
1 x
http://www.newegg.com/Product/Product.a ... 6813131372
2 x
http://www.newegg.com/Product/Product.a ... 6819117258
6 x
http://www.newegg.com/Product/Product.a ... 6820161459
1 x
http://www.newegg.com/Product/Product.a ... 6817153170
1 x
http://www.newegg.com/Product/Product.a ... 6811129179
Although I think I goofed up on the case... not sure the 13" x 12" mobo will fit
Just hope it doesn't blow up my house when I plug it in.
btw, everything is currently running on a 12gb ram i5-2500k desktop with 1 single 64gb SSD os disk... I am wetting my pants how close I am to losing everything

Re: Unplanned Outage - 2013.11.06
Posted: Fri Nov 08, 2013 9:42 pm
by thefoof
<3 upgrades
Re: Unplanned Outage - 2013.11.06
Posted: Sat Nov 09, 2013 9:42 am
by Zcoretri
Looks like one nice server rig JA

Re: Unplanned Outage - 2013.11.06
Posted: Mon Nov 11, 2013 12:14 pm
by John Adams
The outages continue today, likely through mid-week as I get the new server built and configured. I need to move some VMs off my iSCSI NAS, since it apparently is not designed to run 20 VMs at once... murdering my home network
EQ2TC will be back up in a few hours, as will my other emulators I host, and SVN will return shortly as well.
Web and IRC remain up, but running a bit slow.
Re: Unplanned Outage - 2013.11.06
Posted: Mon Nov 11, 2013 8:05 pm
by Secrets
John Adams wrote:The outages continue today, likely through mid-week as I get the new server built and configured. I need to move some VMs off my iSCSI NAS, since it apparently is not designed to run 20 VMs at once... murdering my home network
EQ2TC will be back up in a few hours, as will my other emulators I host, and SVN will return shortly as well.
Web and IRC remain up, but running a bit slow.
IRC seems to be freaking out again >_<
Re: Unplanned Outage - 2013.11.06
Posted: Tue Nov 12, 2013 6:32 am
by John Adams
Hmm, what's wrong with it? I am logged in right now. I see no one else, which is unusual. I usually see Xinux.
Re: Unplanned Outage - 2013.11.06
Posted: Tue Nov 12, 2013 7:02 am
by Secrets
John Adams wrote:Hmm, what's wrong with it? I am logged in right now. I see no one else, which is unusual. I usually see Xinux.
It was disconnecting once every 5 minutes or so. Same with the EQ2Emulator LS.
Re: Unplanned Outage - 2013.11.06
Posted: Tue Nov 12, 2013 10:23 am
by John Adams
Yep, just noticed that my Cox ISP is dropping me every 5 mins so I have a report in to them too to diagnose that problem as well. No wonder I saw so many broken pipes in LS console.
Re: Unplanned Outage - 2013.11.06
Posted: Tue Nov 12, 2013 7:57 pm
by John Adams
Okay, I have restarted every piece of hardware in the house, from modem to firewall to VMs. I have a constant ping happening and not seeing anymore outages. Sitting on IRC too so if I fall asleep any second now

I will see in the morning if it held up.
Re: Unplanned Outage - 2013.11.06
Posted: Wed Nov 13, 2013 8:56 am
by John Adams
Parts are installed. Now if I could just get it to POST
NewServer.jpg
Re: Unplanned Outage - 2013.11.06
Posted: Wed Nov 13, 2013 2:01 pm
by thefoof
John Adams wrote:Parts are installed. Now if I could just get it to POST
NewServer.jpg
Percussive maintenance

Re: Unplanned Outage - 2013.11.06
Posted: Wed Nov 13, 2013 4:35 pm
by John Adams
Well I'm a dummy. Note i the pic, I forgot the 2nd ATX12v connector. Spent all day troubleshooting why the system wouldn't boot... this is my first (man-made) dual proc

Re: Unplanned Outage - 2013.11.06
Posted: Wed Nov 13, 2013 4:45 pm
by John Adams
Ahh, shit. All services coming down for about 10 mins. I just screwed up my iSCSI target and need to reboot both servers. Sorry
I'll take the forum down last, since no one is on IRC.