Recent site outages. Thank you Erin, Piper, Samantha & Cat,

Printer-friendly version

Author: 

Taxonomy upgrade extras: 

for all your hard work and vigilance in keeping the site up and running. Erin, Piper, Samantha, Cat, my eternal thanks for all you do for us. Top Shelf is my home on the net. It means almost as much to me as my real life home.

Huggles and love and gratitude,
Catherine Linda Michel

Comments

Would be nice to know

Angharad's picture

what's happening with all the outtages, although I know our team are probably doing all they can to minimise them.

Angharad

It's a laundry list

erin's picture

Today's big outages were apparently compounded of flaky behavior of our server during backup (we're still investigating that) and a corrupted copy of our DNS being used by one of the big internet backbone companies. On top of all that, our new server we just installed failed and we have to ship it back for a replacement!

Hugs,
Erin

= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.

what are the specs on the server?

MadTech01's picture

I work in a corporate IT, so I was wondering what your new server specs are?
I know for a fact that we have had to max out our servers on RAM just to keep them running with all the SQL server engines we have to run for different Software we use. I mean going up to 64GB of RAM for us was no fun but we had to do it the SQL server applications were just eating the RAM like there would never be enough. But once we broke 32GB the problems and crashes we have disappeared on the main server. It was a dual Quad Core system, with 8 10k RPM HDDs in RAID's. but the access volume was just pushing RAM consumption skyward. Its kind of hard when deploying brand-new servers with Server 2008 R2 you just spent 30,000 USD on to find out they need more resources. But we got it beat. the apps we use, never mentioned the potentially workload demand the database engines could demand and non or our research even hinted at it but I know when you have a lot of people hitting a server at once those database server applications (aka SQL variants demand there RAM).

"Cortana is watching you!"

Dual Quad with 32 Gig of Ram

erin's picture

That's exactly why we're upgrading! The current machine is a quad core with 16 gig. The new machine can be upgraded to dual hex with 64 gig which we figure we will have to do sometime in the next 12 to 18 months.

Hugs,
Erin

= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.

If you continue to have the Crashes later.

MadTech01's picture

I sould suggest the RAM upgrade to the max, just based on how many people I see going through the Site at once unless the 32GB proves to be enough. You should be good with the Dual-qaud's. The Database server applications are not so much processor starved usually as they are RAM starved. Because the entire database can sometimes get to where it is running in RAM caused by client's (aka user access). That is why the big boys run their stuff on blade server farms. They break the workload up as much as they can to reduce hardware workload and prevent crashes. But alas my company is not big enough to have the resources to invest in Blade servers. we get by with traditional servers just loaded to the max with RAM. Just think now you can get in a brand new server dual-eight core processors. and with the new 8GB sticks you can really push it. Can't wait till they get 16GB sticks in production and supported by the hardware. because the only limitation a Server sometimes has for RAM support is how much RAM you can physically fit in the Box. I know that is our issue. but at least we have room to grow before we have to spend enough on servers again to buy a dad gum luxury car.

"Cortana is watching you!"

Server Config

Piper's picture

We have DualQuad opterons right now, we are moving to DualQuad XEON we also use SATA and SSD drives right now, and are moving to SAS


"She was like a butterfly, full of color and vibrancy when she chose to open her wings, yet hardly visible when she closed them."
— Geraldine Brooks


how rude!

What about thanking Sephrena?

It just seems rude to thank all those and not include Sephrena in that address to me.

Why not thank Sephy?

Because when the site went down today, the message that came up included the four names I mentioned, but not Sephy's. Sephrena does a lot for Top Shelf and she knows I thank for all she does.

Rude... I ain't. Abrupt? Maybe. Up front? You betcha. If I have something rude to say, I do it in private, NEVER in public!

Cool? Cool.

Hugs,
Catherine Linda Michel

As a T-woman, I do have a Y chromosome... it's just in cursive, pink script. Y_0.jpg

Im not on that Screen

...because I don't have anything to do with the hardware. Im just a front end monitor. I do a lot of things, but not the hardware. I always get ahold of Erin when things go wrong though! ^^

Besides that, all the names on that screen are the people that actually maintain and run the servers. I work for them.

And I appreciate all the help and thanks I do get. This place can exhaust anyone!!!

*hugs*

Thank you both, Tels and Cathy. ^^

ps: Samantha's name is misspelled on that screen also!

Sephrena

happy.gifT128550568840623eff129f896431aa3874a1dc4c539f.gifT12855057397797c0de0a79a1b70051a5899bf494ba04.gifHi.gif

Sephy helps give me time to help Piper

erin's picture

That's important, too.

Hugs,
Erin

= Give everyone the benefit of the doubt because certainty is a fragile thing that can be shattered by one overlooked fact.

It's My Fault

Piper's picture

The reason why Sephrena wasn't on that is, is because when I made that simple page a while back, Sephrena was taking a well needed break from the day 2 day operations of the site, and so it was a bit of "out of sight, out of mind" so to speak. Sorry Sephrena, I really do appreciate all the work you do around here making Joyce's day easier :)

-HuGgLeS-
-Piper


"She was like a butterfly, full of color and vibrancy when she chose to open her wings, yet hardly visible when she closed them."
— Geraldine Brooks


amen

amen to this messsage. thank you all.
robert

001.JPG

Sure hope it has a good warantee!

Great work Erin and the elves.

I noticed something was wonky but not all that long after it was okay again.

BTW for those of you who CAN but haven't, even *I* the Wisconsin miser, the man who invented copper wire when another person spotted a dropped penny at the same time has contributed to BC.

Time to put our wallets where are interests lie.

Lots of us here are on tight budgets so if you can't you can't but if you can afford a little it surely helps.

If I recall correctly, it costs hundreds a month, HIGH hundreds to maintain the servers and domain names and all.

If a thousand of us gave one dollar a month, 12 dollars a year, -- less than the price of THREE Mc Donald's Value Meals --that would cover the vast majority of Erin's expenses for maintaining all the sites she runs.

And she doesn't get a salary nor do her elves!

Hell if *I* can give a few bucks why not others?

-- grin --

Thanks again Erin, Sephy, Piper and all.

And thank you those who contribute be it money or stories. Both are valuable... though the money is really welcome I imagine.

John in Wauwatosa

John in Wauwatosa

Thanks to All

terrynaut's picture

Thanks to everyone mentioned in the blog post and comments. Hi Sephrena!

I'd also like to thank all the authors, poets, bloggers and readers who fill up and frequent this wonderful place.

Thanks and hugs!

- Terry