Bad Day

My day today :

• Main production database decided to suck up every ounce of CPU power (from the server it resides on) and grinds the server to a complete halt.

• At midday 12pm.. !

• We managed to somehow gracefully shutdown the beast.

• Manager thinking we might as well reboot the server, asks me to restart the box.

• The server blue screens (The blue screen of death)...

• The error msg shows : STOP: Cxxxxxxx unknown hard error

(Thank you to the Engineer at Microsoft for putting in such meaningful error messages).

• Restart the server again - do recovery operation - cross fingers and pray it comes back up alive.

• Server is Online

• Recover database - Start database - database starts to grind server to a halt again - CPU 100%

• Shutdown database - Set job queues to Zero - Start database

• Nothing runs when job queues are at Zero - which makes database useless - Business people start calling again

• Alter job queues to 1/2 of what they were before (so atleast something runs)

• Everything back Online - 2pm

• Sit back - breathe - Monitor Server - Start writing Incident report - Wish I had taken a sickie today and just play WoW at home.

I knew I should have stayed at home today. . . :(