36hrs Straight

Things I learnt from having to work 36 hrs straight in my office:

• In an 'green' environmentally friendly office, when all the lights go off finally for the last time, you can't turn them back on.
• This includes the lights in the toilets
• A mobile phone light is not enough to illuminate a toilet cubical
  
• I don't know how to bring Coffee machines out of standby mode. However, I do know the genius that programmed it, set it to go on standby from the hours of 12am-5am
• The same genius probably figured out that since the coffee machine can automatically grind coffee beans, they don't need to stock any Instant coffee.
  
• 12am - 5am are the hours people would need coffee the most, if they were pulling an all-nighter
• Lights out is at 7pm - after this the lights are activated by sensors. At least until 12am. (see point 1).
  
• Sensor activated lights do not detect a person typing away at a their desk or when they are talking on their phone.
  
• They will however detect something, if that person starts frantically waving a sheet of A4 paper in the air.
  
• Aircon shuts off at 7pm and does not go back on until 5am.

I do know once thing for sure though, the next time I'm looking at pulling another all-nighter, I'll be pulling out this post, and asking if all of the above have been resolved. Else I'm going to go and work on it from home.

Gdnite all.. :)

week in training

I'm down in Melbourne for a week in training.

Melbourne is freezing cold as usual. So far I've been stuck in the training room for the whole day, and haven't been able to go out to do any retail therapy yet. Depending on how it goes today, I'm going to try and sneak out and atleast get to the chocolate shop.

I guess the best thing about being down here is that I get to stay at the Crown Towers.

There is a LCD screen on the wall (with Foxtel), of where the spa bath is.

I guess that says it all... :)

Start at 7am

I volunteered for the morning shift this morning. (?!)

Yeah... I know... the whole 'I'm-not-a-morning-person-issue'. But it actually seemed like the logical thing to do. Mainly because the morning shift guy had been sick for the last two days and didn't sound like he was coming back to work anytime soon. Plus, I had to catch a flight back to Perth this afternoon.

Which means if I start early, I can leave early.

More time to go home and pack (because I still haven't yet). Therefore, less chance of missing my flight.

I have a newfound respect for morning shift people though. Waking up before dawn. Actually getting out of the bed before dawn. Not falling back to sleep on the train. And actually being able to be productive at work.

I think I was dumbfoundedly staring at my screen for atleast 30 mins before I actually remembered how to log in, and start the morning checks on my own systems.

I don't think I'm going to have any issues sleeping on the plane tonight.... :)

Interview Madness

In 20 mins I will be interviewing my would-be-manager. Yeah.... my new manager.

I say 'would-be' because, he has already been through interviews with senior management, but somehow one of my colleagues from the other team and myself was asked to sit in an interview with him as well.

I doubt whatever we have to say after that interview would make a difference.

I have still yet to congratulate the HR personnel or Senior manager who came up with this idea of doing 360° interviews.

All I know is that I have to be careful with the questions I ask him. HR even sent us an email outlining some possible questions to ask.

The Email noted that I shouldn't ask any hard or tricky questions, and outlined the possible implications of doing so, given that this person may very well be my next manager.

I may as well start shining his shoes in the interview.

His resume is impressive. However, his background in IT is not in Storage. He spent his first 10 years working as a software developer.

But then I'm not supposed to ask him questions on how he might handle the position he applied for, given he has no relevant experience in the specific industry.

I guess I better start working on my shoe polishing techniques instead.

busy busy...

Probably been a while since I've neglected this blog for so long.

But I have been busy. Damn !@#% busy.

My manager has gone holidaying in Japan. And literally left me in a pile of sh-t.

Its 11.22pm now, and I'm still in the office.

I still have to come in tomorrow (a saturday) for work, and have been nicely asked to be onsite at 7am the coming Monday.

I need to bring a pillow into the office...

Its the People

It has been a sh-t week.

Thursday left at 10pm.

Friday left at 11pm.

Call outs on Sunday.

One power outage and my whole week + weekend ends up screwed.

IT equipments don't like power outages. 1 motherboard change and 2 interconnect HBA card changes later, and everything seemed to be up and running. Hardware failures -- I can handle, its the people that get to me.

After I got my downtime approved, some IT Continuity Manager called up and said that the outage was not to go ahead, and if we were to replace broken parts, then we were to do it without any downtime.

Nutcase.

He may as well asked a doctor to perform open heart surgery without putting the patient to sleep first.

I explained that the Interconnect cards were down. Which is the hardware that controls the cluster failover links. The ability to have redundancy. And if there was another power outage, or even any other hardware failure, and then if the system goes down, there will be an extended period of downtime during the day for which I cannot account for.

After another hour of phone calls, it was still a 'No-Go'.

I wonder what part of, "If system suffers another problem, and Interconnects not fixed, system will go down indefinitely", did he not understand.

He said that if I couldn't guarantee that the system will be up by 7am the next day, then I couldn't go ahead. I told him I couldn't guarantee that the system would be up in the next 5 minutes.

I was so tempted to just shutdown everything and report a Severity 1.

Although if I did do that, I will probably be spending the rest of the month filling out incident reports.

In the end my manager took the phone, and said, 'If we don't do anything tonight, and system goes down tomorrow, I will put it down that you were the one who didn't allow work to be done tonight?'.

I didn't hear the answer, but once my manager put the phone down, he told me it will go ahead tonight.

That was thursday night. After 4 reboots and the system still didn't come back up after the interconnects were replaced, I realised the problem was much bigger than just the HBA cards. It would take another 4 hours for parts to be dispatched onsite. And it looked like a new motherboard was needed.

Normally, I wouldn't mind staying back. I could just go home, get something to eat and come back when parts arrived. But not this time. I called my manager up and asked if he would want me to stay, he said 'No', I could do it tomorrow night.

So, Friday morning I called to let him know that it didn't go as planned and that further downtime would be required.

He didn't sound happy. But I really didn't care anymore.

Good Job

Got a call from the big boss just as I sat down to log onto my computer.

Luckily I didn't decide to watch the next episode of CSI NY this morning.. otherwise I would have missed the phone call... and be knowingly late.

The call was to congratulate me on a job well done.

Apparently one of the PMs or Users I was working with, sent him an email commemorating my performance.

Its not everyday I receive a call from someone thanking me and telling me I did a good job. 95% of the calls I get everyday are from users calling in to say something is broken. And usually when the problem has been fixed, I may get a 'thank you' at the end of the conversation, but that is usually it.

Its nice to have a phone call thats different... :)

The big boss also told me to go out and have dinner with my bf and put the tab in expenses later on. hummm... :) Where to for dinner tonight ?!

Reason #712 Why I hate dealing with Dimwits

After my last clustered failover occurred, I lodged a case to the support engineers in the US, in hopes to determine what might have caused the system to panic.

After a week of not hearing anything from the support engineer I was working with (he had promised to call me back), I saw this logged into the case notes this morning:

--------
(S) I tried reaching you at 612 9*** ****, the listed number above, is there an alternate number to reach you that number rang busy
--------

I really wanted to put an update into the case notes which says something along these lines:

--------
+612 9*** **** is the correct number, and the reason why it rang busy ... is BECAUSE I'M FOOKING ON THE FOOKING PHONE.

Alternatively you can contact me on my mobile (+6141********), but I may not be able to pick up the call if I am busy on another line. If that is indeed the case, please try calling again later.... fooking dimwit !!!!!
--------

I was tempted, but I didn't. Instead I kept it simple: "If number is busy, pls try again later."

Today is a busy day. Busy days are not good days to be dealing with @#$! Dimwits .. grrrrrr.

Email Stuff-Ups #1023

After sending an email outlining the implementation plans to another department's manager, I got a reply back with only one word.

"Excellent"

And my reply......

"Thank you, Mr Burns."

---------------------------

And No... (in case you are wondering), his surname is not 'Burns'.

I think I've watched too many episodes of The Simpsons... And why do my fingers type faster than my brain works !?!? D'oh!

Busy Busy

I've been so busy for the past few days that I feel like I'm running around, resembling somewhat of a mad chook with my head half chopped off.. Its not a nice sight.

Today I have spent most of my time travelling between two datacenters trying to get my system back online. Properly online.

One of my filers 1/2 failed yesterday, luckily the other cluster kicked in. And now I know a bit more (than I would have liked to) about Netbackup and NDMP set up commands. I think I was actually seeing those commands in my dreams last night !! Well.. maybe not my dreams... I never remember my dreams anyway. But it was what I was thinking of when I woke up this morning.

I'm actually tempted to take up the job offer with another outsourcing company (courtesy of my ex-colleague).. this one though is mainly dealing wintel servers and so it would be back to Citrix \ Active Directory and Exchange. I wonder how long I would last in that role before I get bored again. Hmm.

Oh.. the other 1/2 of my day was spent at the datacenter trying to unscrew the console cable that some tech had attached to my now dead offline server.

The cable was so firmly attached to the COM port, that it actually look like it will come out WITH the attaching screwed-in console cable. I actually ended up giving up on the cable and using another cable to another server instead. :(

To all the techs out there... (especially to the tech that did this)... PLEASE DO-NOT USE POWER DRILLS \ ELECTRIC SCREWDRIVERS WHEN ATTACHING CABLES !!!!

Same goes for using power drills \ electric screwdrivers on Thumb screws !!!

Thumb screws are supposed to be able to be unscrewable using only your thumbs !! NOT take the first 2 layers of your skin OFF YOUR THUMB ~!!!

Grrr... excuse the caps..

Guess its time to go home and douse myself in cold water.

Gdnite all. :)

Don't Wish 2

-----Original Message-----
From: Console Svc Account
Sent: Saturday, 10 February 2007 8:07 AM
Subject: EventLog Error

Event Type: error
Computer Name: **********005
Event Code: 4111
Source Name: cpqasm2
Time Written: 2/10/2007 8:06:24 AM
Category: 0
Message: Environment Abnormality Auto Shutdown (EAAS) initiated due to thermal reasons, either resulting from the system overheating, or from the loss of cooling.

-----Original Message-----
From: Console Svc Account
Sent: Saturday, 10 February 2007 8:10 AM
Subject: Cluster Alert from **********004

SHELF_FAULT on **********004 at Sat Feb 10 08:09:41 EST 2007

-------------------------------


Dear God.... I promise I will never complain I'm bored at work anymore.. I promise I will never wish for systems to crash & die (in this case crash, die & burn - just because I am bored)... I promise I will never complain I am bored at work again ...


NB. I came into work this morning to find 3 clustered systems drop off the network (with the above error msg alerts sent before they died) ... *sob

Don't Wish

I had a typical Monday morning, breakfast, coffee, clearing up my escalation queue, lunch, work on new cases, coffee, follow-up existing cases, 5pm - Go home.

Tuesday, same kind of thing except I cleared up everything by 10am (I got into work around 9.15am) and so was left staring at my screen twiddling my thumbs by 10.15am. By 11am, I had finish reading through all the stories I could click on from news.com.au and smh.com.au.

The queue was still empty. All my project work was either finished, or waiting on other people. I was bored. So bored in fact, I sent an email to a friend who happened to work in the same company (but different department) to asked him to 'break' something. Anything.

I needed something to break in order to have something for me to work on.

I know. Its sad.

On Wednesday morning, I got what I wished for. Server offline - unable to boot up -- busted.

Cool. (I thought) .... Got the HDD for that server replaced, re-imaged server remotely (server was in Melbourne), took me a day given that the current documentation on the process is practically non-existent. The same idiot also changed the administrator login & password on the image. No one in the office has any idea what that username or password might be. So I spent a bit of time cracking the local admin password as well. (See.. http://www.listsoft.com/?id=12378) -- amazing stuff! **Salutes person who wrote that proggy.

I even re-did the documentation, so that the next person who comes along wont have to go through what I did to get the re-imaging software to work remotely.

Left after 6pm. It had been a super productive day. Happy.

Thursday morning, today. Get into office, open MS Outlook, notice that mailbox has been bombarded by mail alerts saying same server crashed overnight and still not responding. Fook. Case queue also starting to pile up. Foook.

Never wish for something you don't want.

You must be dreaming..

Another Friday morning meeting. Another morning wasted.

I felt like ripping up the requirements in front of the client and telling him that he is out of his mind.

This is what happens when you let a technically competent client design his own solution.

It never works.

Let me explain. The client who is technically competent. Is also a person who only knows an overview of the technology employed by this particular project. Believe it or not, no I.T. professional knows everything there is to know in the Information Technology field (Yes, I actually said that). Most of us just specialise in one area (e.g. systems, networking, database.. etc) and know bits and pieces about everything else in the field. I have yet to see an I.T. Guru in all areas of I.T.

So, we have let a person who only knows what he has read in a whitepaper of the technology, modify the proposed solution into his 'requirements' for the new system.

And he wonders why things are not working the way he expects it to work.

The usual problem when working with Techos who think they have the answer to everything.

Except this time, we have gone so far away from the proposed solution it is no longer funny.

I've told him that what he wants isn't going to be possible. He is dreaming. The technology just cannot handle what he is expecting it to do. I've left the Project Manager to do the 'damage-control' business talks. Meanwhile, I've got to go think of a new solution which can hopefully encompass some of his requirements. Or I could just recommend we go back to the original solution and tell him to go stuff himself.

If only I didn't need to pay for rent and food...

Start at 9 ?

For the past two weeks my manager and me have both been trying to out-do each other by seeing who would be the later one to come into the office.

It is all unintentional.

Really depends on what time I end up getting out of my bed and at what time I decide to get out of the apartment to go catch my train to work. And then after that, if I decide to go get some breakfast before showing up in the office.

Call it lazy. Unmotivated. Whatever you want... But it is totally unintentional.

If it was intentional, I think my manager would have had some private words with me already. But then I guess he should also be the one to set a good example. Can't exactly preach one thing and do something totally opposite. I mean, wouldn't that be just having double standards ?!

Then again, maybe he did hint something before I left yesterday. 'Lets both try to come in at 9am yeh ?'. He said in his thick scottish accent. 'Sure.. Your on'. Came my confident reply.

So I decided against watching an extra episode of Boston Legal or House on my laptop before getting out to the train station this morning. Thus arriving in the office at exactly 9.03am.

Except when I got here, my manager's desk was empty. 10 minutes later now, and it is still empty.

I think I've been had.

So True...

(Click on image to view larger version)

That is so true... only with the current project I am working on.. the client expected something like this:

(Image courtesy of www.usa.worldweb.com)

Why?

Why is it always the weekend when you have to go to work that they decide to do track works on your trainline ?

And why is it also that on the weekend that I have to go into work, when the building maintenence people decide to service the airconditioning systems !?

And why must they do it on a sweltering-35°C-Day ?!?

Never tell your clients...

I called up my vendor today to ask about the status of one of my cases. And also if they required any additional information. The usual diagnostic reports and whatnot.

But first I asked to be put through to my case's owner. (TSE or technical support engineer).

The guy on the end of the phone let me know that the TSE I was looking for was currently on another call. He asked me if I wanted to hold for a moment, and he would go and see if he could forward my call onto him. I told him to do so.

A moment later, the guy came back on the phone and told me that my TSE was currently busy on another case (apparently some other guy's system had died) and that he would give me a call back later today or tomorrow.

To all the tech support guys out there, please do NOT tell your clients, that the person responsible for your case is too busy to look\work on the case for you.

You can tell them that you are still currently working on the case..... Trying to reproduce the problem... Testing the new patch... ANYTHING .... Lie to them... But never say that you were busy on someone else's case.

No client is going to be happy to hear that.

I certainly wasn't. And so I asked to talk to their Manager instead.

I told the manager that my case issue was lodged a full week ago, and even though its not a priority 1, it still is important to us that it gets looked at. I also told him that since we have provided all the requested information already and that the current TSE was too busy, I would like the case esculated to the next support level.

The manager was very.... obliging. He was 'ok' with everything I asked for, and promised to keep me updated on the current case status. Although I know that once he put the phone down, he would be calling me a B**ch. How do I know this ? I'm a support engineer too. And I know I'll call her names too if I was on the receiving end of the phonecall today.

Bravo Alpha Charlies...

One thing I have realised since joining this team is that everyone here speaks in the International Radio Operator's Alphabet.

Alpha Bravo Charlie Delta.... etc

(*They use this in an attempt to make ourselves clearer over the phone when trying to convey server names or directories).

Which is all fine, except I don't know it.

So on a typical day when I pick up a request call over the phone, I will hear this guy starting off by saying... "User is somehow getting denied access to November, Sierra, Alpha, Uniform, Victor, Charlie ..... ". While I am on the other end trying to frantically write down what he is referring to by decoding his words. Sierra caught me a couple of times, for some reason, I thought Sierra started with a 'C'.

It kinda feels like I'm working in a airport control radio tower... :P

Previously when I've tried to spell out stuff over the phone, I just say something along the lines of "M for Monica, P for Peter, J for Jeremy, B for Boy, U for Umbrella....". It seemed so much easier when I was just using my friend's names and throwing in the occasional 'B for Boy'. But now, the people around me doesn't seem to understand if I say 'P for Peter' instead of just 'Papa'.

But as they say, When in Rome, do as the Romans do. So if you would excuse me for now, I've got to go and learn some alphabets.

Sunday @ Work

Just spent my whole Sunday at work. Which somehow didn't turn out as bad as I thought. I had another 5 of my colleagues with me. All of us just decided to take our laptops (instead of sitting in our separate cubicals) and shut ourselves inside a meeting room to work through the implementation together.

From 10am till 6pm. I never felt or been so productive on a Sunday.

It actually went pretty well. Although somehow 'The Bank' didn't seem to recognise that anyone would be working on weekends, so there was no airconditioning available. Which with 6 laptops.. 6 perspiring bodies..all in one small meeting room...don't not make a nice combination.

My tummy had been growling the whole day (happens when I skip breakfast). At first no one seem to pay much attention. But after awhile, the noise was quite audible. And the guys had started to chuckle everytime my stomach decided to complain. So on one of my trips out to get some fresh air, I went to the vending machine in the corner of the kitchen, and paid $1.60 for a packet of chips.

I probably finished about 3/4 of my chips before everyone else decided to go out to get some lunch. But ended up forgetting about it after lunch. However, I did remember to take it with me when we were packing everything up. So atleast I can finish it on the train.

That was when I noticed this.

17 Nov 2006 ??????????

Today is the 19th ~~!!!! I'VE BEEN MUNCHING ON OVERDUE CHIPS THE WHOLE MORNING ?!?!?!

Worse thing is I got it from my company's vending machine ! OMG.. How long have I been munching on expired goods from there?!

Grrrr... I probably still would have eaten it ... afterall its only really two days overdue (and I was extremely hungry). Btw...Does that mean its been sitting in that vending machine for over a year ?! I mean they can't possibly be loading the vending machines up with expired goods !? I feel like the time when I was eating this piece of bread which turned out to be all green and furry on the other 'un-buttered' side.

But all the other junk food I've been eating from there...ahhh... Don't even want to think about it anymore.. :(

New job.. New people

I've been so busy at work these past few days, I only realised the Melbourne Cup was today when my ex-manager called me to check up on me - and if I could, then to meet up with the old gang at the Martin Place TAB. (A temporary location thats setup so the city people can get their horse bets in). But unfortunately I couldn't make it.

I haven't even heard of the horse that won the Melbourne Cup today. And I'm not so sure the rest of Australia had either before today - Japanese horses took out 1st and 2nd placings. Grats to them... it was a great race to the finish ... :)

I guess I'm still having issues adjusting to my new environment. The people are really nice. However, I get this feeling that I'm not sure what I am supposed to do here... yet. There are heaps of projects (or so I've been told). But I guess I still have a lot of paper work to get through before I get to touch anything.

I did get to go to the data center today. Its the biggest I've been in so far. I had to go in to change a failed disk. But the disk enclosure that was dispatched didn't fit into the array. (Yes... wtf ?!?). Reminds me of my first day in my previous job when the vendor sent some rail kits that didn't fit into the racks. It was about 1-2cm too short. It doesn't sound like alot, but given that its steel.. you can't really stretch it...so even 0.5cm is alot.

Just my luck I guess.. hope they send the right parts out tomorrow. (fingers X)