Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Would You Benchmark an IT/IS Department?

Cliff posted more than 7 years ago | from the what-metrics-does-the-bar-measure dept.

IT 144

ferretworks asks: "Our IS/IT department has been asked by our CEO to find a way to benchmark ourselves against IS/IT departments from other companies with similar technologies (none specific). This sounds like an innocent enough request, but diving into it has made me realize that this is, not necessarily undiscovered country, but a desolate one and rarely visited. So, my poll to the community is: In your Opinion, what is best way to benchmark an IS/IT department and what categories/sub categories would you base your judgment and ratings on?"

cancel ×

144 comments

Talk about internal benefits first (5, Insightful)

evileyetmc (977519) | more than 7 years ago | (#18965333)

I think the first facet you should look for is uptime / accurate data rates (eg. 1% lost data etc). Beyond that, while being nearly a crapshoot, I think the satisfaction that the rest of your company is getting from your department is paramount - perhaps having a anonymous survey given company-wide to see how you're doing. Also, your upper managers may want to hear numbers such as ROI as well as IT costs as a percentage of revenue brought in...maybe even what revenue would be lost without the department.

A warning about surveys... (1)

MrTrick (673182) | more than 7 years ago | (#18965603)

I'm highly dissatisfied with the IT department in my workplace.

They recently did an anonymous satisfaction survey across the 4000 employees in the company.

The survey was USELESS!
- All of the questions were True/False,
- Many of the questions were not even applicable to me, (but being T/F, I couldn't put a NA answer)
- There was no way to adequately express my dissatisfaction on most issues.
- Many questions were ambiguous.

If a survey is done...
1. Get someone else to write it, NOT just the IT department.
2. Grab some random employees to EVALUATE the survey before it's sent out, and see if it could be improved.
3. When the results have come in, publish them and any conclusions drawn, inviting anyone who disagrees to anonymously comment upon them.

I'm sure my IT department is erroneously patting themselves on the back for their interpretation of the results... don't let it happen to you!

09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c0 (-1, Troll)

Jeremiah Cornelius (137) | more than 7 years ago | (#18966091)

09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c0

Re:09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c0 (0)

Anonymous Coward | more than 7 years ago | (#18966617)

Wow you are really sticking it to the man aren't you? Keep at it, rebel hero!

Re:A warning about surveys... (1)

EmperorKagato (689705) | more than 7 years ago | (#18966497)

Careful, some surveys are not done or controlled by the IT department even if the subject matter is the IT department. This happened to our department and we weren't moved by the development of survey.

Do what the CEO does.... (1)

passthecrackpipe (598773) | more than 7 years ago | (#18968715)

1.) Make a random survey, asking insightful questions 2.) Have several emails go around the company stating how important it is for everybodies voice to be heard 3.) Pull metrics that make you look good out of your ass and use those instead of the answers 4.) profit!!

Get rid of the Lunix d00dz (-1, Troll)

Anonymous Coward | more than 7 years ago | (#18966375)

One sure way to kill an IT shop is to have teh Lunix d00dz wasting time whining about how they can't figure out how Microsoft stuff works, and aren't professional enough to learn. They generally do this when they take a break from downloading pirated music or software over some peer-to-peer network.

Make sure the people you hire understand how Active Directory works (or at least understand how to do things in Windows), and know how Group Policies are used. MS certifications are generally a good metric, but there are also good people without certs.

Also, make certifications mandatory. Yes, you run the risk of having people leave, but IMO happy employees don't look for other jobs, so make sure your place is also a good place to work.

Re:Talk about internal benefits first (4, Interesting)

Mattcelt (454751) | more than 7 years ago | (#18971023)

This is actually a HUGE industry.

Forrester, Gartner, and IDG all offer advanced comparison data from industry surveys against which you can measure your own company. (Help desk costs per call, fully-loaded employee per hour, etc.)

Then you implement some sort of metric to use in the comparison. ITIL, COBIT, ISO 17799, and a host of others are available as frameworks you can use, or you can design your own. So you start taking measurements, compile the metrics, and compare.

[How much does a password reset cost? How much does it cost to terminate a sysadmin? How many staff hours are required to clean a virus infection on one machine? On all machines? &c, &c.]

I think you'll be hard-pressed to find a Fortune 1000 company that doesn't use some form of this - it's how most companies compare themselves against the industry.

Hope that helps.

Downtime (4, Insightful)

Conception (212279) | more than 7 years ago | (#18965353)

One metric you could use is downtime/loss of work due to IT. This could be because you don't do backups right, to you don't have a test/production setup, to you upgraded to vista without training first, etc etc. Though you'd be reporting stuff you do badly, you can use this for a lot of justification.

"Email was down about 25 times for up to a day over the last year. This is because we don't have to budget to buy a redundant system. If you give us more money, we can increase uptime." etc etc.

Re:Downtime (1)

Marxist Hacker 42 (638312) | more than 7 years ago | (#18965781)

Related to this is average time to trouble ticket resolution- assuming you use a trouble ticket system to keep track.

Re:Downtime (1)

rah1420 (234198) | more than 7 years ago | (#18967405)

As soon as you start measuring ticket resolution time, you won't believe how fast your call center people will find creative ways to close tickets without actually resolving problems. They'll close a ticket when it gets bumped from functionary to functionary, for example.

"See, we close tickets 50% faster than before!" But the work gets done at the same pace.

Don't do it.

Worst. Idea. Ever. (2, Insightful)

mcmonkey (96054) | more than 7 years ago | (#18966673)

One metric you could use is downtime/loss of work due to IT. This could be because you don't do backups right, to you don't have a test/production setup, to you upgraded to vista without training first, etc etc. Though you'd be reporting stuff you do badly, you can use this for a lot of justification.

Why would you do such a thing to yourself? With those metrics, the absolute very best you could do is, 'we didn't cost the company anything...other than salaries and resources.'

Why not use metrics/goals that actually help justify your job? Why not use metrics to show how much IT is adding to the bottom line? Seriously.

BTW, before you go out and waste resources on another system you don't understand, how about learning how to handle what you have? (Hint: It's not downtown; it's 'scheduled maintenance.')

Re:Worst. Idea. Ever. (1)

Architect_sasyr (938685) | more than 7 years ago | (#18967263)

Yes... we usually schedule the "maintenance" 5 minutes after we realise we're doing "maintenance"...

Re:Worst. Idea. Ever. (1)

TopShelf (92521) | more than 7 years ago | (#18968133)

Nobody said downtime was the only metric to use, only that it is one that has value.

Re:Worst. Idea. Ever. (1)

DougWebb (178910) | more than 7 years ago | (#18970319)

IT doesn't add anything to the bottom line. IT takes away from the top line, just like HR, management, administrative positions, the mailroom, etc.

IT should be measured three ways (from the CEO's perspective):

1) To what extent is the IT department reducing risks to the business? This covers backups, software updates, choice of software and hardware, ticket resolution (NOT closing) times, etc. All of these should be presented as "To prevent X, we did Y."

2) To what extent does IT improve the productivity of other departments? This includes any custom software that is built, automation of various systems (like trouble ticket reporting and processing) and so on. A survey would be helpful here to get quotes from other departments saying how Project Q has improved their productivity.

3) How has IT measured and improved its own cost efficency, and reduced the cost of other departments? This is where you trumpet any transitions to OSS, if that went well. Mostly you just need to justify the IT department's cost. After 1 and 2, that should be easier to do.

No benchmarks! (2, Funny)

MeanMF (631837) | more than 7 years ago | (#18965379)

If there are benchmarks, then the terrorists win.

Futuremark! (1)

MeanE (469971) | more than 7 years ago | (#18965381)

I think Futuremark has a benchmark for that! Although I hear some IT/IS Department are hiring custom techs just to get high results on it.

Re:Futuremark! (1)

mobby_6kl (668092) | more than 7 years ago | (#18965777)

The trick is to set the benchmark to the most demanding mode and run it at least a few dozen times to ensure the most accurate results possible.

Consulting Firm. (3, Funny)

m0rph3us0 (549631) | more than 7 years ago | (#18965385)

I would start a consulting firm, get the CEO to hire your consulting firm. Spend a lot of time compiling a bunch of numbers, then because the other companies won't want their data revealed by name sort them into averages based on Fortune 20, Fortune 100, Fortune 500.

Make up an "average" for these three sets in which your company does better in most metrics, take the $250,000 you got from this consulting gig and live on it while you go around with your initial report selling it to other companies.

This idea is so stupid and useless that only a consulting firm would offer this service.

Re:Consulting Firm. (1)

m0rph3us0 (549631) | more than 7 years ago | (#18965405)

I forgot, once the report is done, do a white paper or case-study on how much money they saved after changed after firing key people in the first month. Neglect to mention that the company crashed a burned 2 years later.

Re:Consulting Firm. (1, Interesting)

Anonymous Coward | more than 7 years ago | (#18966351)

Are you trying to solve a problem or make money?
Both are fine with me.

However, $250K is low based on my personal experience doing exactly what the original poster asked. That's only $250/hr for 6 months. See the common rates http://www.ticker.computerjobs.com/ [computerjobs.com] Be certain to say how Wireless can help the company become more efficient if properly secured, which you'll help them architect over the following 6 months.

Re:Consulting Firm. (1)

walt-sjc (145127) | more than 7 years ago | (#18969677)

That's enough out of you, Dogbert!

TPS Reports (2, Funny)

Joe The Dragon (967727) | more than 7 years ago | (#18965387)

did you get that memo?

Consumer Surplus (1)

ect5150 (700619) | more than 7 years ago | (#18965437)

One trend out there is if an IT project doesn't really to appear to have lowered costs, didn't really improve profits, and didn't really improve productivity, what was the over-all point? Was it a failure or is tehre some other measurement of success?

But another measure of success is to measure the increase in Consumer Surplus the IT project/dept adds to the firm. Consumer surplus is one possible way to measure a competitive advantage (because it measures the increased value to your customers).

Estimating consumer surplus is another story though...

Just an idea

http://en.wikipedia.org/wiki/Consumer_surplus [wikipedia.org]

Re:Consumer Surplus (0)

Anonymous Coward | more than 7 years ago | (#18965723)

One trend out there is if an IT project doesn't really to appear to have lowered costs, didn't really improve profits, and didn't really improve productivity, what was the over-all point? Was it a failure or is tehre some other measurement of success?

You missed the big reason: legal & regulatory compliance.

Re:Consumer Surplus (1)

jafiwam (310805) | more than 7 years ago | (#18966837)

If yours is like my company, IT projects dont do any of those things, but rather can be catagorized is "filled the gap between the pack of lies the sales department created and what the developers actually said it would do".

Re:Consumer Surplus (0)

Anonymous Coward | more than 7 years ago | (#18969165)

you've just bastardised the concept of consumer surplus. consumer surplus is when a person is willing to pay above market price. the price difference from what he is willing to pay and what he pays is the extra satisfaction the person gets aka consumer surplus.

you are fucked (0)

Anonymous Coward | more than 7 years ago | (#18965471)

you will never get any numbers from any other company to benchmark against.

your best bet is to chart your own org over a few years so you can show how much improvement everyone has made. this has the added benefit of being a great weapon to reverse any decisions you didn't agree with or to attack your enemies, just show that the numbers got worse. for example, if you are pissed they outsourced your support department, show that the support resolution numbers tanked. if you hate the network engineer's manager show that his department has been getting worse on a few measurements.

Survey the users (3, Insightful)

Kohath (38547) | more than 7 years ago | (#18965513)

I would survey the users.

Re:Survey the users (2, Insightful)

wild_berry (448019) | more than 7 years ago | (#18969795)

When you're surveying the users, be careful: remember that the squeaky wheel gets the grease -- in these surveys the people who have nothing to complain about rarely say anything. Do it with the intent of making it easy to propose fixes for issues raised and also to present the data in terms that flatter you and enable you to do a better job. The game should be: work with management, and work with your internal clients.

It can't hurt to have your report include a count of preventative maintenance issues and a provisional costs saved, including parts, downtime (possibly scored as a proportion of quartely working hours / quaterly revenue) and the cost of data loss. This last one will require that you know what's going on within the company (go and chat about what each project lead is doing: flatter them by listening), who's tendering contracts or running an important project, so you can weight the value of their data in your backup system on top of the standard value of e-mail and filestore.

my 2 cents... (2, Insightful)

asdef (261823) | more than 7 years ago | (#18965539)

Having worked on both sides of the IT vs Business user / programmer fence, I think any measure of the success of IT needs to include some form of customer satisfaction. This is important because IT's customers tend to be internal and most customer satisfaction queries are focused on external customers.

I've seen battle lines drawn between IT and everyone else, and nobody wins because energies are focused on battling with the other side instead of finding ways to help the company make money. While I understand that IT's responsibility is to the infrastructure, it should be done in a way that makes it easy for their customers (the rest of the company) to do their jobs quickly and efficiently.

Cutomer Feedback (0)

Anonymous Coward | more than 7 years ago | (#18965591)

Take a poll of the people who use the IT department. What do they think?

Other metrics are just silly and are going to have the IT depart trying to do things not their job, which is to make everyone else work more productively.

How Would You Benchmark an IT/IS Department? (1)

Anne_Nonymous (313852) | more than 7 years ago | (#18965601)

>> How Would You Benchmark an IT/IS Department?

1. Coversheets per TPS report
2. PHBs per employee
3. Salesmen per server

Higher is better for all metrics.

Re:How Would You Benchmark an IT/IS Department? (1)

HomelessInLaJolla (1026842) | more than 7 years ago | (#18965835)

this is, not necessarily undiscovered country, but a desolate one and rarely visited.
It is a perfect opportunity, then, to define the metrics yourself and present your ideas as industry leading and groundbreaking. Without making the time-suck obvious, endeavor to define as many possible metrics as you can. For your final report pick only those metrics which portray your department in medium to positive light so that you can write a full report on how other departments across the nation are doing better, and how other departments across the nation are doing worse. Pick and choose the topic areas carefully such that the departments which are doing better are doing better in areas which you would like to expand--if you can show that they are doing better then you can show where you need greater funding and support. Pick and choose the topic areas so that the departments which are doing worse are doing worse in areas which you have no interest. If those areas can be defined as troublesome areas, or those departments can be shown to be performing poorly at those tasks, then you can make a viable case not to enter into those applications.

Of course you will need to balance your report so that it panders to the personal likes and dislikes of the people to whom you will be presenting. Never fail to keep your target audience in mind.

The data can be made to say anything you want it to. Take the time to write a full-length report which leaves you plenty of material to keep expounding and writing further progress reports on a semi-annual basis. If you look far enough ahead while preparing this initial report then you can secure your job, the expansion of your department, and your own advancement for years to come. If you define the metrics properly and present them shrewdly enough you may even position yourself to gather larger corporate support and you may even find yourself as a groundbreaking leader in your industry.

I daresay this is how the global intelligence community operates.

Use a real standard (0)

Anonymous Coward | more than 7 years ago | (#18965625)

We do consulting for a lot of major companies, and we try to always give reports based on standards such as the ISO 17799 standard for IT management practices and the NIST 800-30 standard for risk assessments. Both of these should give you actionable items to improve your practices and reduce your risk profile.

(Plug: Or we can do this for you, web.interhack.com)

Easy (0)

Anonymous Coward | more than 7 years ago | (#18965651)

I would use bonnie++, hdparm tends not to flush buffers and so gives inaccurate results.

Oh, wait...

Get Ready to be Outsourced (2, Insightful)

desertfool (21262) | more than 7 years ago | (#18965675)

If you are being asked to do that, then the PHB's are looking for a cheaper option. Work on your resume and get out NOW. Otherwise, you will training your replacements.

Been there, done that, don't have the t-shirt.

Seriously, they are looking at cheaping labor costs in India/China/Godknowswhat3rdworldhellhole to send your jobs to. If they are asking for a lot of procedures on your work and maps on how it gets done, leave faster.

How do you benchmark other Operations departments? (3, Insightful)

bergeron76 (176351) | more than 7 years ago | (#18965685)

Operations(IT) isn't like sales. It's not exactly easy to quantify achievements, etc. As long as your teams hit their deadlines within the allotted time frames, your department is doing better than most.

There's only one absolute measure (1)

JavaRob (28971) | more than 7 years ago | (#18970013)

Everything else is more or less made up -- you can calculate whatever stats make you look good.
The real test?

Two words: sack race.

Not simple but doable (1)

McNihil (612243) | more than 7 years ago | (#18965695)

The less the IT department is noticed that it takes care of things behind the scenes the better. It should be perfectly fluid. They shall fix problems BEFORE they happen. If problems happen out of the blue there should be fast replacements/workarounds that have minimum impact on users and the entire systems.

And remember EVERY second counts!

Gartner (or equivalent) (1)

pci (13339) | more than 7 years ago | (#18965713)

If you have a little money,ok more than a little, to come up the results from a third party call Gartner (or another big firm) and ask to participate in a study. They will find other companies your size in and out of your industry to compare you against. They use a common and repeatable criteria that you can use in the future again.

We had one done in the IT department I work in, and while we came in above average in one or two areas it gave us a place to start from in proving we don't just throw money away.

We also directly bill other groups in the company for our services, so they understand where their money goes and how much extra some of there stupid requests cost.

Re:Gartner (or equivalent) (0)

Anonymous Coward | more than 7 years ago | (#18966489)

So did we, and we are no longer doing the work. It got sent offshore.

Service Level Agreements (1)

toolo (142169) | more than 7 years ago | (#18965745)

Tracking by uptime, as a lot of other posts describe, simply is not enough. If you want to compare success by uptime, your management might as well just outsource your department to a datacenter company.

A Service Level Agreement can truly monitor your performance as a customer service organization. This idea would measure the soft side of IT support as well as uptime. Also if you do uptime measures, monitor the services, not just ping times. That is simply a waste.

HEAT is a very expensive system but can track service levels, and many open source or cheap packages can monitor uptimes.

A quick google has an example here [duke.edu] .

Re:Service Level Agreements (1)

nologin (256407) | more than 7 years ago | (#18968027)

I have to agree here. I have worked as a technician and as a manager within IT/IS departments and I certainly recommend that every IT department understands the basics of a Service Level Agreement.

If your IT/IS Department doesn't have a Service Level Agreement, then benchmarks are generally irrelevant as they can be easily taken out of context to make your department look incompetent or as a waste of money.

The good thing about an SLA:

1. It basically defines what your group is responsible for. It's the perfect tool to use with your manager when he/she wonders what you do there.

2. It defines what resources and budget you will get in order to fulfill the responsibilities you are given.

3. It helps protect your group from being blamed for issues that are out of your control, such as a supplier who cannot provide the replacement parts for a business critical system. Of course, your IT department should also have SLAs with your suppliers in order to assist your group in these situations. You need to make sure that such issues, and the conditions that trigger them, are contained with the SLA.

4. It defines the benchmarks as to which your group will be evaluated.

The bad thing about an SLA:

1. Once your SLA is agreed upon, you now have to deliver upon what was agreed. So, if the task your were doing is applicable within your SLA, get to it and stop reading Slashdot you fool!

2. Don't think that management will not pressure you continuously with an SLA in hand. Some people make it their work to haggle with the other parties in an SLA, so don't exepct a blanket of protection in this case. Make it flexible enough to adjust with the company in question, but don't let them roll over you with unreasonable requests.

3. Tracking and reporting. Get ready to do a lot of this stuff, especially with those people who haggle you about the SLA or try to cheat you out of the terms (especially when it comes to money and/or resources). This can save your ass.

4. People do lie, cheat and steal... Just because you have a SLA in place, don't expect this behavior to change with the parties in question.

Once you have your SLA, you should then have your benchmarks. Good luck!

Sounds like management... (1)

NinjaTariq (1034260) | more than 7 years ago | (#18965783)

Management always want things like this, unless it is a subsidiary its unlikely that you would get any meaningful data you could compare against in another company. Unless they agree to benchmark following the same rules and be completely honest with you... Who would tell you if they had a bad IT department?

A managers real question is how much money have you saved me or how much easier has my life become because of the IT department.

Someone suggested asking the users, thats not a bad idea... A simple questionaire on the corporate site (internal or external) would be good... Say 10 questions marked on a scale of 1-10, come up with an average of both including and excluding 1's and 10's on the scale. Your ones with low average scores are where you need to improve, high scores you rock and the overall average is where you are. Convince other companies to do the same thing, you have your comparison. Compare your score out of 100. The average without 1's and 10's would be more meaningful, people who put 1's and 10's either aren't thinking about it, or are trying to skew the results. Get the members of the IT department to do the same questions, comparing that might be interesting.

Self evaluation is another management favourite, but i have always hated that... But that would cover downtime or use of budget and things like that... Budget is a bad thing though, show you have saved money and your budget is cut

Have your IT department visit other companies...

Here's your benchmark... (4, Insightful)

Cervantes (612861) | more than 7 years ago | (#18965791)

I got a benchmark for ya. "Are things running well?" Yes? Yes they are? Then shut the fuck up and stop asking me to waste time comparing myself to other people.
(This method of managerial interaction is derived from the groundbreaking book "Shut the fuck up" by Dr Denis Leary)

Seriously, it's not an "innocent" request. It never is. First, they just want to know how you compare. Then, they want to know why you're not the absolute bestest in the whole universe. Then they compare you against departments with fewer people. "Hmm, seems we could get comparable results with 2 fewer bodies". Then each employee is evaluating their specific performance trying to justify their job. Or you're filling out "Time code sheets" to tell them what you do in each 6 minute block of time. Pretty soon you're hearing about their buddy Steve, who has his entire office and web presence run by one part time summer intern who happily works for $6/hour plus tips.

Evaluate your performance based on internal expectations. You want great uptime on the web server? Why, we were only down for 2 hours last month. Compared to previous months, that's great! You want quick response to employee problems? Our average response time for properly filed tickets was down %4 compared to last quarter. That's the way you evaluate. If you start giving these fuckers examples of how other companies are doing, they'll cherry-pick and start challenging you to match. "Why, there's a place in Wisconsin that only has 1 tech covering 400 people! I don't know what this 'Wyse terminal' thing is, but surely if just one guy can cover all those people, we're overstaffed here!"

I know it seems like it could be good. "Why, we're outperforming most other companies, surely they'll see this and use it as criteria to give us better raises/more benefits/better perks". NO. NO, they won't. Just NO. Come review time, you could be leading 90% of the field, and they'll go on about the top 10% and how those guys earn less than you, so they shouldn't give you a raise. Or they'll go on about how other companies have their helpdesk techs do server support too, so they don't need your $80k server admin. If you're underpaid, and everyone else makes more than you, you can show them that too, and they'll nod in an interested fashion, they'll hum and haw, and they'll end up giving you some bullshit excuse about budget constraints. If you make more than other people, expect a pay freeze or cut. Even if fall in the middle of the bell curve, expect them to find a reason to lump you with the folks on the bottom of the curve.

Management doesn't understand you or what you do. They're scared of you. They want to control you and marginalize you so they can eliminate you. Don't let them, don't help them, don't give them any excuse. No matter how much you train them, or explain to them, or draw them little fucking flowcharts in Visio with pretty colours, what you do is still voodoo to them. You push keys on the magic box, typing 10x faster than they can, and you get it to do things they couldn't even dream of, and you do it really easily, and that makes them feel dumb. And just like stupid bullies on the playground, they want to get back at you, subconciously perhaps, but they still do. They want to sit there across from you, at their big fancy desks in their spotless shiny offices, because that's their one time to feel superior to you, to be in control of you, unlike all the other times when you're the magician who just makes miracles happen.

Don't fall for it. Rephrase their request, turn it around on them, avoid it at all costs. Does your accounting department have to compare themselves to other businesses? Do your managers have to go around finding out how other managers in other companies are doing? Fuck no. Don't let them treat you any different than any other department. They're just doing this because they don't understand what you do. Don't explain it to them, just give them numbers based on internal statistics so they can sit back and go "phew, IT is performing well, I'm happy".

Just remember folks, any time anyone in management opens their mouth, echo these words in your head:

"It's a trap!"

Re:Here's your benchmark... (2, Insightful)

EmperorKagato (689705) | more than 7 years ago | (#18966695)

Yet comparing yourself to others is how companies improve. This is how positive competition is created. This is how problems get fixed.

When I first started working at my company there were no solid metrics in place. We had a backup system not being tested. We had a NAS system growing out of control. What happens when we started placing metrics in place? Things begin to change.

I recall someone wondering why the director would never approve a project until I went to the director with numbers and history (DATA!) and a 6 month fight turned into a quick chat with the result in a decision.

Performance and time spent has to be measured because this is a cost to the company and yourself.

When the IT department performs terribly on it's help desk support you can end up alienating your customers - the company - and always make it feel that contacting help desk is a waste of time.

If some wise guy exec decides to make the quick decision to replace the entire department (although you outperform 70-90% of IT departments in the country) with outsourcing. They'll never find out the impact money/time wise until is too late. Now, you're out of a job yet you were the best.

Metrics can be your friend.

Re:Here's your benchmark... (2, Interesting)

bitty (91794) | more than 7 years ago | (#18966979)

It sounds more like your company benefited by following common sense procedure and industry standards, not comparing yourselves directly to other IT departments. Remember, the article submitter is talking about direct comparisons to other departments, not standard procedures or best practices.

No two companies are even remotely close to identical. Every company has different goals, different organizational structure, different programs they're using, different needs in general. The moment you start comparing your own department against other companies' IT departments, you lose the ability to adapt to your situation. Your decisions will be second guessed every step of the way by people that have no clue as to what you do or why you do it, and have no intention of even trying to understand. "XYZ Company doesn't do this, why do you want to?" "Why is this solution so expensive? XYZ Company did it for $XXX."

Never underestimate the power of a stupid idea in the head of an executive. Should some fool in the upper rungs decide to replace someone for any reason, nothing -- NOTHING -- will prevent that from happening. If they're worth their salt, they'll find another position elsewhere. If not, maybe they shouldn't have been there in the first place.

Re:Here's your benchmark... (1)

EmperorKagato (689705) | more than 7 years ago | (#18969141)

That same information can be vital especially if you're trying to get an idea on how much your project could cost the company.

Re:Here's your benchmark... (1)

Cervantes (612861) | more than 7 years ago | (#18967673)

Yes, Businesses compare themselves to other Businesses. This isn't a business, this is a department in a business, a service department at that. If the services are being provided in an acceptable fashion, that's what should count.

Additionally, even when comparisons are occurring, management isn't in a position to compare. They simply don't have the expertise. They can compare sales, or throughput, or numbers, but asking them to understand that a backup server with RAID and a DAT tape drive is fundamentally different than taking an old PC, slapping a big hard drive into it, and using a scheduled batch file to xcopy everything over... well, they might as well be asking to be fundamentally involved in your selection of a heart donor.

Re:Here's your benchmark... (1)

nacturation (646836) | more than 7 years ago | (#18967861)

I think this is an excellent opportunity for this guy. With that charge, he can go around on business trips around the world, spending a week at each competitive site to evaluate what their performance is like. Naturally, you'll want a large enough sample size and I think 52 companies ought to do it. One year later, having burned through half a million dollars of travel expenses, hand in your report.
 

Re:Here's your benchmark... (1)

Cervantes (612861) | more than 7 years ago | (#18968209)

As much as I dislike external comparisons, your ideas intrigue me, and I would like to subscribe to your newsletter...

Re:Here's your benchmark... (1)

EmperorKagato (689705) | more than 7 years ago | (#18969167)

There's numbers behind the differences in a RAID + DAT solution compared to a using random pc solution.

Re:Here's your benchmark... (1)

bitty (91794) | more than 7 years ago | (#18966721)

Parent beat me to the punch. It's very close what I was going to say, only said better. Excellent post.

Don't ever let them railroad you into that benchmark bullshit. It only creates trouble.

Re:Here's your benchmark... (2, Insightful)

Malfourmed (633699) | more than 7 years ago | (#18966907)

Does your accounting department have to compare themselves to other businesses? Do your managers have to go around finding out how other managers in other companies are doing? Fuck no.

In my experience, all functional areas are compared to other businesses. It's what benchmarking is all about and has been all the rage for at least twenty years now. IT is no exception. Sometimes it can even be useful! If you're doing well, then it can help demonstrate you're doing well. If not, then it can act as an incentive to improve things.

Sure, sometimes it can be used for political, even malicious, purposes; but in those instances the corporate culture is often dysfunctional enough that that's just an excuse to screw you. If it's not benchmarking, it'll be something else.

If you make more than other people, expect a pay freeze or cut.

If you make more than other people, then surely you'd be able to justify that? Maybe by benchmarking your IT department's performance against others?

Even if fall in the middle of the bell curve, expect them to find a reason to lump you with the folks on the bottom of the curve.

In my experience if you fall in the middle of the bell curve, they'll ask the question "Why aren't we ahead of the curve?" The answer can lead to a productive discussion about what exactly the business needs to invest in in order to make IT performance even better.

"Are things running well?" Yes? Yes they are? Then shut the fuck up" [...] Management doesn't understand you or what you do. They're scared of you. They want to control you and marginalize you so they can eliminate you.

And this embodies the attitude that wants to maintain the priesthood at all cost - keep IT mysterious and scary, rather than enlightening the business ... cause that way it makes it easier to justify our toys and inefficiencies.

Don't let them, don't help them, don't give them any excuse [...] avoid it at all costs.

And update your CV while you're doing that, cause you'll be needing it soon.

Re:Here's your benchmark... (1)

Cervantes (612861) | more than 7 years ago | (#18967869)

"Are things running well?" Yes? Yes they are? Then shut the fuck up" [...] Management doesn't understand you or what you do. They're scared of you. They want to control you and marginalize you so they can eliminate you.
And this embodies the attitude that wants to maintain the priesthood at all cost - keep IT mysterious and scary, rather than enlightening the business ... cause that way it makes it easier to justify our toys and inefficiencies.
Most of us have spent many, many years learning the ins and outs and the finicky little details of IT. You simply can't communicate that properly to someone who's spent their life focusing on a different area. Almost every time I explain a process to a manager, I explain "this is how things go when they go well. If something goes wrong, it throws the whole process out the window". And then, when something goes wrong, they come to me...
Them- "I don't understand, you said this usually only takes a day...."
Me-"Well, yes, but I also explained that when things go wrong, it can take longer, and something has gone wrong"
Them- "I don't understand what has gone wrong, but surely it can't be that bad. I expect you'll have this up and running in a few hours"
Me- "Well, I don't know the problem yet, so I can't guess at a time of repair..."
Them- "Well, if it only takes you a day to do this normally, it shouldn't take you that much longer if something goes wrong... have it up by noon".

There are variations of the conversation, but they're all the same. They don't understand what's happening or how it's done, and they apply their limited knowledge in ways that make sense in the business world, but not in the IT world. And then they get upset when it doesn't work that way.

Don't let them, don't help them, don't give them any excuse [...] avoid it at all costs.
And update your CV while you're doing that, cause you'll be needing it soon.
I've been at my current job for going on 6 years. I have a great working agreement with my manager. He has expectations, based on internal requirements. Uptime, response time, standard SLA stuff. I either make it, or give him a reasonable reason why I didn't. He doesn't try to understand the techie stuff, he accepts my word when I tell him how long something is going to take, and if it turns out to take longer, I tell him "The -tech- in the -tech- is broken, I'll need an extra X hours to do it right", and he says "OK".
He doesn't try to benchmark me on how long it would take another person to do -tech-, he doesn't keep a frickin graph of -tech- response times and try to shoehorn me into it... I tell him how long it's going to take, and he trusts my expertise. If there's a business reason that he needs it sooner, he tells me, and we negotiate what I'm going to put on the back burner to get it done. If it simply can't be done for technical reasons, I tell him, and again, he accepts my word.
This is the way things should be. If he was comparing what I told him to a list of reports of unrelated circumstances and trying to compare them, even though he doesn't understand why they're not the same... that's just a disaster waiting to happen.

-------
Rather than quoting all the pay stuff, I'll just say that most of my experience has been with Fortune 500 companies, and they're mostly interested in the bottom line, not in paying their people well or being industry-leading. If you're doing the job they need and they can get away with paying you 10k less than average, they will. If you're getting 10k more than average, and they can get someone to come in and do it for 10k less than average, they'll do it. It's sad, and I would be very happy to be somewhere that wouldn't, but it's the way things run. Money is everything, if they can get moderately acceptable performance for a lot less money, they'll do it.

What a load of crock. (2, Insightful)

dustpuppy (5260) | more than 7 years ago | (#18967401)

The sign of an incompetent department/person is one who doesn't want to be benchmarked.

Look at it this way:
If you are better than the rest, you should make sure everyone knows about it and ask for a payrise.
If you are doing worse than others, then you better improve.

It's the latter situation that most people such as the parent end up fearing (and hence ranting about). Simply, they are scared to change their current work practices - they are content doing a mediocre job. They are the sort who spend enormous effort trying to maintain the status quo whilst the rest of the world improves around them ... then one day they wonder why they are obsolete/getting offshored/getting downsized and then they bitterly complain about their situation and how unfair it is. If you have ever read the book 'Who moved my cheese', you will recognise this position.

Re:What a load of crock. (1)

Cervantes (612861) | more than 7 years ago | (#18968077)

The sign of an incompetent department/person is one who doesn't want to be benchmarked.
Not quite. An incompetent person wouldn't want any guidelines or deliverables at all. Nothing to live up to, nothing to fail at. I've got no problem having deliverables. If my boss was even moderately technically competent, I might even trust him to make some limited judgements in comparing things to outside situations.
But most managers aren't. Most managers have no idea why it would take longer to install and configure Win2k Server than it does to run the Recovery Disk on their HP at home. That's why internal numbers are so much better. Having a server uptime of 99.7 as a goal, and meeting it, is easy for everyone to understand. Managers looking at a report and seeing that banks and ebay and such have 99.9, and then wondering why ours isn't the same... that's when disaster strikes.

As for job performance... again, I agree with you, people who don't want any guidelines are the ones who want to hide their incompetence. I'm happy to not be in that category. We have an internal SLA that gets lived up to, and occasionally someone pipes up and asks "Hey, how does that compare to other places?" and I give them an answer. I don't want them looking it up themselves because they don't understand the differences in situations, but I give them an honest answer, and they trust my response. If a department becomes dissatisfied, we review the SLA and change as necessary.
If I had to have a meeting every time some manager went for lunch with his buddy and found out that THEY get a visit from IT within 5 minutes of their call, to explain why that isn't going to happen here... I'd never get anything done, and equally as importantly, the frame of reference for management would change.
Right now, I have numbers to meet, and people to keep happy. If I do both, then things are good, and if someone comes in with a comparison, it's an improvement to an already good process.
If we went with constantly comparing things to outside sources, that changes the frame of comparison. Suddenly, it's "Things are better elsewhere, why aren't they better here?". And no matter how good you run things, there's always somebody doing it better, or doing it cheaper. And as much as I'd like to believe that management would go "Hey, we're running in the top 20% of all these stats, that's great, keep it up!"... too many times, I've seen them cherrypick and go "Hey, Place X is doing better than us with fewer resources... it's time to trim things up!"

I really do hate being bitter and jaded. But unfortunately, it's all deserved.

Re:What a load of crock. (1)

Scarblac (122480) | more than 7 years ago | (#18969709)

Look at it this way:
If you are better than the rest, you should make sure everyone knows about it and ask for a payrise.
If you are doing worse than others, then you better improve.

Except it isn't that easy, is it? For one thing, the test that gives a correct benchmark of my entire job does not exist; at most it'll measure some small parts of it, badly. Secondly, nobody else in the company has exactly the same job, so it'll always be comparing apples and oranges, but that won't stop them. And thirdly, there will be people who do things like work insane overtime, just to get higher benchmarks - resulting in pressure on me to do that as well, but that's just something I don't have to do. Fourth, a good enough test takes a lot of time, which is frustrating and wasteful. Et cetera/

Set me realistic goals and see if I meet them. The rest is my business.

Re:Here's your benchmark... (1)

Moe1975 (885721) | more than 7 years ago | (#18967697)

An excellent post, one which reflects - and has articulated - my thoughts, feelings, and experiences.

Thank you Cervantes.

What I indeed needed to read right now - I have recently *FINALLY* broken free of the most unbelievably hideous monster of a human being, with an MBA, that anyone can possibly imagine.
I kid you all not, the woman is morally, emotionally, and psychologically deformed beyond what even our powerful imaginations can fathom . . . and it talked so much trash about us tech and engineering types (while being a total MSFT groupie, even though it uses unlicensed products at its company, a small shop that pulls in up to 600K a month, go figure) that I got half a mind to post its contact information in hopes that folks will get in touch and let it know how they feel . . . but I won't, even though I got burned pretty bad . . . something or someone HAS to take care of that, someday . . .

Fellows, I will repeat what has been said and written many times before:

                                                      ***NEVER TRUST A GODDAMN SUIT***

Ever.

Not for one second.

Re:Here's your benchmark... (1)

Cervantes (612861) | more than 7 years ago | (#18968251)

As much as it pains me to be bitter and jaded, I agree with you.

The majority of suits get there by socializing, backstabbing, schmoozing, penny pinching, and slumming. If you can find one who got there based on their virtue, keep them around. Most, however, got there under questionable circumstances, and they'll keep behaving the same way when they're in there. You simply can't trust them until they prove otherwise.

On another note, I believe the BSA allows anonymous complaints to be filed. So if they're doing things they shouldn't, you can pass that along without affecting your future job references.

Re:Here's your benchmark... (2, Insightful)

GovCheese (1062648) | more than 7 years ago | (#18969125)

Management doesn't understand you or what you do.
That's your fault. Once a day I go to a different office, a different section or division head, and to their staff offices and ask, "Hows the IT today? Everything working alright? You have everything you need?" And I listen and get back to them. If they trust that you care, then they'll start listening to you. And if they trust you, they won't slam you the first time there's a mini-crisis on their desktop. The best survey is done over and over at least on a weekly basis face to face. It's not a written benchmark but there's nothing better than getting to know the users, those ungrateful fucks, and letting them get to know you, you bitter, overworked bastard.

the golden formula (4, Funny)

blhack (921171) | more than 7 years ago | (#18965801)

(Monitors on Desk * cups of coffer per day) - (downtime of mailserver + vista installs) / (|the temperature of the server room|) = Productivity

x2 multiplier for Linux install on desktop
x5 multiplier for BSD install on desktop + 1 free very much needed hooker
x10 multiplier for *nix install on managements desktop


Lather rinse repeat for each member of IT department.

Add them together, if the department has somehow manipulated these values to = the number 42, give them all a free car If all of these values are stored in text files encrypted with 4096 bit encryption and spread across a 3x redundant cluster of 10 load balancing servers so that this value can be calculated constantly to the 10 millionth digit....

....pray

Quality of services (1, Insightful)

Anonymous Coward | more than 7 years ago | (#18965805)

I'd say in order, quality of services, security, and cost.

          So, for quality of services, lots of downtime is bad.

        But just about as bad are setups that are clunky to use. I've seen some custom apps that are pretty, but if the person has to enter forms, not being able to tab is bad. Cancel buttons that make the app pop up an error box instead of cancelling, and worst of all, do what you have to do to make the app fast. I use some clunky custom cash register app where I work, the database it uses is crap so it's steadily gotten slower over time. Soemtimes now it'll take over 15 seconds to add an item to the receipt, other random times it's seemingly instaneous.

          Security. An IT dept. isn't top notch if it's left SSNs or credit card numbers up on a web site even if they're really nice otherwise 8-). Also, I don't think it's so good if there's systems failuers due to viruses etc. Beyond that, it might be hard to tell anything, I suppose most companies are tight-lipped with security statistics. If security goes beyond being secure to being a nuisance, that's a problem, which is why I didn't list security first.

          Cost. If several departments have happy users, and good security, then the one that does it cheapest wins. If two departments provide similar service services, but one took 5x as much hardware (per user) to keep everything snappy, then the other department should get props for making things efficent rather than throwing more computers at the problem. This of course also saves electricity and cooling costs. Spending $ billions on an app should put them right out of the running, even if the app ends up being extra nice.

Simple. (0)

Anonymous Coward | more than 7 years ago | (#18965861)

Walk into the IS department and pick a box at random. Tell them to shut it down, disconnect it, whatever. "This unit just died, utterly and completely. How do you recover?"

That's a tough one (1)

wmwilson01 (912533) | more than 7 years ago | (#18965913)

Where are you getting the data about the other companies? I think that's got to be your starting point. Obviously, if there's not data to be had to compare against you're dealing with an impossible task, and I can't imagine that many companies really publicize that type of stuff. Benchmarking your own team against itself is obviously quite easy as you should have some firm requirements/service level agreements/whatever.

Another typical "not what you asked" response (1, Insightful)

Anonymous Coward | more than 7 years ago | (#18965983)

It sounds like your CEO can't think of any new strategic improvements he can make to the company so he's falling back on busy work. Time to look for a company with a future.

How about a business metric? (1)

Phoenix823 (448446) | more than 7 years ago | (#18966039)

Take a look at how mature the strategic alignment between IS/IT and the business is. Unless your business is actually IT, IT and the business usually sit on opposite sides of a fence. It seems to be a reasonable way to rate both IT and the business. http://cais.aisnet.org/articles/default.asp?vol=4& art=14 [aisnet.org]

Salary and benefits. (1)

shaitand (626655) | more than 7 years ago | (#18966103)

The more your staff makes the better you are. By paying the highest salaries you attract the most qualified and talented IT staff around. Just tell them it is the free market at work. That's economics, CEO's understand that stuff.

Also explain that another useful metric is the numbers of hours worked. If salaried IT staff are working fewer hours it is because they are operating more efficiently and doing a better job of earning their salaries.

Benchmarking? (1)

edwardaux (710732) | more than 7 years ago | (#18966169)

How would you benchmark an IT/IS Department?
With this [wikipedia.org] or this [wikipedia.org] , obviously.

--
edwardaux

Forget it (1)

gweihir (88907) | more than 7 years ago | (#18966319)

You cannot benchmark things that are nstill evolving fast. If you have the good fortune to have some really good IT people on your staff, do anything to keep them happy. They may just save your neck.

Typical example: A really good dystem administrator catches most problems before others notice. Consequentially these people are often sacked by clueless management, because "'there was no problem''. Face it: IT is expensive and still experimantal today. And you absolutely need it to work.

If you really need benchmarks, go for things as desaster recovery capability and speed, incident handling speed and quality, flexibility of the IT staff with regard to using different technologies and ability of the IT staff to make competent technological decisions. If technological decisions are made by the management, then don't bother with benchmarking, since you are doomed anyways.

Metrics (2, Informative)

realperseus (594176) | more than 7 years ago | (#18966413)

At the firm I'm employed by, we measure on several items in several ways. We have many departments under the IT "umbrella" so I'll try and break them out and briefly explain each + expectations.. BTW, we employ 1000 users.

Helpdesk (4 employees): Their mainly measured on how many helpdesk "tickets" they can close without sending the problem off to a 2nd level tech. They can reset passwords, add printers, etc.. They can remote also into users desktops to provide assistance. Their resolution rate of about 40%. The remaining tickets are passed up to the 2nd level.

2nd level techs (6 employees): They are expected to close "standard" tickets within 16 business hours from the time the call is taken at helpdesk. They will visit users in their cubes to assist. They're expected to close 95% of their tickets each month within the 16 hour time frame. Anything they cannot fix goes to 3rd level techs.

3rd level techs (4 employees): They get 8 additional hours to close ticket (24 total). They provide final resolution or call the proper vendor for support. They are held to 95% completion rate for non-vendor issue calls.

Telecom (3 employees): They handle call flow, broken phones, fax server, etc.. They have 16 hours to provide resolution from the time call is taken. They also are expected to close 95% of their tickets in that time.

IT Ops (15 employees): They handle AS/400 (logins, reports, administration issues) and backups. Same 95% within 16 hours without vendor assistance.

We also have programmers, database admins, and special system admins. Not sure how they're measured.

About 100 employees in all making up 10% of the company. Here are some other things we measure monthly:

Spam blockage as percentage of total incoming email, viruses stopped/detected, faxes in/out, calls in/out, internet usage as percentage of total pipe ("tube" :-) size, # helpdesk tickets opened vs number resolved by helpdesk/2nd level/3rd level/telecom/ops. Wish I had more but I can't find the metric reports on out Intranet but I hope this post assists a little.

Re:Metrics (0)

Anonymous Coward | more than 7 years ago | (#18967407)

Helpdesk: Their mainly measured on how many helpdesk "tickets" they can close without sending the problem off to a 2nd level tech. ... Their resolution rate of about 40%.
Check

2nd level techs: They're expected to close 95% of their tickets each month within the 16 hour time frame.
Check

3rd level techs: They are held to 95% completion rate for non-vendor issue calls.
Check

And I bet they all meet their quotas. Oh, there might be a few hard-working conscientious people who try to actually fix things, but they'll get fired soon enough. Before long 100% of your staff will be very good at closing tickets. Too bad problems won't be getting fixed, but the people they support will figure out they need to fix their own problems. Now if you excuse me, I need to write another dozen replies before quitting time tonight.

first, find an IT dept that sucks (1)

acvh (120205) | more than 7 years ago | (#18966667)

and then compare yourself to it.

obviously, ... (1)

dltaylor (7510) | more than 7 years ago | (#18966703)

Have you ever lost an entire month's output due to a server crash?

http://hardware.slashdot.org/article.pl?sid=07/05/ 02/1215200/ [slashdot.org]

Seriously, the most gloomy of the responders about how badly your department is about to be hosed are optimists.

Start a meta-metric (1)

jafiwam (310805) | more than 7 years ago | (#18966807)

A meta-metric on this issue is easy.

Start your stopwatch when you start working on it, and stop it when done.

Do that for the duration of the project then add the total time all up.

File the report under "waste of fucking time"

Last time I checked, RAID arrays don't fail in sympathetically because the one next door did. They do or they don't. Your metric should be "did shit break?" and "did it get fixed fast or rendered irrelevant due to failover"? "Yes/No".

What the guy next door does is irrelevant and trying to compare your stuff with his stuff is middle-management claptrap.

Check out Paul Strassmann's work. (1)

emes (240193) | more than 7 years ago | (#18967113)

www.strassmann.com has some of the best material around for
considering questions like this.

Time to get your resume in order (3, Insightful)

Whuffo (1043790) | more than 7 years ago | (#18967167)

When management is looking for benchmarks / SLAs / metrics to judge IT performance by, you can be sure of two things:

1: Management is clueless as to what the IT staff actually does. 2: Management is looking for ways to reduce headcount.

When you see this nightmare raise its head at your place of employment - be worried. Especially if you're at the top of the pay scale - you'll go first.

If you see this at the same time as there's handwringing about the cost of health benefits, don't just get your resume in order, start looking for another job. Even if your job survives the reorg that's just around the corner, you'll be working twice as hard.

Ask your customers (1)

throx (42621) | more than 7 years ago | (#18967217)

IT is a service department (something many bigger IT departments forget) and their primary goal should be to keep the customers happy, within obvious budget constraints. If I were benchmarking IT (or HR, or accounting, or any other internal service organization) I would simply benchmark the level of satisfaction by their internal customers.

I think comparing with other companies is a waste of time simply because no two companies have the same requirements or specific networking setup - usually for historical reasons.

Of course, if you want a good company to benchmark against, my company's IT department routinely costs me days for downtime by pushing out broken OS patches, deleting random DLLs from my system directory because THEY don't want it any more (never mind the devs use it) and flagging internal applications as disallowed just because they were too lazy to even to a second or two of research. Oh yeah - they consider themselves a profit center too by charging us internally for the number of PCs we have sitting on our desk, never mind I wouldn't call them for help if they were my last option.

That's a Tough One (1)

PPH (736903) | more than 7 years ago | (#18967223)

Measuring the performance of any department within a company that is not directly involved in its revenue is tricky. Your costs are 100% overhead so they don't vary as a function of sales.

Comparing your costs to that of another company can be tricky if you provide anything more than commodity services (desktop support, file servers, e-mail, etc.). Does your IT department support any sort of 'enterprise' processes (I can't think of a more descriptive buzzword at the moment? IOW, a business process that your management considers provides your company with a competitive advantage in your market. If so, its going to be difficult to account for additional IT costs to support this and it might not be something your management is willing to give up.

You might need to institute some sort of cost accounting system within IT to track how much is being spent on various categories of support (printers, file servers, etc.) and separate certain key functions (like supporting an e-commerce web site).

Ticketing software with a good statistics package (1)

Daishiman (698845) | more than 7 years ago | (#18967297)

Use something like Remedy for problem and change tracking. When users report problems, you write and document an audit trail and mark the problem resolution and put the root cause for the issue and the downtime suffered and impact of the issue. Then you can take fun statstics to see your downtime, how important it was, the components in question, and whose problem is was.

Obviously the hard part then is saying when everything's good enough, but knowing most corporations, it never is. There's always something to improve, and when you're done improving, there's new infrastructure to add with it's own bunch of issues. That's a good time to look at your favorite set of "industry standards" to say to management, "See, we're more than good enough. Go away".

IT is treated as a cost, just like the janitors (1)

QuantumRiff (120817) | more than 7 years ago | (#18967421)

If your janitors are working hard all night long, the execs will come in, see how clean the office always is, and wonder why they have the janitors. They will fire them, (or replace them with cheap idiots) and only after the office turns into a filthy whole will they realize their mistake.

If IT is doing its job right, the normal day-to-day users should almost forget its there! Updates and upgrades are smooth and invisible, problems are fixed proactively, instead of reactively, and people will wonder why the hell were even there, since there is never a problem. And as long as were just sitting around just in case a problem ever happens (which to them never does) why not pay some guy $2.00/hour to babysit the computers/networks.

Re:IT is treated as a cost, just like the janitors (1)

Billly Gates (198444) | more than 7 years ago | (#18968291)

Sounds like the I.T. department of your company may need to sell themselves better.

Good I.T. helps them make money and not just save it by working computers.

Uptime is assumed. Get strategic. (4, Insightful)

Organic Brain Damage (863655) | more than 7 years ago | (#18967515)

If the CEO's asking you to benchmark your IT group, do not JUST measure your uptime or other simple availability metrics, that's just silly.

Either you're reliable or you're not. But reliability, while important, is only part of the picture.

For struggling, barely competent IT guys, maybe reliability is all they can hope to achieve, but let's assume that's not you and yours. Let's assume you've got reliability down pat. Why do I make this blithe assumption? Because we can purchase reliability off the shelf today for not very much money. Windows XP boxes on programmers desks regularly go 2-3 weeks without requiring a reboot. Servers can run even longer if properly configured, fed and cared for, even running Windows.

So, where do we go from 99.999% reliable with an IT department? We start consulting with our business managers. We work with them as a partner to help select and adopt technology that will make the business more efficient and/or more effective. And we benchmark these efforts by measuring the productivity of the business function before and after the new technology/system is deployed. This is where we show the strategic value of IT. Not in uptime.

And, we take into account cost. If we can get 99.999% reliable and your competitors IT group can get 99.999% reliable, the budget becomes the tiebreaker and the winner is the one that does it for less money.

For instance, if a business is indexing large document sets for the litigation support market and we can figure out how to move that function from Dallas to India thus cutting labor costs by 60% without a loss in quality or responsiveness by cleverly deploying networking technology and low-end PC's into Madras, we have provided an easily quantifiable cost structure improvement which can give our company an important competitive advantage. This is an example of the big payout from IT. The other big one is quality improvement. And the really rare one is opening up a whole new way of doing business...disruptive technology.

How would I benchmark them? (1)

aardvarkjoe (156801) | more than 7 years ago | (#18967581)

BogoMips. What else?

Warning - opinionated opinion below. (5, Insightful)

NeutronCowboy (896098) | more than 7 years ago | (#18967627)

I do this for a living, so you'll probably hate me for some of my suggestions....

Number one thing I'd like to get out of the way is that the people who qualify this request as a waste of time are idiots who shouldn't be in an IT department. I've been to too many places where no one can tell me what they're running, where the servers are, what patch levels they are at, which machine is up or down and what they need to keep pace with growing demand (your business IS growing, right?). IT departments like these are the root cause for bloated IT initiatives, downed web servers and crappy customer service.

Number two is the realization that the IT department is in the business of the keeping the rest of the business up and running. This means two things: your responsibility is to keep the rest of the company happy and productive, and you get to charge them (system doesn't matter, as long as you keep track of what it means to service 200 requests a day for lost passwords) when you perform work to keep someones servers up, the mail flowing and the network lit.

Once you understand the place of IT in a business, metrics are easy to come by:
- how well you're doing is measured by how available your servers and your apps are.
- how efficient you are is measured by how much money you save the rest of the business in avoided downtimes and increased productivity.
- Bonus if you keep track of how long it takes to service requests and what their resolution was.
- Extra Bonus if you can do performance forecasting and figure out when you need to expand your infrastructure.

The performance of an IT department is NOT measured by any of the following:
- budgets of IT departments in other companies
- how many new products you've rolled out
- how many people you fired (or hired)

Anybody who suggets those needs to be smacked around and told not to speak in public anymore.

With that in mind, there are a million products to track your statistics. My company will be happy to quote you anything from a 4-figure to a 10-figure deal for this. There are open-source tools that will do similar things for free (though in my opinion, you get what you pay for). There are in-house and hosted answers to any of these questions. But the one thing you need to remember is that you need to know the answers to the CEOs questions about what the status of your machines and your people is. If you can tell him that your hardware uptime runs at 99.99% for mission-critical servers (the Oracle RAC that holds your financial information, for example) and your app availability is 99.7% during business hours, that it takes you 4 hours to respond to a priority 1 request and 3 days to a priority 3 request, and that based on current response time trends, you need to double the processing power and database space in 6 months, you're golden.

If you can't tell your CEO these things, you will be replaced by someone who can. Or even worse, your job will be outsourced to my company, and I get to work out these things while you're flipping burgers. Yeah, I'm being a jerk. That's because I'm flabbergasted at some of the comments (and their moderation) and how their authors still have a job. IT metrics are simple, and don't have to start with complex SLA measurements and other crap. You can start with a basic ping monitor of your critical servers, and go from there. But for heaven's sake, do something. You'll be a hero if you play it right.

Oh, and just to repeat - do not benchmark yourself against other companies. You don't have access to valid data, and you won't find an identical business. Instead, find out what your other departments need from you, and benchmark from there.

Re:Warning - opinionated opinion below. (1)

funwithBSD (245349) | more than 7 years ago | (#18968435)

If you cannot measure, you cannot improve... or at least you cannot prove you improved.

One thing to watch out for: System uptime and application uptime are different things! A good program on a squirrely system is no good, and a solid box with flakey software is just as bad. Network is also different altogether, if that is unstable nothing else is stable. Again, networking is a well understood problem and you should not have to worry about it if you have good people.

Making the OS stable is fairly well understood, so do that first and show how stable it is.
Getting a house-written software stable is more difficult, but it can be done. One way to show how unstable a program is is to pair it with a well running program. Then when they say our program X failed because of a OS bug, you point to the uptime on the box and on app Y that is been up for a long time as well. Assuming logic has any sway at your company, they will be exposed as the problem child.

That is an excellent comment... (1)

Evil Poot Cat (69870) | more than 7 years ago | (#18969919)

...especially the part where you lead off by slamming the people who don't like the request, then join them a few paragraphs later. :)

And to add to the metrics, long-term bonus if you can report (when applicable) what chronic problems/issues you've resolved (or at least handled). Basically, it's an extension of metric #2 in the parent's list, just be aware that self-analysis can bring some large-scale benefits over the medium- or long- term.

 

Apples != oranges (0)

Anonymous Coward | more than 7 years ago | (#18967731)

I would have thought that the request is fairly bogud to begin with. You would have to find a number of companies which mirror your size, profit, market, technology, payroll etc.. to be able to develop any comparative metrics that might mean anything.
First problem is how you would find enough information about the prospective companies to determine that they could be used in a represntative comparison. You might be able to get some of this from financial reports, but they usually are doctored a bit and might not include enough depth of detail.
second problem is how you would get this other company to provide ongoing data so that you could tune the metrics and compare your company to them anyway.
'Hi, this is Jim from ACME inc, you're competitors. Would you mind giving me a copy of all the financials and support documents pertaining to you IT department so that we can compare ourselves to you? We won't use it for competitive advantage, honest!'

My rule of thumb, never trust management that wants you to define your own KPIs. Actually never trust a management that uses KPIs, full stop.

Re:Apples != oranges (1)

Jack Schitt (649756) | more than 7 years ago | (#18968635)

'Hi, this is Jim from ACME inc, you're competitors. Would you mind giving me a copy of all the financials and support documents pertaining to you IT department so that we can compare ourselves to you? We won't use it for competitive advantage, honest!'


'Yeah, Hi Jim. Thank you for calling. It looks like I might be able to, sorta, ya know, help you out in this matter, but first I'm gonna have to ask if you could go ahead and send us _your_ financials and supporting documents pertaining to _your_ IT department as well. Just go ahead and send that to me, Bill Lumberg at Initech, thanks. And please make sure you don't forget to use the _new_ cover page when sending over the TPS report.'

Yes, I'm trying to be funny, but I also want to bring up the point that if there was an even exchange going on (i.e. I'll show you mine if you show me yours), this request is more likely to go through.

'Hi, this is Jim from Acme. We're in the process of benchmarking our IT department... yeah I know... I got it... OK stop laughing, it's rude... anyway, I was wondering if I send you our IT department financials and supporting documentation so you might be able to benchmark your own department, you could send us yours?'

obviously (1)

textstring (924171) | more than 7 years ago | (#18967781)

use one that makes you look the best

Here's my benchmark. (1)

DrRevotron (994894) | more than 7 years ago | (#18968045)

1) Lock server room door(s).
2) Jiggle loose a power cord or two on a noncritical server(s).
3) Tinker with some of the installations.
4) Secretly time your staff and watch their methods.

I'd say 5-15 minutes, depending on the severity of your tinkering, is average. If it takes any longer, I advise liberal usage of a tile puller on a few skulls.

A few things to measure... (2, Informative)

caitriona81 (1032126) | more than 7 years ago | (#18968169)

First off, as many people have commented, you don't want to blindly give management what they are looking for here, but you also don't want to ignore such a situation, as they are probably trying to justify the cost of IT.

You need to turn this around in favor of your department, and it would be a very good idea to take a look at what metrics you can apply that will serve a twofold purpose - to set a baseline of current performance, and to set a moving target for constant improvement. In other words, the things you measure should paint a picture of things that your department can and should improve on, if given the resources to do so. An open ended reporting task like this is a setup from the start, but you need to turn it into a chance to show both how well the department is working with what it has, and how much better it could be working if management would let it. That means finding out where your human resources are being wasted, and making recommendations for refocusing those efforts, finding out what parts of your infrastructure are money sinks, finding support costs that can be reduced with changes, and justifying expenditures that will have a concrete benefit to your department's ability to meet the business needs.

If you don't have them already, now's also a good time to start setting realistic SLAs and tracking compliance with them - for everything from backup/recovery time, recovery time objectives, recovery point objectives, helpdesk call resolution times, server reliability, etc. Just make sure they are realistic, and within reach, and re-evaluate them frequently.

Formal statistical process control methodology such as Six Sigma can be useful in the IT department, but only with the people to make it work, and an organization large enough that it can see enough cost savings to justify having a formal quality team. In order for statistical process control to help, everybody has to be onboard, from management to the lowliest member of the department. If you can achieve this, the rigid define, measure, analyze, improve, control methods of Six Sigma can probably create a savings of cost, labor, and sanity within your department.

I would also look at regulatory compliance as a benchmark. Even if your company is not publicly traded, the tight controls that are necessary for public companies to comply with Sarbanes Oxley (SoX) often lend themselves to better IT practices - formal validation of your IT controls (access to information, physical and logical access to systems, authentication credentials, least privilege, auditing, etc) is a good idea.
Do users have local administrator access to their desktops? If so, why? What applications are requiring it?
Do you have an audited software inventory? If not, what's stopping you?
Do you have controls to prevent unauthorized software installation?
Do you have controls to insure virus scanning and security patching are done appropriately?
Do you have controls to make sure than users have no more access rights than their responsibilities require?
Do you have controls to keep track of why users have the access rights they have, and who approved it?

Measuring your disaster recovery capabilities, and realistic evaluation of the scenarios they have to survive are also a must:
What are your worst single-failure scenarios?
What are your worst two-failure scenarios?
What can be done to mitigate the above?
What has already been done to mitigate the above?

I'd also take a good look to see whether your department is spending it's time putting out fires, or keeping them from starting - if you are just barely keeping up, then it would be good to look at why, and what can be done to change that. Take a good hard look at your helpdesk, and apply the 80/20 principle - roughly 20% of the causes will be responsible for 80% of the calls - what can be changed that will improve the situation there?

From the end user perspective, what issues are most disruptive to users? What issues are hurting the productivity of users outside IT, and how could your department improve those issues?

Same applies to systems issues - what roughly 20% of causes is causing 80% of your work?

Last, and this is the big one - don't give such a report to management without some serious suggestions for improvement - backed up with risk assessments or cost assessments - an open ended reporting task like this is a good chance to shove the ball back in management's court, and try to justify changes that would really make things better - and an even BIGGER chance for I-told-you-so when the recommendations you make are ignored, and the problems they were meant to solve materialize.

Mention how much money I.T. brings to the company (2, Interesting)

Billly Gates (198444) | more than 7 years ago | (#18968275)

How much money does your inhouse ERP apps bring? How about your wharehouse database that can be used to cut down misplaced inventory? How about your accounting apps that help the CFO find out which areas are hte most profitable?

Forecasting supply/demand apps?

These all make money and help streamline business processes and work with the CEO and his company.

Thats what CEO's want to hear and what a good I.T. department does in a fortune 1000 company.

Not that your high tech janitors like another poster mentioned, which in this case would mean its time to either move to India or update your resume because your a cost center.

What is your SLA? (1)

ditoa (952847) | more than 7 years ago | (#18969621)

First of all what is your SLA? You should measure yourself firstly against the SLA in place. If you are way out you need to figure our why and/or maybe adjust the SLA. Do you offer a standard 9x5x5 (9am to 5am 5 days a week)? You need to have a clear cut definition of what services and at what level you will supply those services. Does your company outsource code infrastructure such as core network and telephony to another company? If so how does their SLA measure up to yours? Your SLA should be lower their your suppliers for obvious reasons.

How many 9s do you provide? In all honesty anything 97% or above should be acceptable to standard office users. Along with allowed scheduled downtime during office hours (such as lunch, first or last hour of the day) it should be fine. For core services this can be difficult (such as email) however you can resolve this in many ways most of which are rather simple (although require more hardware/software).

What is your DR plan? In case of fire/flood can you get your hands on the off site backups in 24 hours? Can you have a fully operational network with all data restored in 48-72 hours? These are realistic numbers for most businesses.

Other areas you should measure yourself on are how well your support your company with new technology? Have you looked into things such as server and storage virtualization? What are your storage technologies? SAN based? What backup systems do you use? Are they the most efficient for your business? I am not saying you have to use the latest greatest but you should know about them, their pros and cons, why they are/are not suitable for your business (And money is a perfectly acceptable answer).

What are your infrastructure forecasts? Things such as storage, CPU power, accessibility (internal and external, VPN?). In 1 year how much additional storage will you require? Can your current solution support this? Will you need to purchase a new system? Can your backup solution cope with this growth? Will you need a bigger tape library? Will this effect your scheduled downtime? What about DR numbers?

Providing you can answer all these questions you can consider yourself in good shape. Don't measure yourself just on uptime. Uptime is just one part of a much bigger picture, yes it's important but there are many other bits just as important.

ITIL (1)

euxle (127207) | more than 7 years ago | (#18970207)

You might want to look into the IT Infrastructure Library - http://www.itil.org/ [itil.org]

Best practices, key performance indicators, you name it ;}

Obviously the best method (1)

kvant (939634) | more than 7 years ago | (#18970505)

First you count the people who are around at 11am, then you use it to divide the amount of empty pizzaboxes in the department.

pizzaboxes/(peopleinoffice + 1) = result

Greater the number, better the department is.

Benchmarking IT (1)

Tzinger (550448) | more than 7 years ago | (#18970535)

I have to answer with a question -- by benchmark, do you mean 1) comparing yourself to other IT departments or 2) developing a system of measurement that proves you are effective and getting better.

I prefer the second approach -- develop an internal system of performance measurement. This is not a trivial exercise but it has been done. I suggest your get you CIO a copy of Martin Curley's book, "Managing Information Technology for Business Value." Here is a link to Intel Press http://www.intel.com/intelpress/offers/bundle5.htm / [intel.com]

Just one measure (1)

SpacePunk (17960) | more than 7 years ago | (#18970889)

Do they keep your shit fixed?

By marking the bench, silly. (1)

Unanimous Cowturd (891079) | more than 7 years ago | (#18970925)

Sit down all the sysadmins and techs on one long bench. Measure the amount of, erm, 'parking room' needed. Do this at the beginning and end of the test period.

This 'parking room' should increase as they become more efficient, automate more processes, and get off their buns less often.

The same method can be used to downsize the department as efficiencies grow; after a while, they'll have less and less room on the bench. Kind of like musical chairs.*

Note: This could have been written in the '1. 2. 3. Profit!' form, but I was too lazy.

*Except the skinniest one would be squeezed out by the 'most efficient'. Evolution inaction.

PS: The captcha text for this post was 'largely'...
Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...