Monday, June 16, 2008

I quit my job today, oh boy

I turned in my two-week notice today. I was working for a fairly large company doing WebSphere Administration and Java development. My new job is IT Director for a much smaller organization. It'll give me the chance to put into practice my own take on IT on a (relatively) clean canvas.

It's bittersweet leaving my current job. I enjoy the camaraderie of my fellow developers, but the IT department itself leaves much to be desired. There are some very smart folks, but they don't get the opportunity to demonstrate it on a consistent basis. Over the past couple of months, we've been hemorrhaging people. Today alone, 3 people turned in their resignation. In the past 6 months, they've lost 6 other high quality employees. Without going into too much detail (in order to protect the guilty), I'm going to use this post to explain why.

IT in Corporate America (herein after referred to as CA) doesn't have to suck. But by and large, it does. Why don't the smart people who work in CA get the opportunity to show how good they can be? Part of it comes by being saddled with inferior tools: bad version control, bad testing tools, bad test data, bad languages, bad platforms. However, these are technological problems that have good technological solutions. Why aren't they embraced? Why use Serena or Harvest for version control when there's svn and git? Why does it take 4 hours to find a user id in development to test a change that took you 30 minutes to implement when a daily refresh of test data would solve the problem? Why buy expensive monitoring software that accomplishes nothing more than sending out snmp messages to a blackberry when you can use nagios for the same thing? And finally, why do the people who propose these changes never, ever, ever get listened to?

The facile answer is to blame management. And, there's something to that. IT Management needs to bear the responsibility for allowing the organization to dismiss their smart people as "friggin nerds" or pie-in-the-sky dreamers. But the problem runs deeper and these phenomena are merely symptoms. The underlying problem is CA's addiction to business-centric IT.

In CA, IT costs are looked upon as necessary evils. CA needs computers and email and Web sites, because that's what CA needs to function and be profitable. Other than that, the IT department can kindly piss off, thank you very much. But oh, before you do, can you add that new feature I discussed to our Web site by next Thursday? Then go piss off, until trades get botched on Friday night, whereupon I'll need you to spend the next 48 hours figuring out what the hell went wrong and why we lost $150,000 dollars.

Most CA IT shops try to solve this problem with processes. They develop SDLC's and design documentation and have change control meetings. These processes do a decent job of mitigating damage caused by rogue change. What they don't address is how change gets introduced and, more importantly, who drives the change.

The majority of changes in CA IT are driven by the business. This has the effect of turning corporate IT into an order taker for the business. The corollary of this is that IT doesn't have the ability to create and drive their own changes. IT infrastructure changes don't immediately (or obviously) fit into any specific business goal – at least from a business perspective. Why should the business care about getting new servers or having developers waste their time rewriting already functioning code? It's anathema to them – at least until IBM stops supporting that version of WebSphere. Unfortunately for the business, this strategy works in the short term. You don't start to see problems until you're well down the path of no return. The code base increases in complexity and redundancy because the entire thing is one layer of band aids on top of another. Now that “simple change” becomes a nightmare of an ordeal because it's impossible to identify all the interaction points. And, without a repository of automated testing scripts, the developer is forced to wonder what all he might have broken. We've had instances where a change introduced on Friday is accused of breaking production on Monday only to find out that it has, in fact, been broken for the past 6 months.

Corporate IT needs the ability to run their own internal infrastructure projects without needing permission from the business. The business has managed to survive this long without that change in the vesting page to convert dollar values into euros. Seriously, what's the difference if the change goes in June versus September? Why not take that 4 month period and refactor the common code base? Or convert those Windows machines running Apache over to Linux? Or research Drupal as an alternative to your current, multi-million dollar CMS that's constantly breaking? Or use nagios to get real metrics on Production uptime?

Most people in corporate IT want to do a good job. Most people in corporate IT are capable of doing a good job. Every developer I know is driven crazy by crappy code; they want nothing more than to get in there and make it better, stronger, faster. Most of the time, however, they are actively stymied from doing a good job because they can't act on their own, best instincts. Sorry pal, but that's not in the business plan.

36 comments:

dacracot said...

Amen.

Mark said...

Let's face it, CA doesn't feel safe unless they are paying a boat load in licensing fees! Good luck in your new endeavor, buddy. -MB

Adam said...

Welcome to Corporate America where the status quo is more important than getting things done. I don't think its just an IT problem. Its just that IT people see it as a problem to be solved.

To me it comes down to good leadership. Leadership that communicates the right message at the right time to the right people. Of course I've never really experienced this in Corporate America.

It must exist. I mean really. It isn't that hard is it? Is it?

pi3832 said...

Typical geek: blame the suit for not bowing before your genius.

CA exist to make money. If your IT proposal will make money, CA will approve it.

Perhaps the problem is that IT doesn't do a good job of selling its ideas to CA?

Damon Allison said...

Preach on brother. Best of luck and congrats!

Mark said...

Thinking like this is why said company is bleeding at the jugular in the IT department.

Mark said...

Previous comment was meant as a response to pi3832's comment.

Nothing To See Here said...

pi3832: I guess that's one for the "friggin nerd" vote?

The problem is that it's difficult to paint a clear path from, say, code refactoring to profit.

Some Creep said...

I recently watched the tendering process for a fairly simple development project. Maybe 5 pages all said an done, all implemented inside an existing sharepoint solution.

Provider 1 quoted about what it should cost. $6000 - cost of the work involved, and a mark up for the hassle of having to spec and bid for the job, and deal with a bureaucracy on completion.

Provider 2 and 3, having had relationships with the company before, quoted 335,000 and 425,000 respectively.

Provider 1 was excluded. Obviously their quote was so low because they were a mickey mouse company that didn't understand what was required. Provider 2 was questioned closely because they were so much lower then provider 3, and eventually after much reassurance from the sales people of Provider 2, they were selected.

One of the techies from another department saw the quote and asked to see the rest of the spec, assuming there was more to the job then what he was seeing. After the project manager confirmed that no no, that was the spec on which the company had quoted 335,000 - the techie expressed his concerns that the quote was vastly too high. The project manager called back and said 'One of my advisors thinks this price is really too high' - the sales monkey put him on hold, came back about 90 seconds later and knocked the price down to 235,000.

Project Manager thinks he got a bargain. Techie got a round of congratulations. And the company spent 40 times as much on the project as they should have.

The problem is that techies almost universally make bad managers, and management types have no understanding of the industry. Finding a middle ground between bad management and bad IT shouldn't be as hard as it is - but no big business I know does a decent job of it.

Sandro Magi said...

pi3832, business runs on two principles: driving sales, and cutting costs. Most IT projects this post discusses cut costs, but over a longer period of time. However, refactoring can also drive sales, since changes will be easier to make, thus decreasing turnaround time and improving customer satisfaction.

Unfortunately, the short-term myopic view that a large bureaucracy introduces doesn't see this. That's why the following principle is important: it's easier to ask for forgiveness than to ask for permission.

jjray said...

Great initial post. Look forward to what's to come and whether you find the grass greener as IT director of a smaller shop. Perhaps you'll find you have traded one set of problem for another?

"the entire thing is one layer of band aids on top of another." -- great observation.

Bobby said...

Love it.

(sniffling) Made me cry...

Jacob Proffitt said...

pi3832 has a good point and commenters so far have been far too eager to dismiss it out of hand. Businesses run on money. If you're working in Corporate America, you count on the business to pay you some of that money. You owe it to them to substantiate your claims in terms that are important to the business--i.e. in clear business language. Business knows all about deferred costs and long-term debt. Just because you haven't made a sufficient case doesn't mean that it's the business' fault for not adopting your favorite new development fad. I know because I've fought the battles and even won some of them.

You may know why spending time and money on your ideas is important to you. Figure out why they are important to the company. And here's a clue for free: sometimes your ideas aren't important to the company.

CA has seen this happen before. The whole dot-com bubble in 2001 saw IT flex their muscle and dictate company policies and demand inflated salaries and other perks. That's left a sour taste and solid distrust that I think we're still paying for.

The bottom line is that getting into a power struggle with the business folks is a bad idea for a whole lot of reasons--not least of which is that they hold all the cards. You want more respect, then earn it by learning to speak their language and presenting cases to them that aren't full of earnest boosterism for the latest development fad. At a minimum, be prepared to give at least two drawbacks for any given proposal and why the payoffs make them worthwhile. If you can't do even that much, then go back and do your homework like the professional you are professing to be.

None of which is to say that some companies aren't boneheaded and deserve to lose their top IT talent. Frankly, a company that experiences the drain mentioned here without management freaking out is in serious denial about something and bailing out sounds like good policy.

The honey monster said...

I have to play devils advocate here.

Absolutely the business should be driving IT, IT does not exist as a revenue center in its own right, it is a business multiplier.

The problem is that the business does not trust IT. CA has been burned time and time again by systems that fail to meet basic expectation.

Is it that IT is incapable of meeting their needs? Not at all, but there is a very strong tendency to re-define any business process in terms of the available technology.

This becomes a standard practice and it is very hard to shake the business analysts out of this thinking. Until then, of course the business will consider extant systems as its terms of reference.

If and only if the business model can be separated from the technology infrastructure, will the mismatch be broken.

To do that you need to insist that the analysts are qualified technologists, not a failed business person, squatting in IT before a suitable HR role opens up.

It can be done, I've seen it done, dare I say, I have been part of it.

Michael B. McClelland said...

I was a CTO for a small company that became a large company.

It illustrated the difficulties faced by both types of organizations.

Some problems are easily solved with simple solutions. When those problems scale though, often secondary or tertiary considerations which had been unimportant at the lower complexity, become all important.

When the solutions began requiring heroic individuals giving herculean effort to sustain (as the number of servers went from 10 to 100 to 1000) it became clear that some of the process, change control, and cost mitigation strategies that "IT Heroes" normally disdain, were the only way to feed the monster, the job became less fun.

When a simple patch to an OS was a monumental undertaking, resistance to change became a survival tactic.

Ultimately the heroes left. Who wants cruft and drudgery? But cruft and drudgery are handy tools for slowing down negative disruption. Unfortunately they can slow down positive disruption too.

Erin Ptacek said...

http://blogs.zdnet.com/security/?p=1287

Maybe this changes things?

Adam Drake said...

I too refuse to work in IT for large corporations. Having been an IT director and trying to implement good changes I was ineffective because the people above me did not want to change anything, even if I could demonstrate that it would be very cost effective.

The bottom line is that the mentality of the people running large corporations makes it impossible to do IT in that area long-term while maintaining the ingenuity and adaptability that makes for good IT staff.

The only place to work in IT these days, as far as I'm concerned, is for yourself or for a small firm that puts you in charge and doesn't care about how things are implemented, as long as they work.

I've been self-employed and haven't worked in corporate IT for 4 or 5 years and I don't plan on ever going back.

doswheeler said...

Heck, I dont blame him. I woulda quit do. Sucks working for other people anyways.
http://www.Privacy-center.net

robottaway said...

Can you please update your blog's template to not be a static width? It really sucks even on my 20 inch iMac... I imagine on larger monitors people are suffering.

Tyro said...

HA, seems CA can stand for Corporate Australia too. I see the similar poor decisions and mandates handed down to myself and other IT persons from above.

A more recent case in point; The Managing Director wanted to release a monthly video message. The decree from above; pay 60 grand for a years worth of propitiatory video compression. Then roll it out the custom player across the company (they also packaged and sent the videos because apparently "streaming would kill the network").
Icing on the cake: all filming required hiring a professional studio to come in with lights + cameras, the works around 10 grand per message.

What makes it truly laughable is a business unit (next door) was already doing the same thing.
One employee with a (decent) company camera and Windows Media encoder. Producing work of equal quality for less than 5% of the cost. Streaming via technology that every user already had on their PC (Windows is company standard).

This whole process still makes me cringe O_o

Erik said...

The large CA company I work for is very explicit in preferring licensed solutions over open source ones. The rationale, f'ed up but also no secret, is that we'd be on the hook legally if we used open source solutions that infringed on a patent. Of course, closed source (licensed) solutions infringe on patents as much or more than FOSS projects, but we're somehow protected from direct liability.

It's quite naive (I'm replying to some commenters) to say that CA's preference for crappy licensed solutions over superior open source solutions is 'bad salesmanship' from IT. The decision to hemorrhage money is quite deliberate.

Likewise, it's naive to believe that CA will do anything to increase the bottom line. Most CA employees learned early on that the different organizations must fully spend their budgets to ensure that next year's budget isn't reduced. Frugality is rarely rewarded.

Thanks for the insightful post, and best wishes for your new job.

tekkie said...

I agree with a lot of the comments on here, and I agree with your post. I've had this feeling that one reason why business is distrustful of IT is they have been burned before. The thing I've seen and heard that is consistent is business IT is risk averse.

I used to agree with the idea that business would listen if you pitched a technology idea showing the cost benefits. I'm sure some businesses do listen to such rational arguments, but there are others who won't, because they consider other factors. Believe it or not there are social implications to technology choices. There are financial relationships established. IT crews are often wedded to particular technologies, and resist change. Management may prefer one technology over another simply because it's what they're used to. They like the way a particular vendor treats them, for example.

Another factor I'm seeing is a mentality that agrees with Nicholas Carr: IT Doesn't Matter. I've met plenty of people who want to believe that technology matters in the same sense that telephones or electricity matter. The only differentiator is how you manage and structure it. The technology is just a commodity. It doesn't matter what you use. Tech is tech. Management and sales are what really make the money. Tech is just the means, and each kind is pretty much the same. That's the thinking. So proposals for innovation are going to fall on deaf ears in this environment.

True, bad management will sink a business no matter what tech you're using. I believe there is still potential for technology to be a competitive differentiator--not by just accepting the pre-packaged stuff that vendors sell. Nick Carr is right here. If you just use off-the-shelf stuff, IT is not going to be any more competitive. Where you can gain advantage is in-house innovation.

scott said...

I write this as a former techie who has his MBA and now resides on the business side of the house:


It problem is that for all the revolution it promises, IT is still a cost center. Are there opportunities for IT to be leveraged for strategic gain? Absolutely. However that is never communicated because IT is notoriously lacking in social skills.

And therein is another problem: Perhaps some techs in IT DO have the answers. However by the time they are done pitching their "technobabble" to the managers, any potential benefit has been lost in translation. I mean most of the time understand what they're trying to say, but I have an IT background. Keep in mind that the decision makers and those who control the funding AREN'T IT guys. Their backgrounds are in finance, accounting, etc.

What does all that mean? If you want to get a project approved or change things, you have to frame it in terms that management understands. Pi3832 had the right idea. If the IT department comes up with an idea that can save money, ANY company WILL LISTEN because now you're speaking their language. So you have an idea that will save some man hours? Big goddamn deal. You're salaried. Salaries are fixed costs. You're paid one way or the other. But if the same idea will also help lower say, inventory or licensing costs....

Same benefit, just expanded in order to benefit a larger (and more influential) group of people

Sara said...

Freaking preach it sister sledge (brother?). I couldn't have said it better myself. SO HARD to make a good IT department scale out.


-schipps
GirlDeveloper.com

Sleazy said...

This is a very insightful first post. Good luck in your new position and your blog.

Tomas Varaneckas said...

Corporate IT does not suck just in America alone, it sucks worldwide.

Nepali Akash said...

Everything said and done, it all boils down to saving their own ass(es).

James E. Ervin, IV said...

The problem is that infrastructure and refactoring goes under the category of internal quality, not external quality. External quality is what is actually seen by CA or clients. Internal Quality is what determines if your IT staff is ready to commit Hari Kari or not. External agents (CA for instance) are always ready to sacrifice internal quality in the name of any number of reasons, mostly those they see in the short term. Enlightened CA or management would see that sacrificing internal quality may "payoff" in the short term, but will doom the project in the long term. Somewhere there must be an organization led by enlightened people, if you find it let me know, I will forward my resume.

zack said...

This post almost made me cry. I Hope you are working for a large enough company where you have your own budget, can set your own projects and goals, and are given enough money to attract good people. I made the mistake of accepting a Director of IT job at a small company where IT encompassed everything including software and product development, entirely driven by the CEO and marketing department. After a long stint as their Lead Programmer. I thought I'd be able to turn things around and in effect I have done some good. But I foolishly did not pay attention to why the past 4 Directors of IT failed. I've come to realize the mistake. Oh well. You live, learn, and GTFO. Strategic Projects Team? Ha. If you hear that, run for the hills. Everything is a strategic project.

the said...

I'm sold... you're lucky you live in CA where there's enough tech to allow you a choice!

Good luck! I have a feeling I sympathize a lot with the ideas you'd like to "experiment" with

Dave said...

I agree, totally. You can say that if IT can show a proposal that will make them money that CA will approve it, but refactoring code for reuse cannot be accurately measured for future projects, but any coder will tell you that it is a valuable asset to any old/new projects. The other problem is the nature of IT vs CA. CA is grounded by business men--I think of those stereotypical blue suit wheeler and dealers that "make the deal happen". Usually, these people have no technology experience, and don't care to. Having someone with no technical knowledge making decisions on technology is a bad combination, especially when the only visible outcome is a web site or one application. But, again, this was the nature of CA before technology was thrown into the mix, but who is to change in this situation? I say that if CA wants technology, they must be willing to deal with the change.

reference said...

I had a chuckle or two I must admit. Great post.

The problems you mention from what I see stem from probably the biggest issue I have seen since I have been working.

The business as a whole must have a vision. Yes, I know you have heard that crap before but without it the company is a rudderless ship. Everything the company does should be geared toward that vision. From that comes, strategy > objectives> goals> tasks.

Once you have that, IT needs to create a technology roadmap (5 yr) that will tell the business how it will help them achieve their stated vision.(This is not a list of projects for the coming year either) Sounds easy, and makes sense but I have yet to work in a company that actually does any of this, and it shows.

The business has one purpose as far as IT is concerned and that is direction. Without it how can we support them in the way they need to be supported and help them achieve their goals?

This goes to the whole trust thing. IT isn't trusted because we can't live up to their expectations when they expect us to implement a new system 3 days after it was agreed that a new system should be built in the management meeting every department was invited to except IT, conversely we can't live up to the expectation because we usually don't know what they expect from us on any given day.

All of this ends up boiling down to one point. Communication, the more of it and the better it is helps everyone in the end. Unfortunately most people don't communicate in their own department properly let alone across reporting boundaries.

blah, blah, blah......

resumedocket said...

This makes me happy to read :-)I have gone through this blog. I found it very interesting .




Resume Writing Service

chandra said...

You seem to be a brave person! i also had the same experience..where in my company there are plenty of politics played and i had to resign looking all this..just got a job in IBM and happy that there work atmosphere is simply superb!


Director Job Descriptions

hunt said...

Oh it's been a while since I've visited here.
I like yur post. Really amazing and productive.

You know, a Creative Director quits his job and runs 100KM in one day!....
For futher information, plz visit my website here - www.bit.ly/khOT0P

Love all of the posts! Good luck to you.

maddy said...

Guys, I see lots of interesting and useful ideas in your posts, great discussion.. Thanks!



Director Resume Objectives