×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

IT Infrastructure As a House of Cards

Soulskill posted more than 3 years ago | from the if-it-ain't-broke-it-will-be-soon-enough dept.

Software 216

snydeq writes "Deep End's Paul Venezia takes up a topic many IT pros face: 'When you've attached enough Band-Aids to the corpus that it's more bandage than not, isn't it time to start over?' The constant need to apply temporary fixes that end up becoming permanent are fast pushing many IT infrastructures beyond repair. Much of the blame falls on the products IT has to deal with. 'As processors have become faster and RAM cheaper, the software vendors have opted to dress up new versions in eye candy and limited-use features rather than concentrate on the foundation of the application. To their credit, code that was written to run on a Pentium-II 300MHz CPU will fly on modern hardware, but that code was also written to interact with a completely different set of OS dependencies, problems, and libraries. Yes, it might function on modern hardware, but not without more than a few Band-Aids to attach it to modern operating systems,' Venezia writes. And yet breaking this 'vicious cycle of bad ideas and worse implementations' by wiping the slate clean is no easy task. Especially when the need for kludges isn't apparent until the software is in the process of being implemented. 'Generally it's too late to change course at that point.'"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

216 comments

All comes down to budget (5, Informative)

Admodieus (918728) | more than 3 years ago | (#32329574)

In most organizations, the IT department is treated as pure cost instead of something that provides strategic value. These IT departments have no chance of getting a budget approved that will allow them to "start over" on any part of their implementation; hence the constant onslaught of temporary fixes and patches.

Re:All comes down to budget (4, Insightful)

Opportunist (166417) | more than 3 years ago | (#32329612)

Budget and the lack of ability to see ahead, on the side of the decision makers.

Far too often decision makers are not the people who also have to suffer, I mean work with the tools they bought. They are often easily swayed by a nifty presentation from a guy who doesn't know too much either but promises everything, and of course the ability to cut cost in half, if not more, so they buy. Only to find out that the solution they bought is not suitable to the problem at hand. And then the bandaids start to pop up.

Re:All comes down to budget (4, Insightful)

Megaweapon (25185) | more than 3 years ago | (#32329734)

They are often easily swayed by a nifty presentation from a guy who doesn't know too much either but promises everything, and of course the ability to cut cost in half, if not more, so they buy.

If you've worked in a huge shop, you know that the big software vendors send reps out to IT managers for golf outings and the like. Screw it if the software works or not, just fluff up the guy with the budget rubber stamp.

Re:All comes down to budget (2, Funny)

schmaustech (766915) | more than 3 years ago | (#32329744)

If you've worked in a huge shop, you know that the big software vendors send reps out to IT managers for golf outings and the like. Screw it if the software works or not, just fluff up the guy with the budget rubber stamp.

I did not realize my coworkers posted on Slashdot.

like bubblegum under a desk... (4, Insightful)

Thud457 (234763) | more than 3 years ago | (#32329822)

There's nothing more permanent than a temporary fix.

Re:like bubblegum under a desk... (3, Informative)

Tridus (79566) | more than 3 years ago | (#32330900)

Yeah, I saw that line and immediately thought about some of the "temporary solutions" people have proposed over the years. The statement is an oxymoron. It's either not a solution to the problem, or it's not temporary.

We've got less of those being made now, because I've taken to listing the previous "temporary solutions" every time someone proposes a new one.

Re:All comes down to budget (1)

Gorobei (127755) | more than 3 years ago | (#32330830)

Yeah, that's why the sane firms have rules on accepting gifts. More than $50 or so once in a while, and you need sign-off before you can even show up.

Re:All comes down to budget (4, Funny)

Cryacin (657549) | more than 3 years ago | (#32330878)

Yeah, that's why the sane firms have rules on accepting gifts.

Yes, and both of them have never looked back!

Re:All comes down to budget (0)

Anonymous Coward | more than 3 years ago | (#32331148)

$50? Our limit's $25!

The VMware rep gave me a backpack with their logo on it and I looked at him and said "this is worth less than $25 right?" while slowly and pointedly nodding.

Note that this wasn't a nice backpack, just a typical nylon thing.

Re:All comes down to budget (5, Informative)

eln (21727) | more than 3 years ago | (#32329748)

The problem is not with kludges themselves, but with the fact that IT management does not stress documentation and proper change control procedures enough. If a kludge works, is documented, was implemented with proper change controls, and can be repeated, is it really a kludge anymore? IT has to screw around with stuff to make it work, that's what they (we) get paid for. If all we ever had to do was click on an install button and have everything work perfectly from there, what would be the purpose of an IT department at all? Off-the-shelf software and hardware can never be made to work perfectly for everyone's requirements. IT folks are paid to get non-unique components to work for unique requirements.

The problem is not with these fixes, it's that nobody ever documents what they did, and documentation is not readily available when needed. So, these kludges become tribal knowledge, and people only know about them because they were around when they were implemented or they've heard stories. When this happens, these wacky fixes can come back and bite you in the ass later when something mysteriously crashes and no one can get it to work like it did because nobody remembers what was done to make it work before. As people come and go, and institutional knowledge of older systems slowly erodes, we end up in a situation where everyone thinks the current system is crap, nobody knows why it was built that way, and everyone figures the only way out is to nuke the site from orbit and start over. The trick is keeping it from getting to that point.

Of course, nobody likes jumping through all these hoops like filing change control requests or writing (and especially maintaining!) documentation, so it gets dropped. IT management is more worried about getting things done quickly than documenting things properly, so there's no incentive for anyone to do any of it. Before long, you get a mass of crap that some people know parts of, but nobody knows all of, and nobody knows how or where to get information about any of it except by knowing that John Geek is the "network guru" and Jane Nerd is the "linux guru".

We will never get hardware and software that works together exactly the way we want them to. We will always have to tweak things to get them to work right for us. Citing lack of budgets or bug-ridden software may be perfectly valid, but those problems are never really going to be solved. Having our own house in order does not mean fixing all the bugs or being able to refresh our technology every 6 months. Having our own house in order means we know exactly what we did to make each system work right, we can repeat what we did, and everyone knows how to find information on what we did and why.

Re:All comes down to budget (3, Interesting)

mlts (1038732) | more than 3 years ago | (#32329934)

Isn't this taught to death in ITIL 101 that every MBA must go through in order to get their certificate in an accredited college? It sort of is sad that the concepts taught in this never hit the real world in a lot of organizations. Not all. I've seen some companies actually be proactive, but it is easy for firms to fall into the "we'll cross that bridge when we come to it" trap.

Re:All comes down to budget (4, Insightful)

Vellmont (569020) | more than 3 years ago | (#32330508)


If a kludge works, is documented, was implemented with proper change controls, and can be repeated, is it really a kludge anymore?

Yes.

You've either don't know what a kludge is, or don't have enough ability to see how fixing things or implementing something the wrong way can really be a horrible mistake that feeds on itself and creates other mistakes. Kludges aren't something you can simply document around. The rest of your post isn't really worth responding to, since it makes the false assumption that kludges are simply poorly documented behavior. If that's the worst you've seen, you're lucky.

Re:All comes down to budget (1)

antirelic (1030688) | more than 3 years ago | (#32331532)

I wish I wouldnt have blown all my mod points today or I would have modded you up. Your dead right. Implementing something the wrong way is similar to building a mansion on the sand. Understanding the sheer destructive force of a kludge is what separates the senior engineers (and sysadmins in particular) from the rest. This is why you get a crusty veteran shooting down all your bright ideas without ever really explaining it. Because its a game of chess, kludges lead to checkmates... but only the pro's can see that far ahead.

Just how much documentation can you read? (5, Insightful)

hsthompson69 (1674722) | more than 3 years ago | (#32330934)

The problem with the whole idea of "if we only had enough documentation and change control" is that it becomes a non-trivial event to actually read through the documentation. Let's take an imaginary system that's been in production for 5 years...assume every last drib and drab of change has been documented...now you've got a 2000 page document and several hundred change records that tell you *everything*. Except, when it comes right down to it, mastering that 2000 pages of documentation and all the changes made afterwards is a months if not years long project - hardly effective for dealing with production problems that need to be solved in minutes or hours.

The illusion being perpetrated here is that people are interchangeable, and if you just have enough documentation, you can replace Mr. Jones with 20 years of hands on experience with the system with Mr. Vishnu living in Bangalore (or even Mr. Smith in the next cube, for that matter), with a net cost savings.

Now, I'm not saying documentation is a bad thing -> lord knows, it helps to have a knowledge base you can search...but knowing what to search for is knowledge you only get by real world experience with maintaining a production system. This is not digging ditches, boys and girls, this is skilled, if not essentially artistic labor.

Simply put, people matter more than process.

Re:Just how much documentation can you read? (1)

AchilleTalon (540925) | more than 3 years ago | (#32331616)

I agree completely with this comment. I haven't seen yet any in-house documentation database that is valueable at all. Writing good documentation is not an easy task and the guy fixing problems usually just don't know how to write a simple plain non-technical letter. Imagine now he is having to write technical stuff for the next guy knowing nothing about the system or knowing too few about it to be able to ready a few short notes.

Re:All comes down to budget (1)

dido (9125) | more than 3 years ago | (#32331642)

It's not so much that no one wants to do them (although this is true to a degree), but that most people, especially managers, never consider that making quality documentation and obeying procedures takes time and effort, sometimes almost as much as developing the software itself. IT professionals are almost always under unrealistic deadlines to get the project finished and working and so these "non-essentials" are among the first things to fall by the wayside when the project meets a looming deadline. This is the main reason people don't want to do them: they don't directly contribute to the "real" work that is visible to the end users of the project, and as such are considered expendable. Eventually the whole project becomes nothing but kludge piled upon kludge, and the only fix is to start over, again repeating the same mistakes that brought the system to that pass.

Re:All comes down to budget (2, Insightful)

BigSlowTarget (325940) | more than 3 years ago | (#32329962)

To be blunt, most IT departments act like cost centers and don't provide any strategic value. Business units help by shorting the budgets and whining about band-aid technology instead of seeing how IT can build the business. It takes an exceptional move by IT or amazing insight from a business unit to raise IT above the slog and allow it to provide a competitive advantage to the business units. Projects that do this get firehosed with funding.

Consultants take advantage of this catch 22 situation when they sell new projects. It lets them get the new implementations and cutting edge development. This situation also causes application oriented mini-IT organizations to pop up in the business units from time to time. That, in turn, causes more headaches for central IT.

Re:All comes down to budget (3, Interesting)

Grishnakh (216268) | more than 3 years ago | (#32330108)

I don't think budget is a problem at all here. The problem as described by the article is with vendor-provided software being crufty and having all kinds of problems. The author even mentions that normal free-market mechanisms don't seem to work, because there's little or no competition: these are applications used by specific industries in very narrow applications, and frequently have no competition. In a case like this, it doesn't matter what your budget is; the business requirement is that you must use application X, and that's it. So IT has a mandate to support this app, but doing so is a problem because the app was apparently written for DOS or Windows 95 and has had very little updating since then.

The author's proposed solution is for Microsoft to jettison all the backwards-compatibility crap. We Linux fans have been saying this for years, but everyone says we're unrealistic and that backwards compatibility is necessary for apps like this. Well, it looks like it's starting to bite people (IT departments in particular) in the ass.

Re:All comes down to budget (1)

Lifyre (960576) | more than 3 years ago | (#32330176)

You should strive for SOME backwards compatibility not honestly supporting more than a generation or two back is too far. Especially when you have a major OS revision (Win2k and Vista come to mind) to act as a breaking point.

Re:All comes down to budget (2, Insightful)

Grishnakh (216268) | more than 3 years ago | (#32330602)

To be honest, though, Linux is generally very good at backwards compatibility if you statically-link everything when you compile (as is frequently the case with commercial software). The Linux system calls never change, except to add new ones once in a while, so it should be very rare that something doesn't run.

Of course, if something is compiled with dynamic links, this isn't the case, as many of the dependencies will change over the years, but that's why static-linking is available, to avoid this problem. Dynamic linking is better for software that's distributed by the distro, as they can make sure all dependencies in place. Boxed commercial software doesn't have this luxury, so it needs to stick with static linking.

The main place where people complain a lot about Linux's backwards compatibility is with drivers, but that's a design decision. In Linux, drivers are supposed to be included with the kernel. If you don't want to do that, then you'll suffer the consequences. Application software doesn't have this problem as it doesn't link directly to the kernel.

Re:All comes down to budget (1)

Ritchie70 (860516) | more than 3 years ago | (#32331160)

They are slowly doing this, they just have to do it slowly.

For example, 16-bit programs don't work on 64-bit Windows.

Re:All comes down to budget (3, Insightful)

almitchell (1237520) | more than 3 years ago | (#32330192)

That is very true. Unless you are working for a highly visible technology company or high-profile corporation, most companies simply want you to keep the mess you've got going, no matter if it meas bandaids and soldering irons. Over the course of my 20 years career at four different companies, and from talking with colleagues, it is much the same story - the steering committee says, we initially invested hundreds of thousands of dollars, but we sure as hell aren't overhauling and starting over. The best boss I ever had did what I now refer to as guerrilla network administration. We had an aging infrastructure supporting hundreds of banks that we wanted to migrate off Wintel based systems, because they were end of lifing and we were sick of holding it together with bandaids and baling wire. It kept breaking, we sysadmins were sick of sitting through post-mortem meetings, and we were sick of upper management's refusal to acknowledge that it wasn't that the sysadmins sucked, it was that we didn't have a magic wand to keep the old nag on its feet to pull the plow. We repeatedly added a new plan into the budget to change out the system, and just as repeatedly got turned down because of cost. One day we had what was a near catastrophic hardware failure on one of those systems and we had to wait three days to get the parts on it because no one supported it any more. My boss told us to let it sit, and we did, chewing Tums the whole time because it's not like he was the one who would get fired over it. When upper management asked why we were still down after 24 hours, he took a PO in to them and said that when they signed off on new and functional systems, the problem would be fixed. When they balked, he asked them was the cost of a new system really so exorbitant compared to the manhours and effort it took to nurse a sick system well beyond its years. They signed, we had it changed out in two months, and we went another four years without so much as a hiccup. If he hadn't moved out of state I would have followed that man anywhere.

Re:All comes down to budget (4, Insightful)

HangingChad (677530) | more than 3 years ago | (#32330538)

the IT department is treated as pure cost instead of something that provides strategic value.

I can't count the times I've gone in somewhere and saw major deficiencies in their IT infrastructure. I mean really bad, O-M-G size problems. And when you point them out they act like you're trying to pad your billing. Just fix whatever isn't working that day. One of them was a doctors office.

Imagine if their patients acted that way. I don't care if I have cancer, just remove that lump in my underarm.

That's what you get when the problem is dictating the solution.

Re:All comes down to budget (1)

antirelic (1030688) | more than 3 years ago | (#32330586)

All comes down to bad management. And what I mean by bad management, is management that does not understand how intricate IT is to their bottom line, and does not plan accordingly. IT is just as important as accounting, or HR. If you neglect it, the problems will cost you down the road.

It doesn't matter what line of business you are in. If you do not understand the role has to play in your company, then you need to hire someone who can help you figure that out, or simply be replaced by someone who can.

Re:All comes down to budget (1)

Bing Tsher E (943915) | more than 3 years ago | (#32331578)

In most organizations, the IT department is treated as pure cost instead of something that provides strategic value. These IT departments have no chance of getting a budget approved that will allow them to "start over" on any part of their implementation; hence the constant onslaught of temporary fixes and patches.

In most organizations, IT is an infrastructure thing, like the file cabinets, cubicle walls, and the pencil sharpeners. So management doesn't think in terms of 'strategic value' regarding IT. They're in the business of producing whatever widgets or other product their company is in business to produce.

IT people tend to think about IT for the sake of IT. What color file cabinets do you think we should order next time?

Kludges are short-time fixes and long-time problem (0)

Anonymous Coward | more than 3 years ago | (#32329616)

If you don't invest the time to do something properly the first time*, you've increased the cost several times over since the wrong decisions will progress & amplify into even worse decisions & implementations meaning the cost to get back to a correct solution will only increase. And since you'll accrue dependencies on your wrong decisions, the costs will just amplify more.

* Doesn't mean you solve the problem the right way the first time. But it does mean that you are able to identify when things go beyond implementation problems into architectural & process problems & solve those before they become even bigger problems.

Re:Kludges are short-time fixes and long-time prob (4, Insightful)

Grishnakh (216268) | more than 3 years ago | (#32330132)

It doesn't look like "doing it right the first time" is an option here. RTFA. They're talking about vendor applications being crappy and crufty, and IT departments being required to support them. The IT department didn't pick the app, and isn't allowed to not support it. They can't switch to another app (usually apps like this have little or no competition, and they're probably locked-in anyway).

So there's really nothing they can do but complain as long as they're required to support some shitty application on the latest version of Windows, as these are the requirements set down by upper management.

Re:Kludges are short-time fixes and long-time prob (1)

dykmoby (830547) | more than 3 years ago | (#32330670)

... The IT department didn't pick the app, and isn't allowed to not support it....

And they expect you to integrate it with all the other apps that other departments/teams picked without consulting IT for the past 10 years

I don't believe in a lot of things (5, Funny)

Culture20 (968837) | more than 3 years ago | (#32329636)

...but I believe in Duct Tape.
As long as your backup and tertiary machines have different kludges keeping them running, there's no problem...

As a non-developer, this is what I see (4, Insightful)

Em Emalb (452530) | more than 3 years ago | (#32329650)

Maintaining code is boring.

Everyone wants to work on the latest and greatest stuff, no one wants to maintain or even release patches.

It sucks, especially since it isn't limited to just software development.

I've seen companies where their "core switch" was a Cisco 2548. This wasn't 10 years ago, this was last year! Unreal.

Re:As a non-developer, this is what I see (4, Interesting)

drachenstern (160456) | more than 3 years ago | (#32329718)

As a dev, what's the problem with a 24 port gigabit switch as the "core" on a medium sized office? Aside from the fact that 10Gb is becoming popular (has become popular?) in the datacenter? Most desktops are only at the 1Gb level (and most users at below 100Mb), and most inbound internet pipes are much smaller. I don't understand the downfall here.

Can you elaborate?

Re:As a non-developer, this is what I see (2, Interesting)

Em Emalb (452530) | more than 3 years ago | (#32329876)

No redundancy, is the biggest one. No real layer 3 switching is another.

Re:As a non-developer, this is what I see (1)

dbIII (701233) | more than 3 years ago | (#32330612)

Redundancy? In such a small setting if thing dies you either go out and buy a single easily available item or more likely you dust off the old dumb 100mb/s switch it replaced and pop it in the rack.
Layer 3 switching is nice but has only very recently become cheap enough to justify in many settings and it really isn't needed much. In a small office if somebody brings in something which acts as a DHCP server the sysadmin simply wanders around with a blunt instrument until the culprit is found, and unused ports are not even patched in. Security features are really more important when you can't easily see what is going on everywhere - hence something a lot bigger than what you are smirking about here.

Re:As a non-developer, this is what I see (2, Informative)

Em Emalb (452530) | more than 3 years ago | (#32330698)

The network it was running was not a small network. Not at all. It was a travesty that this poor switch was running the network. Well over 200 devices plugged into other 2548s all bridged back to the poor "core" switch.

Re:As a non-developer, this is what I see (1)

drachenstern (160456) | more than 3 years ago | (#32331486)

Well so 200 devices still sounds like a small network to me. But this is the first reply I've read since posting earlier, let's see what else is on the thread.

Re:As a non-developer, this is what I see (2, Interesting)

oatworm (969674) | more than 3 years ago | (#32329902)

Ditto this - plus, in a medium-sized office, you're probably not getting 10x24Gb/sec out of your server infrastructure anyway. Your network is only as fast as the slowest component you rely upon; at 10Gb/sec, you're starting to bump into the limits of your hard drives, especially if you have more than a handful of people hitting the same RAID enclosure simultaneously.

Re:As a non-developer, this is what I see (4, Interesting)

JerkBoB (7130) | more than 3 years ago | (#32329914)

As a dev, what's the problem with a 24 port gigabit switch as the "core" on a medium sized office?

If all you've got is 24 hosts (well, 23 and an uplink), then it's fine. I suspect that the reality he's alluding to is something more along the lines of multiple switches chained together off of the "core" switch. The problem is that lower-end switches don't have the fabric (interconnects between ports) to handle all those frames without introducing latency at best and dropped packets at worst. For giggles, try hooking up a $50 8-port "gigabit" switch to 8 gigabit NICs and try to run them all full tilt. Antics will ensue... The cheap switches have a shared fabric which doesn't have the bandwidth to handle traffic between all the ports simultaneously. True core switches are expensive because they have dedicated connections between all the ports (logically, if not physically... I'm no switch designer), so there's no fabric contention.

Re:As a non-developer, this is what I see (2, Insightful)

Anonymous Struct (660658) | more than 3 years ago | (#32330642)

Absolutely nothing. A 24 port gigabit switch makes a great foundation for a small to medium-sized network with typical business use. It's a stretch to call it a 'core', but anybody who tells you that you need some kind of crossbar fabric chassis switch at the center of your average branch office is just trying to sell you hardware and service contracts.

Re:As a non-developer, this is what I see (1)

David_Hart (1184661) | more than 3 years ago | (#32331590)

First, the Cisco 2548 is a 48 x 100Mb + 2x 1Gb GBIC. Which means that it is old tech. My guess is that it may only support ISL and not Dot1q trunking, and likely only supports VTP 1.0. In other words, it's really old tech, has no support, and for which you likely can only find parts on eBay, time to upgrade. In fact, our company is in the middle of a hardware replacment cycle. Some of our guys didn't have the right cables for one site and tried to connect some 2924 C XL switches to a new 3750 stack. They just didn't want to talk. I'm guessing that a firmware upgrade would have fixed the problem (latest firmware was 8 years newer than what was on the box), but they decided not to bother and to get the right cables in the morning.

Second, it depends on what you call a medium size company. To me a medium size company would be 1000 to about 2500 users. In that case, you need a decent layer 3 switch at the core. This usually means a pair of 6500 switches for the core and 4500 switches for access.

And just to be clear, I am not a huge Cisco proponent. I believe in the right tool for the job. I also beleive that Juniper has better products in certain areas for 1/3 the price. In fact, I believe that Juniper will be eating Cisco's lunch in the small and medium size environments over the next few years. It is one of the reasons why Cisco has been working to diversify.

David

Re:As a non-developer, this is what I see (1)

sexconker (1179573) | more than 3 years ago | (#32329864)

I've seen companies where their "core switch" was a Cisco 2548. This wasn't 10 years ago, this was last year! Unreal.

Solid product lasts a long time and does the job.

Is this a problem in your Bizarro world?

Re:As a non-developer, this is what I see (1, Troll)

Em Emalb (452530) | more than 3 years ago | (#32329984)

Is this a problem in your Bizarro world?

Yes, in my "bizzaro world", most people call it the real world, using an edge switch as your core or "main" switch in your network is considered a very very bad thing.

Re:As a non-developer, this is what I see (0)

Anonymous Coward | more than 3 years ago | (#32330932)

In the real world for most people, IT infrastructure is taken for granted and underfunded, so using an edge switch as the core switch is all they have. If they have a decent brand that doesn't crap out after a year it will get the job done. This is especially true in small companies with 50-100 employees.

I think your idea of real world only applies to some people, not most.

Re:As a non-developer, this is what I see (1)

Bing Tsher E (943915) | more than 3 years ago | (#32331674)

Solid product lasts a long time and does the job.

Is this a problem in your Bizarro world?

Maybe he sells networking hardware for a living.

Re:As a non-developer, this is what I see (1)

pilgrim23 (716938) | more than 3 years ago | (#32330274)

Its wonderful to ride the developer ship. But once the brilliant code is down on silicon, and runs into reality it must be patched. Patching is so humdrum, so tedious. No admiring fans, just plodding day after day finding that routine that seems to always call the variable from nowhere "Object not set to an instance of an object". Frank Loyd Wright was a great architect. People marvel at his design. Few know the name of the roofer who has to repair the design flaw that makes every Wright Roof Leak....

Re:As a non-developer, this is what I see (1)

Cryacin (657549) | more than 3 years ago | (#32330904)

Few know the name of the roofer who has to repair the design flaw that makes every Wright Roof Leak....

You're not doing it Wright!

I was torn between modding this up and commenting. (3, Insightful)

tlambert (566799) | more than 3 years ago | (#32330618)

I was torn between modding this up and commenting.

I picked commenting.

This statement:

Everyone wants to work on the latest and greatest stuff, no one wants to maintain or even release patches.

is very, very true. We (Apple) have a hard time getting applicants who want to do anything other than work on the next iPhone/iPad/whatever. Mainline kernel people are difficult to hire, even though the same kernel is being used on the iDevices as is being used on the regular Macs. Everyone wants to work on the new sexy. For some positions, that works, but for most of them, you have to prove yourself elsewhere before you get your shot.

I think that, for the most part, we see the same thing in marketing for higher education (with the exception of one track, one of the universities I went to has become a diploma mill for Flash game programmers; sadly, I would not hire recent graduates from there unless they have an experience track record). There are video game classes at most universities, but while it might be sexy, you are most likely not going to be getting a job doing video games, 3D modeling for video games, or anything video game related, really, unless you get together with some friends and start your own company, and even then it's a 1 out of 100 chance of staying in business.

I don't really know how to address this, except by the people who think they are going to be the next great video game designer remaining unemployed.

-- Terry

Re:I was torn between modding this up and commenti (2, Funny)

by (1706743) (1706744) | more than 3 years ago | (#32330944)

We (Apple)...

...

...has become a diploma mill for Flash game programmers; sadly, I would not hire recent graduates from there...

Sounds about right to me!

Odds of +1 funny over flamebait/troll/offtopic: slim to none. I just hope your 2548 dies before you can mod me down!

Vendors ... not their customers. (0)

Anonymous Coward | more than 3 years ago | (#32329652)

Paul was mostly talking about the software vendors peddling completely outdated designs, where presumably the product is something they that are selling (i.e. s/b a profit center). The problem (in this case) does not reside with the users (customers) of such products, who are probably blissfully unaware of the sort of c**p that they are dealing with.

Take responsibility and stop the magical thinking (3, Informative)

DragonWriter (970822) | more than 3 years ago | (#32329702)

The constant need to apply temporary fixes that end up becoming permanent are fast pushing many IT infrastructures beyond repair. Much of the blame falls on the products IT has to deal with.

Well, sure, IT departments place the blame there. The problem, though, is not so much with the products that IT "has to deal with" as with the fact that IT departments either actively choose the penny-wise-but-pount-foolish course of action of applying band-aids rather than dealing with problems properly in the first place, or because -- when the decision is not theirs -- they simply fail to properly advise the units that are making decisions of the cost and consequence of such a short-sighted approach.

When IT units don't take responsibility for assuring the quality of the IT infrastructure, surprisingly enough, the IT infrastructure, over time, becomes an unstable house of cards, with the IT unit pointing fingers everywhere else.

And yet breaking this 'vicious cycle of bad ideas and worse implementations' by wiping the slate clean is no easy task. Especially when the need for kludges isn't apparent until the software is in the process of being implemented. 'Generally it's too late to change course at that point.'

If your process -- whether its for development or procurement -- doesn't discover holes before it is too late to do anything but apply "temporary" workarounds, then your process is broken, and you need to fix it so you catch problems when you can more effectively address them.

If your process leaves those interim workarounds fixes in place once they are established without initiating and following through on a permanent resolution, then, again, your process is broken and needs fixed.

You don't fix the problems with your infrastructure that have resulted from your broken processes by "wiping the slate clean" on your infrastructure and starting over. You fix the problems by, first, improving your processes so your attempts to address the holes you've built into your infrastructure don't create two more holes for every one you fix, then by attacking the holes themselves.

If you try to through the whole thing out because its junk -- blaming the situation on the environment and the infrastructure without addressing your process -- then:

(a) you'll waste time redoing work that has already been done, and
(b) you'll probably make just as many mistakes rebuilding the infrastructure from scratch as you made building it the first time, whether they are the same or different mistakes.

Magical thinking like "wipe the slate clean" doesn't fix problems. Problems are fixed by identifying them and attacking them directly.

Re:Take responsibility and stop the magical thinki (4, Interesting)

FooAtWFU (699187) | more than 3 years ago | (#32329770)

"they simply fail to properly advise the units that are making decisions of the cost and consequence of such a short-sighted approach."

In the defense of IT, those people they're trying to advise aren't always the best at taking advice. (But then again, neither are IT admins always the best at giving it.)

Re:Take responsibility and stop the magical thinki (4, Insightful)

Bunny Guy (1345017) | more than 3 years ago | (#32330334)

I'm going to tackle some of the conceptual problems that are hinted at above, which is usually where the difficulties lie, usually in trying to use the wrong software and expecting to somehow "make everything better" if you just make it work "my way" - the true "Magical Thinking".

I tend to agree with your conclusions, "wipe the slate clean" is a drastic action. I disagree with some of the approach you use to arrive at them:

a.) Problems are solved by people being invested in solving them, not process. This requires the antithesis of "Units" - Ownership; Ownership in the company, Ownership of the mission, and a direct heart felt connection to the success of the company. Until you have staff, from the CEO down, that own problems, from the mess in the coffee room to server down time, you will have a "business house of cards" no matter how good the process. In fact, most of the time, fixing things involves re-writing and/or reconsidering process - usually starting with asking the question - "Do we really need that?"

b.) Sometimes you really do have a train wreck on your hands. If you have mastered a.) b follows almost effortlessly, because now, you can *talk* about this behemoth that is eating your company and everybody sees the discussion for what it is, not empire building or managerial fingerprinting.

when you run into a train wreck - assess your tech problem - is the fix easily found? Are your processes using the software at cross purposes? if so, which is cheaper to fix? No amount of bug fixing will repair using the wrong software. It won't even fix using the right software in the wrong way.

In the end, re-asses often, be frugal, not cheap, if it truly is a requirement to run your business, buy the most appropriate. If you've made the mistake of buying a Kenworth long hauler when you needed 3 old UPS trucks - admit it, sell it back, take your loss and get what you really need.

Thats not "magical thinking" it's just common sense.

Re:Take responsibility and stop the magical thinki (2, Insightful)

DragonWriter (970822) | more than 3 years ago | (#32330534)

Problems are solved by people being invested in solving them, not process.

Both are, IMO, essential, which is while while I pointed at particular areas of process, my big picture message was about IT shops taking "responsibility for assuring the quality of the IT infrastructure."

Neglect of process is a symptom of people not being invested in solving problems that leads to bad results on its own, but even a good (nominal) process isn't going to work well if people aren't invested.

This requires the antithesis of "Units" - Ownership; Ownership in the company, Ownership of the mission, and a direct heart felt connection to the success of the company.

I prefer "responsibility"; "ownership" is, IMO, misapplied here. (Though, arguably, one of the reasons people do not take responsibility is because they don't, in fact, have ownership -- but ownership is a material relationship, and responsibility is the relevant attitude.)

But I think in substance we generally agree.

Until you have staff, from the CEO down, that own problems, from the mess in the coffee room to server down time, you will have a "business house of cards" no matter how good the process. In fact, most of the time, fixing things involves re-writing and/or reconsidering process - usually starting with asking the question - "Do we really need that?"

You kind of contradict yourself there: if fixing things usually requires changing the process, then "how good the process" is obviously has fairly direct bearing on success. The key thing is that processes aren't good (or bad) in a vacuum, they are good or bad based on the effects they have in your organization, in acheiving your mission; the same nominal process that is good for a group of people when considered against one mission is going to be bad for the same group of people when considered against different goals, and the same process that is good for one group of people with a given mission will suck for another group of people with the same mission, because people matter.

Re:Take responsibility and stop the magical thinki (1)

Bunny Guy (1345017) | more than 3 years ago | (#32330868)

I prefer "responsibility"; "ownership" is, IMO, misapplied here. (Though, arguably, one of the reasons people do not take responsibility is because they don't, in fact, have ownership -- but ownership is a material relationship, and responsibility is the relevant attitude.)

But I think in substance we generally agree.

We do, I think the difference is that my experience has been fixing projects starting from a technical complaint to an outside organization and helping those in an IT/Technology organization drive changes up thru their organization, often into the CEO space when needed. From your choice of wording, I suspect, your experience in this might start higher up the product chain.

Until you have staff, from the CEO down, that own problems, from the mess in the coffee room to server down time, you will have a "business house of cards" no matter how good the process. In fact, most of the time, fixing things involves re-writing and/or reconsidering process - usually starting with asking the question - "Do we really need that?"

You kind of contradict yourself there: if fixing things usually requires changing the process, then "how good the process" is obviously has fairly direct bearing on success. The key thing is that processes aren't good (or bad) in a vacuum, they are good or bad based on the effects they have in your organization, in acheiving your mission; the same nominal process that is good for a group of people when considered against one mission is going to be bad for the same group of people when considered against different goals, and the same process that is good for one group of people with a given mission will suck for another group of people with the same mission, because people matter.

I probably expressed that poorly. To put it another, hopefully more correct way - For organizations you can help (there are plenty that are unreceptive to this kind of help) you have to have to start with the culture. Identify those who have true involvement, are willing to risk take, have decision power... and get all of them committed before you can fix the process, which then finally lets you help fix the bugs/tech. Lather, rinse, repeat till organization functions.

In particularly ill organizations, there is no way to separate these items (or the combatant parties :-)) long enough to obtain a fix. You have to wait till the financial realities drive some common sense into the organization. Sadly, for many businesses, this is too late

Re:Take responsibility and stop the magical thinki (3, Insightful)

AK Marc (707885) | more than 3 years ago | (#32330992)

The problem, though, is not so much with the products that IT "has to deal with" as with the fact that IT departments either actively choose the penny-wise-but-pount-foolish course of action of applying band-aids rather than dealing with problems properly in the first place, or because -- when the decision is not theirs -- they simply fail to properly advise the units that are making decisions of the cost and consequence of such a short-sighted approach.

I've found the problem to almost always be the last thing listed. It's the contractor syndrome. "If you give me $1,000,000 now, I'll save your $500,000 a year for the rest of the time you'd have used that." Well, they think you are lying. They think that you wouldn't actually save the $500,000 a year, but would take the $1,000,000 this year and add it to your budget as a permanent line item, costing them $1,500,000 a year, rather than saving $500,000.

You can blame the IT director/manager/CIO/whatever for not being convincing enough, but there seems to be a pattern where people bid low then have massive overruns where the highest bidder would have been cheaper. As such, the people the IT person is talking to are often so jaded they don't trust anyone with price estimates.

When IT units don't take responsibility for assuring the quality of the IT infrastructure, surprisingly enough, the IT infrastructure, over time, becomes an unstable house of cards, with the IT unit pointing fingers everywhere else.

And when the IT units have the responsibility, but not the authority to fix things, what then? Most all places tie the hands of IT then complain when the solution isn't perfect.

Don’t patch bad code - rewrite it (4, Interesting)

D4C5CE (578304) | more than 3 years ago | (#32329740)

Don’t patch bad code – rewrite it.

Kernighan & Plauger
The Elements of Programming Style [wordpress.com]
2nd edition, 1974 (exemplified in FORTRAN and PL/1!)

Re:Don’t patch bad code - rewrite it (4, Insightful)

eggoeater (704775) | more than 3 years ago | (#32330684)

I couldn't agree more, but that's very expensive and very very dangerous. Why? Two factors:
1. Rewriting means rethinking; most legacy code is functional and is usually rebuilt in OOP. Whenever you rethink how something works it tends to change the entire behavior to say nothing of all the new bugs you'll have to hunt down. You're customers will definitely notice this.

2. Scope creep!! Rebuilding it? Why not throw in all that cool functionality we've been talking about for the past 10 years but couldn't implement because the architecture couldn't handle it. You get the idea.

Want an example? Netscape 5 [joelonsoftware.com]

implemented (2, Insightful)

convolvatron (176505) | more than 3 years ago | (#32329742)

i guess its ok that the sysadminds coopted the work 'implemented' where one would normally
say 'installed'

but that kind of leaves the actual implementors without a word now

and in this particular usage, its kind of odd, because usually the best time to
find and fix these problems is exactly when its being implemented, rather than
when its being installed

House of Tape (1)

Renraku (518261) | more than 3 years ago | (#32329756)

The whole system is held together by superglue and duct tape because no one wants to pay for a replacement part. This goes for both physical and software analogies.

Written for a P-II 300Mhz? (5, Funny)

damn_registrars (1103043) | more than 3 years ago | (#32329764)

Wait, you mean there have been newer and faster processors released since then? So Mordac really has been hiding something from me...

Placebo effect - bandaids and kisses (1)

peterofoz (1038508) | more than 3 years ago | (#32329766)

With regard to band-aids and kisses, it looks to me like IT benefits from the same placebo effect as little girls do:

Adventures in Mommyville [gazettextra.com]

It a similar effect as when management brings in a new product or consulting group that is the "silver bullet" that will solve all the problems.

pay off your credit cards? (5, Informative)

Matthew Weigel (888) | more than 3 years ago | (#32329788)

This the essence of technical debt [wikipedia.org]. Whether you're programming or deploying IT infrastructure, it's inescapable that sometimes you're going to have to include kludges to work around edge conditions, a vocal 1% of your users, or whatever. These kludges are eyesores, and fragile, but they're also as far as you could go with the time and budget you had.

Sometimes, accruing debt like this enhances your liquidity and ability to respond to change, so avoiding all kludges introduces other more obvious costs that slow you down and make you seem unresponsive to users or customers. But you can't just go on letting your debt grow all the time and not eventually come up technically bankrupt. Let it grow when you have to, but just as importantly make time to pay it down. A lot of this stuff can be paid down a little at a time, as you come across it a few months later. The pay-off if you're vigilant is that the next ridiculously urgent fix to that system can often be handled much more easily, without dipping down further... with patience and attention to maintaining this balance, you can reduce your technical debt and make the whole system hum.

The downside is that there isn't a quick fix when you find yourself deep in technical debt. You can't just spend all your time reducing it; your highest aspiration at that point should be maintaining the level of technical debt, rather than letting it grow, but it's generally been my experience that altering the curve of debt growth even a little can set you on the right path.

Re:pay off your credit cards? (1)

Len.Wright (1817002) | more than 3 years ago | (#32329940)

Brilliant. I wish I had said this, but in any event, I do follow it and fully believe in what you're say.

Re:pay off your credit cards? (1)

Rob Riggs (6418) | more than 3 years ago | (#32330200)

The downside is that there isn't a quick fix when you find yourself deep in technical debt.

Actually, there is a very simple and quick fix when technical debt becomes overwhelming. Often it's the only fix. CTOs and CIOs do it all the time.

Quit.

Often that is the only way to catalyze the change required for companies to begin reducing their technical debt.

I've seen similar issues with hardware (1)

peacefinder (469349) | more than 3 years ago | (#32329838)

Long ago, at one customer, the desktops grew weak. So they hit upon the idea of using a Windows terminal server to prop up their aging desktops. By gum, that worked so well that they replaced their desktops with thin clients throughout over the next year. Now, a few years after that, their terminal server handles everything from Solidworks viewers to web browsers and it's sagging under the load... and out of warranty to boot. Time to get a new server... only the economy has hurt their business terribly and they currently can't afford one.

In short, they're screwed. Too tied to Windows to easily jump platforms, too tied to thinstations to easily return to the desktop environment, and too broke to maintain it.

Re:I've seen similar issues with hardware (1)

oatworm (969674) | more than 3 years ago | (#32329986)

Ran into a similar situation with an old client of mine, only with more "hilarity" - basically, they had a consultant tell them that they could have a dirt cheap server, a bunch of thin clients, and save money. Queue one Pentium 4-powered server with software mirrored PATA(!!!) drives serving as a terminal server for over 30 people. Queue one Pentium 4-powered server quickly succumbing to the heat and wear that was generated from such a load. Queue one confused customer that was wondering why the company I worked for was suggesting they buy a server that cost three times as much (i.e. one with a SAS-driven RAID 5 and more RAM) as the one they "just got a year or two ago". Queue them telling us to get lost.

Queue them getting bought out by a competitor a year or two later, followed by the competitor rolling the former customer's IT into the competitor's IT infrastructure, since the competitor actually knew how to spec for growth and maintenance. I don't know who won on that one, but... yeah. Think about it.

Re:I've seen similar issues with hardware (0)

Anonymous Coward | more than 3 years ago | (#32330062)

The only way they can really fix that is a graduated move, where the Solidworks users are moved to standalone machines, while the rest of the people remain on the creaky terminal server.

The problem is that this is a misapplication of technology. A call center where people only run an app that takes down info and updates accounts is perfect for a thin client setup. However, once people are starting to do more than something which essentially a 3270 terminal does, they need either hardware on their desktops, or dumb monitors/keyboards connected to ClearCube blades.

Re:I've seen similar issues with hardware (1)

timmarhy (659436) | more than 3 years ago | (#32330250)

huh? replacing one terminal server is way cheaper then replacing a lot of desktops. you would just buy a new one, put it side by side with the old one and split the users between the 2 of them to balance the load.

plus there's nothing stopping them buying desktops and having them log into the same domain as the terminal server. your post makes no sense.

Actually (0)

Anonymous Coward | more than 3 years ago | (#32329892)

It's the products that are still based on the old, supposedly 'crusty' C/native instruction code that I look for. Obviously, some products are dependent on libraries that have changed and are thus no longer compatible, but those that do still work tend to work VERY WELL on modern hw.

The modern stuff that's supposedly superior is slow and bloated as hell.

Solution is obvious - Linux (3, Interesting)

seyfarth (323827) | more than 3 years ago | (#32329960)

From the original message we read that the "code was also written to interact with a completely different set of OS dependencies, problems, and libraries." This seems to imply that the IT organizations are allowing outside interests to dictate the rules of the game. If there were a stable set of operating system calls and libraries to rely on, then the software vendors would have an easier time maintaining software. I recognize that Linux changes, but the operating system calls work well and API is quite stable. I have used UNIX for a long time and I have compiled programs from 25 years ago under Linux. There have been some additions since then, but the basics of Linux work like the basics of UNIX from 25 years ago.

At present there are some applications available only on Windows and some only on Windows/Mac OSX. This might be difficult to change, but going along with someone's plan for computing which is based on continued obsolescence seems inappropriate. At least those who are more or less forced by software availability to use Windows should investigate Linux and negotiate with their vendors to supply Linux solutions.

Computers are hard to manage and hard to program. It is not helpful to undergo regular major overhauls in operating systems.

Re:Solution is obvious - Linux (2, Insightful)

Anonymous Coward | more than 3 years ago | (#32330206)

I've been saying exactly the same thing since about 1994---since I got into linux thing. Every program I wrote since just "works" without changes (granted, I don't write many gui apps; mostly data management stuff). My Windows counterparts (same corp, doing semi-related apps) have to release a "new version" every time .net is patched---or something along those lines. Your environment shouldn't make your things break or not work right.

Re:Solution is obvious - Linux (1)

Grishnakh (216268) | more than 3 years ago | (#32330218)

The article seems to be about vendor-provided apps that were written for creaky old versions of Windows. Windows, as you probably know, has a whole slew of APIs, and some work better than others. Of course, MS keeps them all around (though not all in good working order) because of their famed backwards compatibility that supposedly lets people run all their old software from 1995 (even though in practice it doesn't work well).

However, "negotiating" with vendors to supply Linux solutions (as great as that would be) isn't really viable. This is really about market sectors where the business needs some weird application which has little or no competition, so the vendor can basically name their price and do what they want. The vendor probably has done very little to update the app in the last decade, because it's a cash cow and there's no competition.

So what is the customer supposed to do? Your "negotiation" idea isn't going to go far: the vendor is going to say, "you will buy our product at whatever price we want, and put up with its crappiness. If you don't like it, go somewhere else! Oh yeah, there is nowhere else to go... hahahaha!!!" So either you deal with the problems, or you don't use the application (and you don't get your business done). That's it. The only other solution is to make your own in-house application, but that may be easier said than done, and many small and medium-size organizations don't have the resources to take on a project like that.

Basically, the IT people are complaining about a situation that has no real solution.

Terry Childs rebuild network and look at him now! (1)

Joe The Dragon (967727) | more than 3 years ago | (#32329972)

Terry Childs rebuild the network and look at him now!

If you start over you may be the only one how know how it's works and then some day the PHB will want to mess with it / replace you with a min wage guy who may not know much about how things are setup and can make it to a big mess.

you know what else is a house of cards? (-1, Troll)

Anonymous Coward | more than 3 years ago | (#32330008)

heterosexuality.

http://www.youtube.com/watch?v=iObGIkqhJJw

it can collapse at any time.

Software = untouchable mentality (3, Insightful)

Stiletto (12066) | more than 3 years ago | (#32330018)

This happens in commercial software development, too. There's this belief (often held all the way up the management chain to the top) that software, even bad software, represents some kind of massive, utterly permanent investment that must never be thrown away and re-written.

I've worked with managers who would think nothing of throwing away a million dollar manufacturing machine to replace it because it's old, yet cling with all their might to ancient software code that represents a similar level of investment.

Re:Software = untouchable mentality (3, Informative)

Ichijo (607641) | more than 3 years ago | (#32330664)

There's this belief (often held all the way up the management chain to the top) that software, even bad software, represents some kind of massive, utterly permanent investment that must never be thrown away and re-written.

Ah yes, the sunk cost fallacy [wikipedia.org].

It's good that windows 64 kills the old 16 bit app (1)

Joe The Dragon (967727) | more than 3 years ago | (#32330030)

It's good that windows 64 kills the old 16 bit apps off.

But it will be a long time before the old windows 32 bit windows 9x code base it gone in many apps.

Re:It's good that windows 64 kills the old 16 bit (1, Interesting)

Darkness404 (1287218) | more than 3 years ago | (#32330092)

How is it good? It leaves the entire internet vulnerable. It pushes people not towards Linux but towards outdated versions of Windows and more or less guarantees the future has 32 bit OSes.

Look at what is keeping people from adopting Linux: Small, niche programs.

With outdated versions of Windows already online, can we afford to push even more people to old, closed, OSes with no future of getting patches?

You have to keep up with the times! (1)

bertok (226922) | more than 3 years ago | (#32330304)

I'm a consultant that gets to see into the IT world of over a hundred organisations, and I see one major failing that companies make that cost them later:

They fail to keep up with the times.

Nowhere is this more evident than the leviathan government organisations still running Novell and Lotus Notes, on barely patched versions of Windows, with the browser restricted to IE6. Every time I see that combination, the word "expensive" comes to mind.

Solutions that used to be the best are now simply redundant, outdated, incompatible, and expensive to maintain. Inevitably, the whole thing becomes so flaky that it will crash if you apply even security hotfixes, let alone major OS upgrades. This then makes everyone nervous about making any changes at all, which just exacerbates the problem, because by moving more slowly, the organisation falls even further behind the latest generation.

If you're a developer, than the analogy would be a software development organisation that doesn't keep up with the bug fixes until the code becomes so unstable that making any change to it is a gamble. This then means that even applying bug fixes is likely to just break more things, and then the whole thing snowballs until it's best to cleanse the source server with purifying fire. Or better yet, it's like a development company that never bothers to switch to newer versions of the Java SDK, eventually falling so far behind that it's too hard to make the huge leap required to catch up, instead of the many small steps that could have been taken.

The source of the mistake is short-sighted managers always choosing the easy thing instead of the correct thing. Taking small upgrade steps to keep up is critical, because large steps are very hard. A system that doesn't change for years is a temptation for lazy developers and administrators to bake in the peculiarities of the environment, instead of being forced to develop generic and supportable solutions that won't break after an upgrade of a related component.

heres another issue (1)

nimbius (983462) | more than 3 years ago | (#32330318)

companies that over-promise and assume linear growth, observe startup companies with great features and expect retooling their product to compete is a 20 minute job, and have a culture of worthless slags content to wallow in old code as a means to avoid having to learn anything new. reacting in IT to parabolic growth is difficult and patches are cheaper than extended downtime or the methodology to prevent it being instituted in a live system.

The meaning of Quality (4, Insightful)

bartwol (117819) | more than 3 years ago | (#32330376)

More than any other type, businesses are run by salesmen. These are people whose strongest attributes are the ability to build relationships, to communicate value, and a strong inclination to increase their personal wealth.

Increasingly, the stuff salesmen sell is based on complex technologies that, really, are beyond the reach of their comprehension. They kind of understand the products they sell, but really, they don't. If the world only had salesmen, there wouldn't be any sophisticated products.

Say hello to the engineer...a person who builds products. His strongest attributes are a desire to solve problems, a willingness to absorb the tedious but essential details needed to build a complex system, and a personality that derives gratification from doing so.

We now begin the business cycle. The salesman says, "Build me something I can sell."

The engineers says, "I will build you something that works well."

And therein begins a lifetime of the two, symbiotically, talking past each other. The engineer serves the salesman, and the salesman serves himself. But make no mistake about it: the salesman is in control.

For a salesman, QUALITY means it works well enough for him to sell more, and most importantly, to make more money for himself. For an engineer, QUALITY means it works reliably and efficiently. To be sure, QUALITY is an abstract and moving target that varies according to the eyes of the beholder. But to understand why we have the predicament described in this article, we need only understand the SIGNIFICANCE OF QUALITY TO A SALESMAN.

I would continue to expound, but then, most readers here need only reflect on their already frustrated pasts to understand the mechanics of this convenient but often vacuous relationship.

No! (3, Insightful)

Greyfox (87712) | more than 3 years ago | (#32330758)

The current patchwork of duct tape and glue that works today is much better than the pie-in-the sky "lets build it from scratch" architecture that IT is pitching that will be late, over budget and eventually have its feature set scaled down until it's less functional than what you have today when it finally is delivered.

There is constant pressure to re-implement existing architecture. Most of the time, the people who want to do this do not have a clear understanding of the business process involved, don't realize that the existing frameworks represent years of bug fixes and are at least stable for that reason. They only think "Wow this sucks, a new one would HAVE to be better."

I'm not saying that you should never rebuild something from the ground up, but the scope of the project should be limited and the entire endeavor should be well documented and well understood from the beginning. And if the guy who's pushing for a rewrite can't demonstrate a deep and fundamental understanding of the business flows being automated, he should be taken out and shot (Or at least pummeled soundly.)

Corporate Discipline (1)

WheelDweller (108946) | more than 3 years ago | (#32330858)

I was once a system administrator; a pretty good one, too. Every tool I wrote would let me know if it exceeded it's expectations, I re-used a lot of code, and did a lot of work after-hours. Sysadmins with their feet up aren't as "done" as they think. Documentation, from labelling wires to straightening out archives, takes a lot of time to do right.

From my vantage point, the job of corporate discipline was mine: if a piece of equipment was about to get overloaded, it was ME that found replacements, considered the compatibility, and searched by price to find the replacement. The changeover was scheduled on a weekend with a set of test-data that would let me rollback in case of failure.

We were 'just' running cash registers and PCs that both ran specific software and did research on the web...no one was in a capsule nor was their life on the line...but if you're gonna do this job, you have to _own_ it. And every second a cash register is down is a second the boss might be losing money. So yeah, it's mission-critical.

From my perch as number-2 in the organization, I had *only* the rights to make suggestions and to stop the boss from driving off into a canyon. It was really the only power I had. It felt kinda like the coroner of a small town: the only guy that could arrest anyone but the sheriff...and that person WAS the sheriff.

If you're being asked to do things with infrastructure that won't allow it, someone in your organization has succumbed to the need to finish something without _actually_finishing_ something. The job's not over until everything's back to the way it was...or better..than when the disturbance began. It's not just about setup for a project, it's for life after it, too.

That's not management's fault, unless it's for hiring people who don't look past their own keyboards. We don't just work when it's glorious...and it's never glorious. We work all the time in anticipation of getting swamped by the next increase of workload....and those of use good at it, we love it for that challenge.

Good luck (2, Interesting)

PPH (736903) | more than 3 years ago | (#32330906)

Been there, done that.

If you've got even a small or medium sized enterprise application (whatever that buzz word means) at a larger company (Boeing, for example), it might have its hooks into a dozen or more peer systems/hosts/databases/whatever. They are all 'owned' by different depatments, installed and upgraded over many years. Each on their own schedule and budget. When one group gets the funds together to address their legacy ball of duct tape and rubber bands, they roll the shiney new hardware in and install the spiffy new app. But everyone else is a few years away from affording new systems. And so the inter-system duct tape is simply re-wrapped.

The IT department tried selling everyone on architecture standardization. But due to the gradual pace of system upgrading, the plan was out of date before everyone got caught up to the old one. And today's 'standard' architecture wouldn't play nicely with what was state of the art a few years ago (thanks Microsoft). The whole architecture standard ploy is a salesman's pitch to get management locked into their system. Unless you've got a small enough shop that you can change out everyone's desktop and the entire contents of the server room over a holiday weekend (another salesman's wet dream), it ain't gonna work.

The solution is to bite the bullet and admit that your systems are always going to be duct-taped together. And then make a plan for maintaining the bits of duct tape. There's nothing wrong with some inter-system glue as long as you treat it with the same sort of respect and attention to detail that one would use for the individual applications.

This scales to your stack and apps as well (1)

Boosch (1570421) | more than 3 years ago | (#32330918)

DragonWriter sort of hit on this - and I'll leave the whole 'start over' concept to Joel and that older thread that seems to surface every few years. I don't develop software, I run an IT shop. And this issue bites me all the time - band-aiding apps and communications together until you don't know what is driving what.

You see, I'm constantly battling application creep - sometimes it's unavoidable because there really are not many options for some specialized needs. Sometimes I don't get to choose - it was purchased and the first I hear of it is to install. Although I'm morally in the clear to say 'go screw' - it's wasting money in the truest sense and I can limit that by 'getting it to work'. Now we hit on Weigel's post on technical dept. In an application - this might be easier to measure. With infrastructure, it's harder to document these future issues or temporary workarounds in some meaningful manner (open to suggestions). You of course are reminded when they bite you, but faced with time pressures of broken processes, wailing users, and the scramble to fix - the fix doesn't always get the proper attention it really needs. Again, we'll do the 'real' fix next week. (lather, rinse, repeat).

On the hardware front, sure I often get to choose the vendor and product - but it's still gotta perform some needed function - and at times I get to choose from bad, or really bad, or just won't work. I also have both inherited and purchased various servers that were available cheap or free (you keep on using that word - I don't think it means what you think it does), and then deal with odd ways of dealing with things like authentication, unique settings and variables (example = unique ways to identify 'today' or count time), and the growing skill set to maintain them. Again, not by choice or design, but by the reality of larger institutions.

I'd love to see or hear about ways to minimize this - or at least fend it off fairly well. I'm certain I'm not alone - but keep in mind the politics and realities of being in a larger institution.

When the corpus is more bandage than not... (0)

Anonymous Coward | more than 3 years ago | (#32331006)

You have a mummy. And they have amazing powers. This is also why all software hates Abbot & Costello.

Scrap the old model. (1)

w0mprat (1317953) | more than 3 years ago | (#32331072)

Design your user interface as a standards complient web interface. Why is there any need to code a native client to a system at all? A web interface can easily and rapidly be tweaked as browsers evolve, tweaking or rewriting binary applications is much more involved.

house of bandaids n/t (0)

Anonymous Coward | more than 3 years ago | (#32331076)

n/t

ignoring the SDLC (1)

WiPEOUT (20036) | more than 3 years ago | (#32331268)

While the technology environment can degrade over time due to poor processes, just as important is getting it right in the first place. In my experience, the IT team often lacks any architectural capability when the department is set up, typically because it's "too small" or their needs are "simple". As a result, nobody with the necessary qualifications/experience/mindset really thinks about the business' strategic requirements, they just go for "best practice". Unsurprisingly, "best practice" by itself simply means whatever fad the technology industry is pushing at the time.

Many of the posters above are right in saying that simply throwing it all out isn't the solution, however, throwing away all the assumptions, determining the strategic requirements and then reconciling those with the existing landscape is the way to do this. Sometimes that then results in things being replaced, but sometimes it does not.

Every other business unit does (or should be doing) this during the corporate strategic planning cycle, why does IT not? Sadly, the reason is normally the organisation only really has one "primary" business unit and it excludes IT from the strategic planning altogether. It's IT's job to convince the decision makers that they ought to be partners in strategy, but IT's normal lack of rigour in processes and particularly in financial management that prevents them from having the ammunition to fight for a place at the table.

At the time it was good (1)

AHuxley (892839) | more than 3 years ago | (#32331316)

Seems to be the syndrome. Everybody used Intel, MS and other devices to save cash and swap out over the next few years.
Why buy a huge expensive box that would rot in place when you could buy a cheap unit and swap it out later.
Nobody understood really where tech was going.
They went cheap so when they had better cash flow and market direction they could slot the future in at a fair price.
The good news is someone made the right calls or got consultants who did. They might have got big boxes and it scales or a smaller setup up that was designed right and it can scale too.
If your mission critical the feds or states will bail you out, for the rest, learn, share, name brands that worked or failed and move on.

Microsoft doesn't start over (1)

drumcat (1659893) | more than 3 years ago | (#32331348)

For what it's worth, Microsoft outsourced their IT to India. You're asking a question of fountains, and your bosses see extraneous plumbing...

A contrarian opinion (1, Insightful)

Anonymous Coward | more than 3 years ago | (#32331454)

This reads like those "articles on investing" written by mutual fund companies. After all, isn't the publisher in the business of selling software and services? If your stuff works, it will continue to work for the same requirements if you don't mess with it. Sometimes there's more risk in starting over: "Hey (insert-born-in-1990's-name here) let's rewrite the airliner's flight control software as a facebook app, it's out-of-date."

A Guy Named DepEnds is not part of the solution (0)

Anonymous Coward | more than 3 years ago | (#32331580)

Where does one start?

1. Management is old. They grew up with horses, not computers.The times were booming, they were arrogant, and never really learnt what computers were. These old fucks think saying "I don't know" brings about immediate death and dishonor by Klingons. Never back down. Never let them see you sweat. Never say I'm sorry. WEAKNESS!! Bluff and Bluster and Bullshit. Testosterone, Balls not Brains. Well, it has caught up to them. And their portfolios are down 50%

2. Computers should have changed every profession. But the established professions won, and IT pros were never given the status of doctors and lawyers. IT are glorified houseboys that fix the gizmos.

3. Management cannot tell a good programmer from a bad programmer from a superstar.

4. We needed Leonardo Da Vinci, or Newton to be the super programmer, the man who mastered many fields to head up the premier vanguard computer company, to blaze new trails, to go where no one has gone before, to rewrite the book. Instead we got the man child Bill Gates (spraying the ocean into the clouds), The Control/Acid Freak Jobs, and the 'I am CEO, bitch' Zuckerberg. Not good role models.

5. Sales, of course, saw the stupidity of upper management, and has sold them a never ending supply of silver bullets. Go Sales! Good Work. At least part of the world still works.

6. Here we sit, a bunch of old obsolete programmers, generations of wasted Cobol, Fortran, Visual Basic programmers, GENERATIONS wasted so that BIGCOMPUTER could sell another round of snake oil languages. No upgrades, no transitions. Willful, purposeful obsolescence of GENERATIONS of the best talent computing will ever seen, purposely obsoleting the founders.

7. Here we sit, Obama pumping out new regs and laws, that need to be complied with, 1099 form for every purchase over $600. LOL!

8. Here we sit, with piles of obsolete XP gear, OFFICES FULL of obsolete computers so that BIGCOMPUTER can sell another round of upgrades.

9. Here we sit, the WITH NO CAPITAL to upgrade with, and no where to borrow it from.

10. Here we sit, no fresh round of cheaper and cheaper and cheaper programmers anywhere on the planet

11. Here we sit, with the economy in the crapper, the very currency we get paid with threatening to disappear faster than privacy on Facebook

12. Management wants disposable, interchangable cogs. Management HATES craftsmen. How long have we been called RESOURCES?

13. The disposable people religion and outsourcing has all organizations gutted of institutional knowledge and continuity.

14. The emerging world and upstarts like google are inside the OODA loop of the establishment. Fear. There is nothing deadly than an old man, losing his grip.

15. No succession planning. No new blood. No grooming. The oldest of the boomers and the silent generations still run things. Their cold dead hands will need to be broken from the wheel. Where is Bill Gates stable of new hot shot new superstars? No where. Because these old fucks can not and will not share the stage.

16. Corruption. The last 15 years have been built on corruption, gambling and Ponzi schemes. If you are under 30, all you have every known is the bubble world. Why produce value, when you can leverage and gamble? Rape and Pillage?

17. IP laws. Lack of R and D. Nobody wants to spend money on innovation. And who let the lawyers in? You know, lawyers. The guys that still keep ALL THEIR FILES ON PAPER? Which profession speaks gobbley gook and hides behind confusing langauge again?

18. Time has sped up. Shit is changing outside the walled garden and the barbarians don't care about your delusions or your status quo.

19. External Threats. The developing nations buy more oil and cars than the G17 countries. Yep.

20. Documentation. There is none. And the primaries and principals and the founders are gone.

What are we going to get? Some dufus with a cool handle, a diamond encrusted ipod and blog with groovy graphics, born with a silver spoon stuck up his ass, will give another round of buzzwords, another language, another methodology, another layer of shit and crap and virtualization. You'll all buy in. Yeah! We are saved. There will be dividends this quarter.

Your management has repeatedly fallen for this crap and will fall for it again. You can't con an honest man, but you can fool all of the managers all of the time, as long as the salemen buys enough drinks.

The whole computer industry has been turned and churned and burned and BIGCOMPUTER made a lot of money.

The nerds have all been put into their place, and management is spending your IT dollars on junkets and gold plated ipods and shinier and greener computers.

Management NEEDS to fashionable.

So what if your organization is now DONE because of it.

Many organizations are terminal.

And why not burn them down. Or let them crumble. It IS fun to watch.

I'm melting! I'm melting!

The kids will build something new and much better.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...