×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

The Cult of DevOps

Unknown Lamer posted more than 2 years ago | from the but-risky-deployments-are-exciting dept.

Google 114

packetrat writes "I was at OmniTI's Surge conference today, which turned out to be, among other things, a meeting of the cult of DevOps. Ars Technica covered the keynote and some of the presentations, but some of the best stuff is in the comments. Google CIO Ben Fried told the tale of a really poorly engineered trading application at Morgan Stanley that he was associated with, and how the way IT was structured there contributed to that engineering and to its spectacular failure, costing the bank untold millions in stock trade processing fees from its institutional customers. He said what he learned from cleaning up the mess has informed how Google runs its IT operations, and a culture that promotes generalist skills. A lot of how he describes Google's approach sounds like the DevOps kool-aid a lot of the other speakers were serving, but it also sounds like common sense — are most IT organizations really that poorly run that developers are totally unaware their software is sending messages that are generating network storms, or network engineers are clueless enough about QoS to route leased lines into their data center through their public-facing Internet?"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

114 comments

Yes (0)

Anonymous Coward | more than 2 years ago | (#37576156)

Was that a rhetorical question or do you really have to ask?

Cult of DevOps? (4, Insightful)

merlinokos (892352) | more than 2 years ago | (#37576182)

I'm not sure I can take anybody who calls an attempt to make IT and Development more aware of each other a cult, seriously.
The traditional way of doing things didn't work for 30 years. Why is it that when people are trying to make (and apparently making) a difference to how companies work, they're regularly denigrated by a large subset of the very people whose working lives they're trying to improve.
Haters gonna hate, I guess.

Re:Cult of DevOps? (1)

Lennie (16154) | more than 2 years ago | (#37576258)

I read the article, I think the title is basically talking about:

'he thinks Google gets this right. “We go to great lengths to hire people with engineering skills, put engineers in operational roles and give them power and accountability."'

The big issue is lots of people are being replaced (1)

OeLeWaPpErKe (412765) | more than 2 years ago | (#37581346)

By scripts.

Add to that that devOps is much, much harder than system engineering. A DevOps engineer needs to know enough about multiple programming languages *and* algorithmics *and* data and telecom infrastructure to tweak the programs that are really controlling the network. There aren't many people like this at the moment (so if you want to move as a programmer to a $200k/year job, understanding the packet datapath and it's controlling plane is where it's at, provided you're good at optimizing algorithms. The first company to make a real control plane that merges control of the application inside the host and control of the network above the host, controllable by an api is going to make a bundle).

Most sysadmins and netadmins these days know a bit of scripting, and have lots of knowledge over specific details of the OS/switches/routers/firewalls, like the access subsystems or networks and firewalls. But almost none of them could tell you exactly what happens, in code, when ssh refuses or accepts a connection, and what access that implies for all parties involved. I've yet to meet the first sysadmin that can tell me how to optimize a non-trivial problem. An example devOps issue that is requested is to increase that access (e.g. we control the firewall, and we would like it to log ssh connections. Can you tweak the ssh daemon with as little interference on the client side as possible. And then use that to log all access to production systems, in other words : execute a man-in-the-middle attack on the firewall to get full accountability of the production network).

But this is child's play compared to what the devOps guys really want. They want an API they can use for problems like "we have a network with nodes N, links L(n1, n2, capacity, cost function), peering links P, and data transmission requests from the applications R(app1, app2, dataSize, profit, constraints). Configure the applications and network so that cost is minimized, profit is maximized and all specified constraints are followed (e.g. "maximum latency for video : 2s, max latency for voip 50msec, ...). Oh and do tell us which improvements to the network would bring the most benefit in terms of $ per increase in maximum bandwidth transmitted."

If you have a company whose profit is mostly determined by the capacity of it's network, any advantage you have in this is going to pay out hugely. For a netflix, amazon, bing, microsoft's online office and other apps, google, yahoo, any online advertising company, any online voip company, ... having a functioning algorithm like this means a vast advantage over competitors.

The big issue is that it's much easier for a capable programmer to become a devOps engineer than it is for operational people to become devOps engineers. Not that we have any kind of surplus of capable programmers at all, but I doubt that even if they have no choice many sysadmins will become good programmers.

So the issue is, this is about automating network/system operations the way ford automated car production : if you work in operations, one of 3 things is going to happen :
1) you move to development/engineering, and spend most of your time writing and debugging programs
2) you become a robot, a call center employee, the interface between a computer system and external parties, in other words, your job is reduced to "you want fries with that ?" level
3) you're fired (which it will be for a lot of people)

For the vast majority of people in operations it will mean a gradual move to option 2, and after a few years option 3 if you didn't manage to get recruited for engineering ... Of course this does mean that if you do succeed it will mean an increase in salary, potentially a big one. But most people won't succeed (which is of course the reason for the salary increase in the first place).

And that is of course the big issue. The IT workforce is going to drop massively if these guys succeed, and consist entirely of call-center employees and developers. And this will make things harder for developers, so even they will need to buckle up at least a bit. Like the car workers, the current crop of operations people is not going to leave without a fight. This is why we're moving to entirely thin-clients, even if they're implemented as a web browser on a thick client.

And sadly, this is the future. The plan is :
1) move everybody to thin clients, ie. the web browser. So every support request is either
        a) I can't connect to the web
        b) I have an issue with your website
and the devOps people only have to keep 1 webserver operational per 1000 customers (just a number out of my ass)
2) massively increase profit by massively increasing the datastream delivered to end-users

Re:Cult of DevOps? (1)

visualight (468005) | more than 2 years ago | (#37576570)

I don't think DevOps is new or non-traditional. The *label* might have been invented recently but it's not like Dev and Ops never worked together before. Everyone should know something about everyone else's job, or maybe (from your post)IT and Development should maintain an awareness of each others' status. I've seen a bunch, they all pretty much mean the same thing, and they're all one sentence.
The people who can take one sentence and turn it into this [wikipedia.org] might be a cult.

Re:Cult of DevOps? (2)

cduffy (652) | more than 2 years ago | (#37578396)

I don't think DevOps is new or non-traditional. The *label* might have been invented recently but it's not like Dev and Ops never worked together before. Everyone should know something about everyone else's job, or maybe (from your post)IT and Development should maintain an awareness of each others' status. I've seen a bunch, they all pretty much mean the same thing, and they're all one sentence.

Huh?

DevOps isn't about mere communication or awareness -- it's about operations being automated via tools maintained by development, using traditional development methodologies.

That's a far tighter integration than what's traditional, even in a shop without a wall between the groups.

Re:Cult of DevOps? (1)

visualight (468005) | more than 2 years ago | (#37579592)

it's about operations being automated via tools maintained by development, using traditional development methodologies.
That's pretty much how it is where I work, it's been that way for as long as anyone remembers (at least 15 years). But no one ever called it DevOps.
In any case, however you define it, it's a concept and as such it can be expressed in a sentence. Most of us are more than capable of extrapolating the implications. Well, at least those of us that understand all seven sigmas. ;)
My view is that giving this concept a label and then going on to "over define" it serves only to introduce yet another dogma.

Re:Cult of DevOps? (1)

cduffy (652) | more than 2 years ago | (#37579710)

it's about operations being automated via tools maintained by development, using traditional development methodologies.
That's pretty much how it is where I work, it's been that way for as long as anyone remembers (at least 15 years). But no one ever called it DevOps.

15 years is quite some time -- by any chance were y'all an early adopter of CFEngine? That's the only toolkit I can think of under an OSS license for doing OS-level configuration programatically that was available at that time, unless y'all were using in-house tools.

[Personally -- It was about 10 years ago I evaluated CFEngine, decided I didn't like it, and wrote my own tools for OS-level configuration (sadly, proprietary); 4 years ago I was hired into a Puppet shop; 2 years ago I jumped ship for Chef, and have been a relatively happy camper since... though I'm anxiously awaiting better orchestration capabilities; Zookeeper and Noah are steps in the right direction, but need better integration with the rest of the stack].

My view is that giving this concept a label and then going on to "over define" it serves only to introduce yet another dogma.

There's utility to the label. First off, it gives a name to the people inside development who write tools for automating OS-level configuration. Second, it means that when I see a job opening described as a "devops position", I have a pretty good idea of what kind of skillset is going to be required, and I can be reasonably confident that someone with a "devops" entry on their resume knows how to think in terms of idempotent operations, notification triggers, and all the other primitives that go into modern config automation tools.

Folks can talk about "dogma" and "cults" all they like, but as long as a word or phrase has meaning and utility to those that use it, it's not so likely to be given up.

Re:Cult of DevOps? (1)

visualight (468005) | more than 2 years ago | (#37580196)

by any chance were y'all an early adopter of CFEngine?

All in-house tools back then (though I wasn't there). Well, still actually. There are hallway conversations about chef and puppet. Some very well thought out scripts and rsync still work so there isn't a very strong motivation to experiment. A few people want to be able to put Puppet on their resume so it ( or something else ) will probably show up somewhere sooner or later.

I have to agree that labels are useful, but I also think this devops thing is a bit silly. Specifically I mean the enthusiasm around it, as if it's some kind of revelation. Well, I work in an environment where everyone is a developer and everyone has root. Maybe that's not as common as I thought.

Re:Cult of DevOps? (1)

cduffy (652) | more than 2 years ago | (#37581248)

A few people want to be able to put Puppet on their resume so it (or something else) will probably show up somewhere sooner or later.

Personally, I can't recommend Puppet -- it makes simple things simple, but when you get to the point where you want to do hard things, you end up needing to make use of integration points that aren't so well-thought-through... whereas Chef has extra primitives ("data bags", search, &c) that are well-integrated and more power in the language (real Ruby, as opposed to a limited DSL) making it easier to do your own thing without someone having to have built in syntax for the purpose. Not saying there's anything you can't make Puppet do if you're willing to work at it, but it's going to be more work.

Specifically I mean the enthusiasm around it, as if it's some kind of revelation. Well, I work in an environment where everyone is a developer and everyone has root. Maybe that's not as common as I thought.

It's not so common in my experience -- especially in shops where growth happened before they started putting real work into infrastructure automation, so they started hiring out traditional sysadmins without development skillsets.

Even in places that started early, though, trusting your entire development staff with root is something that I've never seen happen -- and for good reason. One of those employers dealt with medical records, another with customers' email archives, another had credit card numbers pass through our hands... in every one of those cases we were audited by people who wanted to know, and with very good reason, exactly who had access to our production infrastructure, and why they had it, and what measures we took to make sure that nobody retained such access after they left. It's pretty hard to be sure that someone who's had root hasn't slipped in a backdoor somewhere, so keeping that list to a minimum is critical just for operational security reasons. (Beyond opsec, there's also the matter of avoiding the case where you have developers playing cowboy in production... but that's a different discussion).

There's also a Fortune 50 one of my former employers was bought by that was so particular about separation of responsibilities that talking to someone on the other side of the wall was verboten without going through channels. One person having access to both source control and production was utterly unthinkable. Now, that is just silliness, but giving root to everyone in your development staff -- particularly if you're doing enough dev work to have that be a significant number of individuals -- is something that doesn't scale too well.

Re:Cult of DevOps? (1)

Jawnn (445279) | more than 2 years ago | (#37579306)

I agree, to a point. Sometimes that point is forced by the size of an enterprise, as in - eventually, you just can't afford to waste developers' time on teaching them admin skills. By that time, operations leaders should have forged two things: 1) A set of policies and procedures that will define "the way things work", and 2) A good working relationship with development leadership so that each understands and respects the other's challenges. Development will know that they can't roll out something new without adequate testing - meaning a methodology that at least approaches real world workloads on a platform that is more or less equivalent to that which will be used in production. That's easy to do when the combined numbers of both departments can be enumerated on the digits of one hand. Beyond that, not so much.

Re:Cult of DevOps? (1)

turbidostato (878842) | more than 2 years ago | (#37579980)

"eventually, you just can't afford to waste developers' time on teaching them admin skills"

OK, maybe you can't afford it. But then you will be forced to afford the alternative -and it's not going to be any cheaper.

Re:Cult of DevOps? (4, Insightful)

wisty (1335733) | more than 2 years ago | (#37576704)

I'm guessing the main haters are sysadmins, who see threats to their importance and way of working.

It's no longer desirable to have a gradually grown system. Everything should be built on virtual machines from scripts. You no longer need a wise old guy with a beard to change anything - who cares if your modification might trash the system? You can spin up a test instance, and dump it if it doesn't work right. If you have two apps that don't play nice with each-other, just put them on difference machines. The philosophy of a DevOps guy is very different to that of a sysadmin - you don't do brain surgery on a horse while it's in the middle of a race. Just shoot the damn thing, and send in a healthy clone. (Or would you prefer a car analogy?)

There was a world of difference between "works on the dev box", and "works in the production environment, without screwing up anything else". That gap is getting bridged by DevOps practices. OK, you still need people with sysadmin skills, but the way they work (and the things they have to learn) is becoming a lot more like development than sysadmin work, and sysadmins are not bred to like change.

Re:Cult of DevOps? (1)

Anonymous Coward | more than 2 years ago | (#37576760)

This guy gets it. How many manhours - manmonths even? - have I wasted at previous jobs trying to revive a system or piecemeal fix a misconfigured or broken system? How many other companies have done the same? It's just not worth it - we have the technology to spin up a new VM instance to replace it... and while that seems obvious, the implications on how your build process ought to be created and the benefits you can get from it are pretty significant.

Re:Cult of DevOps? (4, Interesting)

gmack (197796) | more than 2 years ago | (#37577298)

And how do you know the new VM won't have the same problem? If you never know what went wrong and fix the actual problem you will just end up restarting VMs constantly.

Re:Cult of DevOps? (1)

tirerim (1108567) | more than 2 years ago | (#37577340)

You don't, but that's okay -- spinning up a new VM is just one thing to try. If the problem doesn't reoccur, then you've saved yourself a lot of time and hassle trying to figure it out. If it does, then it can still help you eliminate some possible reasons. If you're an idiot, yes, you'll just keep creating new VMs and not solving the problem. But you're not an idiot, right?

Re:Cult of DevOps? (2)

gmack (197796) | more than 2 years ago | (#37578260)

Sorry but no. What you are suggesting is a more modern "reboot the server and see if that fixes it" and it solves nothing. Somewhere the bug is lurking and waiting to rear it's ugly head again. It could happen in a week or it could happen tomorrow but it's very likely it will happen.

Re:Cult of DevOps? (0)

Anonymous Coward | more than 2 years ago | (#37578364)

Sure it solves something: the service now works, from the customer point of view. Which, in the end, is all that matters. For the customers, halting a system because you want to diagnose an obscure bug they don't care about doesn't make any sense.

Re:Cult of DevOps? (1)

gmack (197796) | more than 2 years ago | (#37578750)

You have the time between when a service starts misbehaving until it's detected and then reported to someone who can start a new VM, That time gap is something the customer will notice Fixing it right the first time prevents more outages and the customer will care about that. A VM/Backup server/ whatever may help with shortening the outage but it will not reduce the time spent debugging.

Re:Cult of DevOps? (1)

Lisias (447563) | more than 2 years ago | (#37578770)

Sorry, but no. What the parent is saying is something like

"reboot it once and let's see what happens. If the shit hits the fun again, clone the damn thing, reboot it again to keep the service alive and try to figure out what's happening on the clone."

There is no point, for the patient, to have his disease figured out after his death.

Re:Cult of DevOps? (1)

gmack (197796) | more than 2 years ago | (#37578876)

Better idea: bring in backup server/VM or whatever then figure out what went wrong on the original. If it happened once than it will happen again given the same config. You might have some bad hardware, or you could have a bug that only happens under select conditions.

Case in point: the sasl software most people use with Postfix has a memory leak when the wrong password is entered (yes they know, no they don't actually care) Some would say a memory leak isn't a problem because it would take thousands of bad password attempts to take down the software right? But what happens when some bot sends several thousand requests in an attempt to guess system passwords? 4 GB of ram later the process goes dead. Reloading the VM? good thing but you still have an exploitable glitch that can be used to take down the mail service and unless someone goes through the logs and discovers what exactly happened you will just end up with the same problem on the next bot scan.

Re:Cult of DevOps? (1)

slifox (605302) | more than 2 years ago | (#37579504)

You hit it right on.

I think many admins are ok with the "reboot fix" technique, because they don't work in high-stakes environments. If Bob in Accounting's windows desktop goes down twice this week due to the same issue, who cares (just reboot it again). The same goes with most non-mission-critical services (and those without strict SLA). The same can sometimes also apply in vastly-redundant heterogeneous environments, where having one application instance go down is not a big deal since you have 1000s more identical instances (e.g. content delivery networks, etc; unless the bug ends up taking down everything!).

That approach can't be applied to high-stakes environments, where peoples' lives or money depends on bug-free operation. Try explaining to management why markets have been halted several times this week due to a problem that could've been fixed the first time with a little bit of investigation.

Sweeping important problems under the rug is a great way to drive customers to your competitors.

Re:Cult of DevOps? (1)

cduffy (652) | more than 2 years ago | (#37578354)

And how do you know the new VM won't have the same problem? If you never know what went wrong and fix the actual problem you will just end up restarting VMs constantly.

Sure -- but that's a thing to be replicated and fixed as a development task, not a critical-production-maintenance one. Not like you can't keep a copy of the bad VM around for inspection.

More to the point, if you're Doing It Right, the "spin up a new VM" process involves applying configuration-as-code, not replicating off some handcrafted mystical golden image that's been tweaked who-knows-what way, so every change that was ever made to the system since its OS installation? You have that change (in a version-controlled source tree), you know its state, you've replicated exactly that same configuration in your staging environment, and so bizarre one-off bugs don't happen so much to start with.

Re:Cult of DevOps? (1)

gmack (197796) | more than 2 years ago | (#37578918)

On a properly deployed system you should expect config problems for the first couple of days after you deploy something and someone does something you didn't expect even if the same config worked in testing and after that the bugs get into the "one off" variety. In either case you want to find out what went wrong.

It's not the "load a VM to get the service working again" that I object to. I object to the idea that you can just reload a VM and never debug.

Re:Cult of DevOps? (1)

Vanders (110092) | more than 2 years ago | (#37578598)

If your server is misconfigured then you've already failed at DevOps. Your configuration management platform (I.e. Puppet, Chef etc.) should be keeping the server in a known state. If it's not then you need to fix your manifests/Cookbooks. Redeploying a new VM with the same incomplete configuration wont help.

If you don't have configuration management then you're barely even doing Ops, let alone DevOps.

Re:Cult of DevOps? (1)

mounthood (993037) | more than 2 years ago | (#37577834)

I'm guessing the main haters are sysadmins, who see threats to their importance and way of working. ... Everything should be built on virtual machines from scripts.

The sysadmins like virtual machines; they make life easier. What they don't like is developers telling them how to do their job. Internal software doesn't come with instructions, it comes with a coworker making demands about server resources, DNS/firewall/network configuration, licensing and backup requirements, etc... Not all those developer requests will be reasonable or correct; devs make mistakes too.

Re:Cult of DevOps? (1)

cduffy (652) | more than 2 years ago | (#37578436)

The sysadmins like virtual machines; they make life easier. What they don't like is developers telling them how to do their job. Internal software doesn't come with instructions, it comes with a coworker making demands about server resources, DNS/firewall/network configuration, licensing and backup requirements, etc... Not all those developer requests will be reasonable or correct; devs make mistakes too.

Thing is, that's not how it works in a mature devops shop.

Development writes the code that set up the DNS, the firewalls, even the kickstart files (or whatever your local equivalent is) that control OS installation. If development's code doesn't work, developers' dev VMs and QA's environments break first, because they're configured just the same way.

Licensing servers? Development writes the code that manages them. Backup? Development writes the code that automates it.

Or... well, in practice, a small group of people with deep sysadmin experience managed under the development group writes that code. Ops kicks tickets up to development when things break, but for the most part their layer of importance gets thinner -- hands-and-eyes on the physical servers, assigning roles to hardware they spin up using the dev-provided tools, diagnosing easy bugs and writing tickets (or pulling a developer in to diagnose the hard ones), etc.

Re:Cult of DevOps? (0)

Anonymous Coward | more than 2 years ago | (#37580036)

Letting developers do anything that involves DNS, firewall, routing (network in general) or backups is horrible horrible idea. Why ? Because they do not have expirence or knowledge about that, and unless they are also syadmins, they will fail horribly.

Re:Cult of DevOps? (1)

Vanders (110092) | more than 2 years ago | (#37580136)

unless [developers] are also syadmins, they will fail horribly.

Hmmm, perhaps someone with some form of cross-disciplinary Developer/Operations knowledge and experience could help?

Re:Cult of DevOps? (1)

segedunum (883035) | more than 2 years ago | (#37580274)

Development writes the code that set up the DNS, the firewalls, even the kickstart files (or whatever your local equivalent is) that control OS installation. If development's code doesn't work, developers' dev VMs and QA's environments break first, because they're configured just the same way.

Development does that? So developers know all about DNS, firewalls and OS installation? Errrr, no. They think they do and they think it's easy, but they simply don't. Those that do get burned. Badly.

Backup? Development writes the code that automates it.

Backup as well? How fabulous. In reality it's sys admins who should be encroaching on developer turf, not the other way around, and the reason why this Devops cargo cult crap has come into existence is because developers now need to know about it, they don't want to and as such they want to pretend it's something they can take over. Why? Because it doesn't matter how many lines of code you write, if it doesn't work in production no one cares and you won't get paid.

Re:Cult of DevOps? (1)

cduffy (652) | more than 2 years ago | (#37581290)

Development does that? So developers know all about DNS, firewalls and OS installation? Errrr, no. They think they do and they think it's easy, but they simply don't. Those that do get burned. Badly.

And that's why you have a crossdiciplinary team -- DevOps -- taking care of that stuff rather than foisting it on the application developers.

Good DevOps folks tend to also be system-level developers -- ideally people who know everything from the kernel level up -- as well as having a good chunk of system administration under the belt. It's a rare breed, sure, but that's why we make the big bucks. :)

Backup as well? How fabulous. In reality it's sys admins who should be encroaching on developer turf, not the other way around, and the reason why this Devops cargo cult crap has come into existence is because developers now need to know about it, they don't want to and as such they want to pretend it's something they can take over. Why? Because it doesn't matter how many lines of code you write, if it doesn't work in production no one cares and you won't get paid.

Developers "want to pretend"? Riiight.

Last place I was at it was a 2-man team (in the devops group -- app development was of course much, much larger) writing software to automate the configuration of 7 datacenters worldwide, the largest of them being well over 1000 hosts (for a storage- and search-centric application). And it worked, beautifully -- we scaled up our DCs to cope with customer demand without needing to scale up our operations-team headcount, which was exactly the thing we were investing in devops to do.

You look at the folks who are big names -- Google, Twitter, Disqus, Flickr -- and they're all investing in devops, because it gives repeatable, scalable results. You want to go stick your head in the sand? Be my guest. Me? I like being able to spin up a new cluster (or double the size of an existing one) by clicking a few buttons and letting the automation do the rest.

Re:Cult of DevOps? (1)

lennier (44736) | more than 2 years ago | (#37585884)

Development writes the code that set up the DNS, the firewalls, even the kickstart files (or whatever your local equivalent is) that control OS installation.

I'm unfamiliar with the term "DevOps", but in my kind of workplace, Operations do that kind of OS configuration stuff (including writing automated scripts to manage it), and then we have Application developers who don't know and don't care anything about the OS, but want their precious application to write to C:\Windows\WhyDontIhaveRightsHere and have an always-on two-way RDP connection on port 31337 to a data centre in Korea.

OS and Applications Development will fight eternally, in my experience. It's nothing to do with "development" vs "operations" or automation vs manual configuration - the OS guys want to lock everything down and use standard systems, and the applications guys want to open everything up and do everything custom and bespoke. How do you resolve that conflict?

Re:Cult of DevOps? (2)

Vanders (110092) | more than 2 years ago | (#37578198)

I'm guessing the main haters are sysadmins, who see threats to their importance and way of working.

I started life as a traditional sysadmin. Now I'm very much a DevOps guy. I love it. DevOps is a massive improvement on the old throw-it-over-the-wall, compartmentalised way of doing things.

Re:Cult of DevOps? (2)

gmack (197796) | more than 2 years ago | (#37578410)

I think you have missed the entire point. You still need people who know what they are doing to make the scripts and setup in the first place.

What DevOps fixes are places where any problem gets passed off on someone else. Software is too slow? Software devs blame the database people who blame the systems people who blame the networking people (if they can) and as a result pretty much any problem is fixed with hardware upgrades and when that becomes impossible you end up with a badly running system but either way the whole thing costs more than it ever should have.

Case in point: At a company I previously worked at they were building some some bingo software for another customer that required a front end server and a dedicated database server (with hot failover). That's three top of the line servers for 300 concurrent users. Software devs blamed the hardware and when that failed they blamed Java.

Re:Cult of DevOps? (2)

mverwijs (815917) | more than 2 years ago | (#37578840)

I'm guessing the main haters are sysadmins...

And that is exactly the chasm of the "us" and "them" attitude that the DevOps folks (IMHO) are trying to bridge. Thank you for your fine example, sir.

WTF are you talking about? Devops is nothing new (3, Interesting)

Colin Smith (2679) | more than 2 years ago | (#37579250)

Historically a sysadmin has been able to and does write, fix software, and administering large scale systems has always meant templating and then deploying from updated templates... which means having a build system which can build to a template usually automatically (it's boring) which apparently is now called continuous integration. You also need an infrastructure designed to easily deploy changes. VM or physical is almost totally irrelevant, VMs make life a little easier.

Lets see... Infrastructures.org for example was definitely there mid nineties (and still is, cool.). That's almost a whole generation ago, how old are you?

I have no idea what makes people think this devops stuff is new. Is it just the fact that there are now thousands of newbies trying to make names rediscovering good practices? Is it just that we now have more large scale systems, or is it that we now tend to have highly distributed vs centralised systems?

And it's not development. It's engineering. The difference is maths.

Go on, get off my lawn!

Re:WTF are you talking about? Devops is nothing ne (1)

segedunum (883035) | more than 2 years ago | (#37580344)

I think it's more of a case of developers having to now do a lot of sys admin work, not wanting to do it, not having the expertise to do it, but trying to make out that they can 'develop' things away and it will all be easy. Small companies have this problem, and small IT companies like to believe they can just hire in sys admins and farm crap out to places like Heroku and Engine Yard.

In my last job I pointed out to our developer when the development work dried up, as it does, that our clients paid for their applications to be kept running and live and it was the sys admin they were paying for. He was most upset.

Re:WTF are you talking about? Devops is nothing ne (1)

wisty (1335733) | more than 2 years ago | (#37580862)

I'm guessing in the 90s, it was pretty rare to actually use those practices. A new shop would have one server called "server". Then it would get a few, named after the Seven Dwarfs. Eventually, you would get to the point of needing to get a new sysadmin who could handle large-scale deployments (or have the old sysadmin change the way they worked), but that would be around the post-IPO stage (if ever), not the "three guys, one of whom is figuring out how to script EC2" stage.

Re:Cult of DevOps? (1)

turbidostato (878842) | more than 2 years ago | (#37580384)

"I'm guessing the main haters are sysadmins"

And then you are wrong.

There certainly is *some* uneasiness on *some* sysadmins, but not because what you seem to think but because of very valid concerns.

Devops, as other have stated, while a new word is not a new concept -other highlighted infrastructures.org for instance, and it is basically having the sysadmin mindset to use the developers' tools. And here comes the uneasiness because using the developers' tool puts developers in a tactical advantage... for the worse of the systems to be delivered.

"It's no longer desirable to have a gradually grown system."

It is, or else it won't work.

"Everything should be built on virtual machines from scripts."

Which is as stupid as it can get. Not, the "from scripts" part but the "on virtual machines one". While virtual machines are certainly an advantage, it is neither a critical one nor a qualitative difference from "the old way".

"You no longer need a wise old guy with a beard to change anything"

You'd better have that wise old guy with a beard to plan how is the whole thing to be layed out, or you will suffer the consecuences.

"who cares if your modification might trash the system?"

Certainly not your average developer, and that's exactly my point.

"You can spin up a test instance, and dump it if it doesn't work right."

And with that mentality you'll find yourself all of a sudden with an outage that spans half a continent as the recent Amazon one can testify.

"If you have two apps that don't play nice with each-other, just put them on difference machines."

And then you have to different apps that use two different versions of libfoo -without any real/technical reason for it but that their different development teams happen to use them, that will be left unpatched when the next security problem arises. After all, the development team's goal is not to secure their last month apps but to deliver the next shiny version.

"The philosophy of a DevOps guy is very different to that of a sysadmin"

I claim bullshit on that. The philosophy of a DevOps guy is *exactly* that of your berated "wise old guy with a beard" sysadmin: having production systems that are properly understood, manageabe, solid and that you can be confident at. And of top of that, that are flexible enough to be modified to support today's changing needs as fast as possible.

"OK, you still need people with sysadmin skills"

Of course you need them, because they are the only ones with both the skillset to understand the big picture and the mindset to be paranoid and anal retentive enough to make it work.

"but the way they work (and the things they have to learn) is becoming a lot more like development than sysadmin work"

Yes, that's right, and probably the only right thing in your whole post -that's why I said devops is mainly sysadmin mentality with developers' tools.

"and sysadmins are not bred to like change."

Sysadmins were who made the whole thing to work together. Sysdmins *do* like change, it's only they don't like "change for change's sake" nor "change that breaks things": they like "change that brings new things while not breaking the ones already in production".

Re:Cult of DevOps? (0)

Anonymous Coward | more than 2 years ago | (#37580564)

Boy are you wrong. It sounds very much like you have come to the conclusion that you don't actually need to know what you're doing anymore because you can just sandbox things in a VM guest. Please please please stay away from my organization. Running guests in virtual environments may make the uninitiated feel like good sysadmin skills are less relevant, but it's just not so. They are more important than ever, because now you not only have to consider interactions among the applications running on a single piece of hardware, but also interactions between multiple guests which are all sharing CPU's, memory, and storage. That job used to be entirely the responsibility of the operating system. Now it is the dual responsibility of the operating system as well as the underlying virtualization technology. Just because you can click a couple of buttons to make a new guest doesn't mean the system as a whole has gotten simpler - quite the opposite. Your mail server just went down? May have nothing at all to do with the mail server per se, but rather with Jim Bob changing out 100 virtual desktops to run a computer lab.

The DevOps concept under discussion has nothing to do with shooting horses or other horrible analogies to your intellectual misfires. The idea here is that in order to be effective, you have to understand how everything is connected. And you can't understand how everything is connected without having broad exposure to and understanding of the entire gamut of technologies involved in getting applications into production: networking, databases, operating systems, shell scripting, programming, virtualization, storage, interface design, web technologies, and so on.

who cares if your modification might trash the system?

Me. VMWare is not an excuse to celebrate your ignorance.

Re:Cult of DevOps? (1)

CAIMLAS (41445) | more than 2 years ago | (#37581734)

Sorry, that's just a bit of bullshit. In today's IT organizations, a sysadmin can't be a curdmungeonly small-minded person. He's got to accept and embrace newer technologies for the specific purposes of reducing the amount of work he has to do.

If the sysadmins are resisting, it's because it means it's more work, or they're simply not up to the task of newer things. The distinction may be hard to understand by someone who doesn't do sysadmin work. Often, changing to a new approach means they won't be able to get the things done they know they need to get done in a timely fashion if they comply with some policy pushed down from above.

Having just looked up what this DevOps cult is, it's largely just good principle, from a systems perspective. Often, however, those things aren't possible in an environment without significant ground-up rebuilds of not only the systems but the networks. Virtual infrastructure requires storage and concentration of machines into fewer hosts. This means network ops needs to be involved. (They're even more stick-in-the-mud and difficult to work with than sysadmins, IME.)

Re:Cult of DevOps? (1)

packetrat (245386) | more than 2 years ago | (#37584166)

I'm not sure I can take anybody who calls an attempt to make IT and Development more aware of each other a cult, seriously. .

Actually, the people who were espousing devops were the ones who called it the "cult of devops". I'm not hating, just wondering if it's just relabeling.

It is always IT's fault (1)

sgt scrub (869860) | more than 2 years ago | (#37576222)

are most IT organizations really that poorly run that developers are totally unaware their software is sending messages that are generating network storms

How is it the IT organizations fault the developers are unaware of the effect of their software? Is it the IT departments responsibility to debug software?

Re:It is always IT's fault (1)

bsane (148894) | more than 2 years ago | (#37576282)

Is it the IT departments responsibility to debug software?

Where I work? Yes.

Re:It is always IT's fault (1)

Junta (36770) | more than 2 years ago | (#37576284)

Well, solely blaming 'IT organizations' for developer unawareness is unfair, but in answer to you second question, yes, a good IT department can and will debug software to the extent it does not take down other services in the process. Sometimes a behavior cannot be readily replicated and knowing enough to capture viable debugging data before having to revert it to a working state can be critical. You don't want to leave it messed up until a 'developer' can get to it because you need it back to working ASAP and the bad condition might be transient anyway.

Outsourced offshore software is at fault again. (0)

Anonymous Coward | more than 2 years ago | (#37576360)

These problems are, by far, much worse when the software is developed via outsourcing, and the worst when this outsourcing is done offshore.

There's absolutely no incentive for a programmer in Bangalore or Pune or Mumbai to give a damn about software he wrote that's running in NYC or London, operated by people at some other company. There's no incentive for him to even bother to develop the software correctly. He won't have to deal with the system when it fails. He won't have to deal with the angry administrators or users. He won't have to deal with the costs the faulty software will result in.

In fact, it's in this offshore programmer's best interest to not even learn how to develop software properly in the first place. After all, that'd take some small degree of effort.

At least in-house American or British developers have to deal with the people affected by their mistakes. They have to see their co-workers suffer. They have to hear the upper management complain. It hurts their pride and sense of professionalism when such mistakes are made. These factors combine to form great incentive to learn how to do the job properly, and then to actually go ahead and do it properly. While American- or European-developed software isn't perfect, anyone who has worked with it will know just how much better it always is than the crap developed in third-world nations.

Re:It is always IT's fault (3, Interesting)

bobaferret (513897) | more than 2 years ago | (#37576372)

I think this is the whole point of the devops stuff. It's not IT's fault, it's not the developer's fault, nor is it QA's fault. The larger your organization, the larger communications gap between these 3 areas. The devops folks are there to coordinate and communicate between the three. They may review/write a little code, they may consult on infrastructure, they may make sure things get tested appropriately and meet the customers expectations. For small shops/teams this fairly straight forward. Big shops have to have someone who's job is to know everything about the project. There is a similar position in commercial building. It's the "site manager" they make sure that the contractor is building what the architect asked for. From the number of bathroom stalls, to the depth of the foundation. Which at the same time working with the architect to fix problems that arise from misplaced city sewers lines, an unavailability italian terrazzo tiles in rural Illinois. All the while making sure that the customer is getting what they paid for and understanding and approving changes. Sorry I didn't have a good car analogy.....

Very Well Put! (1)

leftover (210560) | more than 2 years ago | (#37578674)

Wish I had mod points handy.

The separate department structure for computer systems encourages everyone to take the approach of 'Not My Problem' even when they could fix it. From a broader perspective, the barriers among Dev/QA/Ops are artificial: it is all computer people doing computer-ish stuff. External constraints are vastly simpler than the construction analogy -- no opcode shipments stuck in Customs, no delays from rain or freezing temperatures. If the computer industry can not overcome these lesser barriers there is little hope of handling bigger barriers such as Marketing|Development.

And they really need to be overcome. Industry has spent a truly astonishing amount of money on 'computer systems' with frequently negative results. Recall the number of anecdotal reports of large computer system efforts that are abandoned after YEARS of work for many millions of dollars. Although the computer system company has a 'loss' of anticipated future revenue from the project, the host company loses real money and irreplaceable time. Further, they are often left with a confused and discouraged work force that will affect them for years to come.

Re:It is always IT's fault (1)

guruevi (827432) | more than 2 years ago | (#37579150)

The problem is not that organizations are large or any of that. Small shops have the same problem. The problem is management. It's a manager's job to make sure the lines of communications up/down and horizontal are open and that his team gets the resources (whether physical or not) it needs to do a better job. SOOOO many managers fail at that. Some think they are good programmers and get in the way of architecting the software, others think they are good talkers and start acting as a filter. That's why there are so many managers because 80% of them micro-manage their domain that they're inflexible at them or their team taking additional tasks.

There is no reason that most dev, QA and test teams shouldn't have a single manager while each team has their own lead. However I see often that each team has their own manager, the 3 teams have a manager, IT has their own teams, managers and there is usually another manager or 2 and a VP leading the whole "IT" governed by a CIO. That's 9 managers for a group that is maybe 20-50 people large.

Re:It is always IT's fault (1)

Rich0 (548339) | more than 2 years ago | (#37582638)

Maybe my workplace is just unusual, but the problems I've seen include:

1. A chargeback system that discourages anybody from helping anybody else with anything unless that help is formalized and expensive and planned in advance.

2. Since only the most routine functions of internal service organizations actually get formalized and planned in advance, the ability to do anything else gets whittled out of those organizations in the name of efficiency. The windows guys know how to create/edit groups, the oracle guys know how to create schemas, and the workstation guys know how to reimage. Heaven help you when a new Windows version comes along and you have to figure out how to re-engineer all that stuff again.

3. So, when a complex problem comes along, all anybody can do is scratch their head and talk about hiring a consultant, assuming it even gets noticed.

Re:It is always IT's fault (1)

shish (588640) | more than 2 years ago | (#37576410)

How is it the IT organizations fault the developers are unaware of the effect of their software?

Because they didn't take the time to send a quick email saying "hey, your code breaks in production"?

As far as I can see, DevOps is just a fancy name for "people who are working together on the same project should probably talk to each other", and I'm amazed at the number of people who are surprised by or even hostile towards this suggestion.

This worked 4 me 29 times since 1996 (0)

Anonymous Coward | more than 2 years ago | (#37576702)

Professionally, & for "enterprise-class" business development solutions done in Client-Server designs (or more complex ones using Terminal Server/Citrix, or going "cross-platform" to larger systems (think "Big Iron" midrange/mainframe class stuff thru middlewares)) & the "general overall breakdown" (hopefully not missing too many parts, I had to think about it) of who does what & when:

BUSINESS/SYSTEM ANALYST RESPONSIBILITIES: Know the projects' data & processing requirements for input and what needs to be done for outputs, Where the pertinent data resides (if it needs conversions or batching, or if it can be done via pure SQL straight out of a database etc.) AND, who needs to get to it, & when (sometimes for security, this is necessary too). Use cases come into play here to go to mgt. after speaking to end users &/or dept. heads also (use cases usually go to mgt.).

DBA RESPONSIBILITIES: Make sure the data is accessible (or filtered off in a "view" instead, only showing the pertinent material but not all (possibly confidential info., etc.)) & setup stored procedures IF the coders are not allowed to, many are though (better than SQLExec calls directly from code both in terms of security AND network traffic). Sometimes, this means creating views, reindexing (create &/or drop) indexes, tables, & a lot more... This is one of the crucial FOUNDATION parts really (because if the overall schema of it is bad, it can NEVER be a good efficient system imo).

SOFTWARE ENGINEER/ARCHITECT/LEAD DEVELOPER RESPONSIBILITIES: In combination with the DBA + System/Business Analysts inputs? This is the guy that makes the overall design of the package "go" - They're the guys that make "the foundations" work & create the overall design architecture in data AND CODE for it to work (coding goes along with the ride here too). Use cases come into play here also as an "aid to design". Delegation of workload & responsibilities to subordinates & schedules for deadline come into play for they too here.

PROGRAMMER/ANALYST RESPONSIBILITIES: Write the code that the System Analyst, Software Engineer, + DBA outline to you as to process & data requirements, that makes the "show run" from the front end (generally) so users can utilize it for business planning &/or decision making.

PROJECT LEADS &/or MANAGEMENT RESPONSIBILITIES: Make sure all goes above as planned, on schedule, & not over-budget (and to "run interference" for dev. teams/DBA's, with the end users/customers). They also make decisions based on use cases & other constraints, & sign off on the whole show to go into motion (or, to "kill it" too).

---

Then, you have a "bug testing/killing" cycle, via a Q/A dept. (IF it exists, many times it does not) & regression testing...

OR

You let a select group of primary "end users" beat up on it for a week or two in a testbed environs, offering suggestions for changes they need (be they useability issues, or actual problems in data results (the most important part is the latter)).

FINALLY - THIS IS WHERE "IT" COMES IN:

They have to do the hardware & networking requirements, initial deployment (once they're informed on what rights etc. users need (if need be, making new usergroups for easier mgt.)) &, that the data is safe/secured (& backed up), AND, where analysis of network segment packet traffic can be an "issue" (which is part of HOW/WHY stored procedures help a LOT vs. "packet storms") because here, they can determine if a network segment(s) needs to be "busted up" more into more manageable & monitorable parts (especially in the case of network congestion), OR if query designs need improvements (vs. packet storms as the article calls them) - this last part alone can STOP having to use QoS as a "bandaid on a bulletwound" in fact, many times... they also do the maintenance later for users too.

* So - Once a larger program "passes muster" for those things above, & projects of size take more than 1 yr. usually? Then, you can "let it rip"... & I've NEVER, EVER had one fail in production, with some running for more than 15++ yrs. solid @ this point in time in fact.

(In the end people? It's a teamwork thing - with lots of moving parts... & everyone has a job(s) to do to make it happen - So, did I hit ALL of the parts & tasks? Probably not, but that's what I've seen based on a career of doing that line of work for decades now...)

APK

P.S.=> Could 1 guy do ALL that? Yes, but it'd take a lot longer, probably not make deadline or budget either... it does help having "many eyes" pore over the entire process as well...

Now, I don't know how many of you have built systems of the nature I speak of here (750,000++ - many millions of lines or more of code (front-end stuff), + 100,000++ or more lines of SQL in stored procs (& upwards), & documentation etc./et al), but that's the process, & personnel breakdown I've seen actually WORK & do well over nearly 30 such type systems I've written since 1996 professionally (mostly co-written in Client-Server designs, because of the SIZE involved & the # of "moving parts")...

... apk

Re:It is always IT's fault (1)

mikael (484) | more than 2 years ago | (#37576964)

The developers are using third party API's which can do all sorts of odd things, like send out multicasts or broadcasts for servers on the local network (eg. 127.0.0.1 -> xx.yy.zz.255, xx.yy.255.255, xx.255.255.255), while sys-admins are responsible for the network topology including firewalls, bridges, routers, and general IP routing.

Sysadmins fail to set up the IP filter tables correctly, and local broadcasts intended for an small network can go circulating around the entire corporate network.
Any in-house protocol can implement a ping/whois feature to process multicast/broadcast messages. One stray packet, and ten thousand machines try to reply, but the messages aren't received because the replies don't get past the correctly set IP filters.

It's bad enough with admin's submitting staff E-mails to every mailing list that gets created, only to get a "please remove me" E-mail storm every afternoon.

Re:It is always IT's fault (1)

yourmommycalled (2280728) | more than 2 years ago | (#37577480)

Is it the developers fault that IT refused to listen when the developer described EXACTLY what the system configuration should be for the software to work, but IT MANAGEMENT changed the specs because IT MANAGEMENT knows more about the developers software than the developer? Sorry about the rant, but I ordered a visualization system using my grant money and IT MANAGEMENT changed the PO into a thin client claiming I didn't need anything more powerful and then said "what's this OpenGL stuff you'r talking about"

Re:It is always IT's fault (0)

Anonymous Coward | more than 2 years ago | (#37577914)

Do you work at the University of South Florida?

Re:It is always IT's fault (0)

Anonymous Coward | more than 2 years ago | (#37577614)

How is it the IT organizations fault the developers are unaware of the effect of their software? Is it the IT departments responsibility to debug software

Development is part of IT, just like Operations is part of IT. You appear to be operating under the misconception that IT == IT Operations.

Recently (2)

prefec2 (875483) | more than 2 years ago | (#37576242)

A while ago I thought most OSS application and framework projects including such Gnome and KDE are in trouble, due to the large use of the fumble-around development approach. Also known as the first code then think approach. All the great model-based, model-driven and agile development methods seam to be far away from the way many OSS projects are developed.

However, lately I came out of my ivory tower and stood eye in eye with experts from the industry who largely believe that they are professionals and really do great stuff. They also use the fumbling approach. The main difference is the call it agile. Even though it is far away from such an approach. I always thought that one of the problems of our new economy company back in the 2000 originated from being too deep in code and too light on design, planning and documentation. But it looks like, that tinkering is a more widespread way of software development. So I guess that leaves me with bad management (which was not my responsibility).

Most of these tinkering approaches originate in the absence of developer discipline and the "Add this quick"-management method. but I am telling nothing new. We all know for decades now what is wrong in software development. A lot of people wrote books on patterns (design and otherwise), but in the end if no one follows these patterns the problems remain.

The "Add this quick"-management originates at large by a misconception of IT and its importance for businesses and other organizations.

Re:Recently (1)

Lennie (16154) | more than 2 years ago | (#37576290)

These problems will always exists.

You can not predict everything and you can't generate all possible testloads.

There is always friction between:
- adding features
- lowering operational expenses
- trying to predict what part will fail first under load, increasing scalability
- deploying quick fixes because you did not anticipate something
- deploying real fixes for quick fixes

I could be wrong of course :-)

Re:Recently (1)

hitmark (640295) | more than 2 years ago | (#37576352)

The last one appears to almost never happen. One just end up with a house of cards in the form of quick fixes until the whole system is scrapped and a new one erected (likely because some manager got a nice dinner out of a silver tongued sales rep).

Re:Recently (2)

DrgnDancer (137700) | more than 2 years ago | (#37576358)

It's like anything else. The better your initial plan or design the less likely you are to get surprised and the more likely that the surprises will be small. There's an old saying about plans in the military "The best plans only last until the first bullet flies" and it's true... but that doesn't mean I didn't spend hours and hour planning operations. Sure nothing ever goes exactly according to plan, but a good plan considers as many variants and contingencies as possible in order to anticipate reactions and minimize disruptions to operations.

Software design is the same. The more time you spend planning and designing, the more likely the final product is to look like the design. It's never going to free the process entirely of surprises and gotchas, but it can minimize them and help to predict how to fit them in. Of course if you keep working toward the "perfect" plan you'll never get anywhere, but spending at least part of your development life cycle on a realistic design rarely hurts and often helps.

Re:Recently (1)

Lennie (16154) | more than 2 years ago | (#37576426)

Now that I think about, getting back to what the article is about.

I think the point of the article is: know the environment where your code gets deployed and have some idea of how it will probably get used and actually monitor that so you can better adjust.

An other thing I got from the article is that this does not really fit with the idea of cloud computing.

Most people would have no idea how the environment looks where their code will run if it 'runs in the cloud'.

Re:Recently (1)

prefec2 (875483) | more than 2 years ago | (#37577158)

You can do all the stuff in the cloud as well. However, I do not know why this cloud thing is such a hype. But that's a different story. Good cloud environment provide you with the right information on provided performance properties and the avail virtual machine/platform. The problem is that people think in code. And code is not good to think in. Especially when you have to handle multiple sets of aspects which even are not all part of the implementation, but of the requirements to the application.

I know this is not the center argument of the article, but it is my opinion on why so many projects fail. They code first, they never think. And they never check if their model conforms to the requirements they had in the beginning. And this applies to agile project even more.

Re:Recently (1)

Chris Mattern (191822) | more than 2 years ago | (#37578950)

There's an old saying about plans in the military "The best plans only last until the first bullet flies" and it's true... but that doesn't mean I didn't spend hours and hour planning operations. Sure nothing ever goes exactly according to plan, but a good plan considers as many variants and contingencies as possible in order to anticipate reactions and minimize disruptions to operations.

To quote Eisenhower, who possibly faced the biggest challenges of this sort one man ever had to: "Plans are worthless, but planning is everything."

Re:Recently (1)

sourcerror (1718066) | more than 2 years ago | (#37576316)

"However, lately I came out of my ivory tower and stood eye in eye with experts from the industry who largely believe that they are professionals and really do great stuff. They also use the fumbling approach. The main difference is the call it agile."

You mean, cowboy coding is now called agile? I thought Agile still has documentation/process just somewhat lighter than RUP/waterfall.

Re:Recently (3, Interesting)

Junta (36770) | more than 2 years ago | (#37576334)

Also known as the first code then think approach

No, it's think *as* you code. You do think some before starting to code, a vague rough picture of the pieces, but you don't invest a lot of time in that 'pure' design phase because the more detailed you plan without proving it out, the more you *should* throw away as you start implementation and realize how ill-conceived your design was or that maybe your design was adequate, but a better way makes itself apparent when actually implementing. Generally when I see a development team incapable of operating at all in this manner and fail to achieve any measure of 'complete', they only 'complete' under other strategies of development on a technicality and produce low quality product. Some management people think that methodology can be used to make piss-poor (cheap) developers serviceable and avoid having to reward/retain good talent.

A lot of people wrote books on patterns (design and otherwise), but in the end if no one follows these patterns the problems remain.

I think there are two aspects to this. One is that most development an organization is predestined to operate in a specifc way depending on who comprises it, regardless of what name they pick to describe it. I know organizations that did 'waterfall' and 'changed' to 'Agile', but really just renamed things they did, acted the same, and called it 'Agile'. However, I don't know if this is a bad thing. I think getting too hung up on a specific 'pattern' others preach in conferences is bad, and organically feeling your way for a process that works for your team is better. Awareness is good, but getting locked in is bad.

However, I have found in the sea of Agile and Waterfall and all sorts of buzzwords a methodology that really resonates with me:
http://programming-motherfucker.com/ [programmin...fucker.com]

Re:Recently (2)

dbIII (701233) | more than 2 years ago | (#37576376)

It makes me think of a conversation from quite a few years ago.
"Flowcharts? What is it with you engineers. It's software, it's not as if we're building a bridge or anything. We just keep on typing stuff in until it happens and that gets the job done. If it doesn't work we can clean it up later."

Re:Recently (1)

Rich0 (548339) | more than 2 years ago | (#37576388)

Yup - we use "agile" but we still start with a requirements spec and all the requirements are written before the first line of code. In fact, even if we have multiple "sprints" we still write all the requirements before we start the first one.

The only thing agile about it is that we use the right buzzwords, and we do a chunk of requirements at a time (as if that wasn't what everybody always did), and we demo the software to customers and maybe refine the requirements from time to time (as if that wasn't what we always did anyway).

Re:Recently (1)

russotto (537200) | more than 2 years ago | (#37579418)

Some management people think that methodology can be used to make piss-poor (cheap) developers serviceable and avoid having to reward/retain good talent.

That's exactly the point of methodology. It is to reduce the process of software development to a bunch of rote steps which low-skill people can perform consistently. Assembly-line software.

Re:Recently (1)

CAIMLAS (41445) | more than 2 years ago | (#37581758)

As a sysadmin dealing with these so-called 'agile' developers, I'd mod you through the fucking sky, if I could.

You describe it as it is. They write what I would call "quick one-off shit scripts/programs" and call it production quality code. Sure, it gets a 'finished product' done quickly, but it sure as hell won't be fun for the people who get hired on months later to fix it.

People need to realize that custom applications take a long damn time to develop properly. Maintaining custom software is expensive largely due to the time involved to do it right. As in most things in lunch, there is no such thing as 'a free lunch'.

You are kidding, right? (2)

sanborn's man (687059) | more than 2 years ago | (#37576244)

The answer to that question is "yes". In my experience what is it uncommon is to find a truly competent IT organization. And normaly the biggest it is the worse the problems are. But, what could you expect in an age where your IT group is composed with the cheapest guys you could hire with no experienced (and more costly) ones?. Or what about the clueless IT managers with little or no experience on IT?. They can't plan against what they don't know. Yeah, as a colleague told me not long ago, IT life is nowadays like war: long periods of boredom punctuated by moments of sheer terror.

My guess is (1)

garaged (579941) | more than 2 years ago | (#37576252)

From what I've seen (I work as sysadmin), that most of the problems are caused by the rotation of people, companies are not willing to give value to experience, and always want to make a new bet, switching contractors, giving emplooyees management positions and letting total noobs do all the jobs that actually matter (but is not visible?).

As soon as someone has the right skills and internal knowledge, he deserves a better wage, but only a tiny fraction actually make it there, it has been posted here, rotation is the only way to improve salary, and people are willing to pay twice as much for contractors than for employees

The curse of 'enterprise' (3, Insightful)

Junta (36770) | more than 2 years ago | (#37576262)

I've been around enough to know that this 'new' 'DevOps' philosophy is just the way it always has been done at many successful companies making extensive use of technology.

I have come to associate the phrase 'enterprise IT' with those who *don't* work that way, and make their lives needlessly complicated. Of course, every last party to the mess will generally recognize it and know what could hypothetically be done about it, but only bitch in private about it and rarely ever push for meaningful change. The reason is simple, so long as it is a complicated mess, it requires a great deal of human care and feeding, meaning job security. Management can't force things to change without huge risk as everyone has sufficiently entrenched themselves.

Re:The curse of 'enterprise' (2)

dkleinsc (563838) | more than 2 years ago | (#37576406)

Size of an organization definitely also makes a real practical difference.

When you're part of (as I am) a technical staff small enough to all fit in the same room, it's very easy to consult across the admin-developer boundary, along the lines of a quick phone call or stop by somebody's desk. Admins and devs regularly interact, both professionally and socially. That also creates a culture where the admins and devs are very much on the same team, are working towards the same goals, and collectively creating less suckitude. Someone who tried to hold back on helping out someone on the other side would find themselves in hot water.

When you're part of a giant technical staff, though, it's much easier to get silo effects, where in order to consult across the dev-admin boundary you have to go up your chain of command until you get somebody who knows somebody on the other side, then work your way down the other chain of command until you find somebody who can actually work with you on solving the problem. And chances are not tiny that you'll encounter somebody somewhere along those chains of commands that really doesn't like the cross-team work (officially because it's allowing people to work outside of their area of expertise, but probably because they think it will help job security, as you suggested).

For instance, consider a dev who's trying to make sure they're not doing something stupid with a database. In a smaller organization, they might do something like "Hey, DBAs, is the plan on this query hitting the indexes I think it's supposed to if we run it in production, and is that a reasonably fast way of doing it?" In a large organization, this may be a multi-day multi-meeting effort to track down somebody, so that dev may skip that step so that they can meet their deadline and find that what they did that worked perfectly well in a development or test environment wreaked havoc in the production environment.

Re:The curse of 'enterprise' (0)

Anonymous Coward | more than 2 years ago | (#37576574)

At one job a buddy was a smoker, and there were 1 or 2 smokers from pretty much every team in the building. I actually started smoking, because it sometimes was the most productive 10 minutes of the day, you could go down there and ask why things were done this way, or I had an idea about something, does this make any sense, etc. etc. etc.

And of course it worked both ways, we solved a couple of exquisitely annoying user problems that we'd never have heard about otherwise, like the system that took 20 minutes to return a query. Now that system returned the proper results, just agonizingly slowly, and their boss never escalated the issue, we didn't even know it was a problem. If everything had followed "proper" channels it would never have gotten fixed. Those 2 or 3 dozen people were very happy when we got it down to under a minute.

Anyway, if you can build informal channels in a big organization go for it.

rhetorical question (1)

Anonymous Coward | more than 2 years ago | (#37576278)

That last was a rhetorical question, right? You could have stopped at "Are most IT organizations really that poorly run". Yes. Upwards of 95% of CIOs are completely unqualified to hold their position. They are grossly overpaid talking heads with NO real understanding of the technical underpinnings.

Seriously. The single largest problem that most IT organizations have is that the people running them are completely unqualified to execute the duties required. It is not that IT workers are overpaid. Good IT workers are worth their pay several times over. The problem is that far too many are overpaid for their actual skill and knowledge level.

I like to compare current IT workers to the medical personnel of the 1700s. Many of them had some extremely rudimentary knowledge of how a body worked, but much of what many of them believed was rubbish and they had no idea about the depths of their ignorance. They have good intentions, but they are going to milk you for every penny they can and many honestly believe they deserve the pay.

Finally, those at the "top" of the food chain, highest in management, are also most likely to be the most unqualified. They rose to their position through personal connections and snake-oil patter.

Until IT becomes a disciplined hard science with the same kind of internal reviews and requirements as the medical field, you are going to have an awful lot of hedge-witches, snake-oil salesmen, and field medics running around causing more harm than good.

More methodologies! (1)

pinkeen (1804300) | more than 2 years ago | (#37576280)

Yeah, that's exactly what we need. More rules, more books describing how to do *doing* something. Meta-meta-meta-everything...

And more companies that take a methodology which has quite sensible premises and transform it to a paper-pushing-based freak-child.

Yes (0)

Anonymous Coward | more than 2 years ago | (#37576310)

...developers are totally unaware...

You're supposed to sniff the kool-aid, not drink it FYI.

The real issue is the word developer. Nowadays that can mean anyone.

We need special notations like developer+1 (extra dice) or developer^5 (power of five developers)... also because the limited ANSI character set must be conquered in the face of mathematical oppression. Yeha!

it's not about having a clue (1)

superwiz (655733) | more than 2 years ago | (#37576448)

Whenever you see someone dismissing what you think is a genuine concern, it's not because they don't get it. It's because they don't care. Everyone has their own priorities. Dev's priority is functionality. Security is just a necessary burden. But no one prioritizes burdens. Managing priorities is a management job. So if you see a system in which important priorities are not given weight, that's just poor management. Management's job is identifying priorities and then creating an incentive structure (I don't mean salary, I mean day-to-day incentives) which emphasizes the important and de-emphasizing the unimportant. Btw, that doesn't mean that management gets to ask everyone "what are your priorities?" and pretend they have done their part. Asking that question is tantamount to shifting the burden of identifying the priorities to those who don't have that responsibility.

oblig. Fred Brookes quote (1)

Anonymous Coward | more than 2 years ago | (#37576494)

The project was large enough and management communications poor enough to prompt many members of the team to see themselves as contestants making brownie points, rather than as builders making programming products. Each suboptimized his piece to meet his targets; few stopped to think about the total effect on the customer. This breakdown in orientation and communication is a major hazard for large projects.

This cancerous attitude happens in any project, across any set of roles, once things get big enough. If the organizational culture is rigidly defined roles and CYA, this crap is inevitable.

There's no silver bullet for this obviously, this is a pure management problem.

Re:oblig. Fred Brookes quote (0)

Anonymous Coward | more than 2 years ago | (#37576594)

You're right - you can shoot the managers with ordinary bullets. No silver bullets required.

No Overseer??? (0)

Anonymous Coward | more than 2 years ago | (#37576688)

I am little concerned about Fried and his conclusions. He ran all these specialized silos without an Architect(s)?? Of course if was cluster fuck, that is the dumbest thing I ever heard of. And yet this guy kept his job at the Bank and then got a job at Google.

Here is how you are suppose to do things, ass hat. Each silo has a Architect/Manager who oversees the generalist that run the day to day ops in that silo. These silos would be Voice, Network, AppDev, Server/Storage, whatever works for you. Implement PMO and Change Control. Then Once a day/week all the Architects get together to decide any changes to anything. All items are discussed and any Architect is allowed to ask questions and raise concerns. Think knights of the round table.

It's been my experience... (0)

Anonymous Coward | more than 2 years ago | (#37576914)

Trust me: all IT shops are mismanaged and all for the purpose of making a buck or appeasing some fat fuck and his or her wallet / purse... The IT market in America is a complete mess--much like everything else at the moment--but the good thing is that it really is a sleeping giant. The capabilities in this country has staggering amounts of potential, and as soon as it makes its way through the constant barrage of red tape and time-wasting policies, it will be unstoppable... Unfortunately for right now at least, it's going through that phase of maturity where the Baby Boomers--those that know nothing about computers--are holding all the pivotal positions responsible for the very mismanagement of things. Once they die-off, people of more modernized ilks will take over and re-factor processes to leverage this dormant cache I'm eluding to... That's when things will change. Mark my word...

Not a triangle (1)

loufoque (1400831) | more than 2 years ago | (#37576968)

They see things as a triangle: developers, Q&A and IT, and in the middle DevOps.

That's not at all how it works from my experience.
While developers and Q&A are very close, developers are not particularly close to IT at all, while Q&A is.

Testing is what requires large infrastructure set-ups. Development doesn't really: the Q&A guy directly gives you access to a working configuration that demonstrates the problem. As a software developer I never have to interact with IT.

Re:Not a triangle (1)

SuperQ (431) | more than 2 years ago | (#37578820)

And that's the whole point of DevOps. To bridge the gap and bring the teams closer together. This is one of those "if you don't know who the chump is, you're the chump" situations. Because you haven't experienced it means your company(s) are missing DevOps and the whole point behind it.

Get rid of dev/qa/ops silos.

IT is poorly run? Sky is blue? (0)

Anonymous Coward | more than 2 years ago | (#37577242)

... are most IT organizations really that poorly run[?]

Well, yes. This is news to you?

The biggest problem I saw at my workplace (0)

Anonymous Coward | more than 2 years ago | (#37577330)

The biggest problem I saw at my workplace, was lack of vision from the direction. They don't have clear goals set so they don't have a strategy to get there and they don't have tactics to wins the battles over the different road-block. They wait until they are hold against the wall to make a random decision, they don't listen to the feedback from the talented professional they hired, they kill home-grown project that are on budget and on time only to buy some monstrosity from commercial "partners" and they allocate more resource to failed project of little strategic importance...

I worked hard to change the mentality of my devs, and I succeeded. I single-handedly brought our dev team from outdated and badly followed 1990 practices to 2010 practices that are followed automatically as almost the enforcement is automated, we have gain a lot's of velocity but we are now idle as they cannot plan as fast as we implement, since they as I said they lack vision.

Dripping with Contempt (1)

Anonymous Coward | more than 2 years ago | (#37577530)

In an attempt to answer the barely articulated question buried amidst the run-on editorializing, I will answer "yes". Many otherwise extremely talented developers carry little understanding of how their code performs once exposed to chaotic and nondeterministic production environments. Likewise many ops engineers hold little appreciation for the motivations of and the pressures on those developers.

Much of this isolation stems from culture, but at least as much is from experience. Being paged at 3am because an entire pool of servers simultaneously ran out of file handles or because some unknown engineer failed to properly sanitize their input tends to temper ones soul. Having your deployments constantly stalled because one failed to meet some ill-defined barely document seemingly nebulous operational requirement too tends to galvanize ones opinion about "those people". Both teams scream, "they just don't get it," I lot. And they are both right.

The result if these tentions are unaddressed is clear. Many live system outages and stymied projects.

Devops, cult or not, is a conscious effort to bridge these frequently opposing forces inside technical organizations. The teetering platform of fast, cheap, and good need not be out of balance. Rather it is the express job of the devops engineer, by having equal interests both in execution and in sustainability, to create a world where deployment happens freely and fast, where quality is not sacrificed, and where we are not mindlessly throwing money at the inefficiencies created by conflict.

The kool-aid is actually a pretty tasty beverage. You might take a sip.

DevOps is a security violation (0)

Anonymous Coward | more than 2 years ago | (#37577972)

A gig I had at one of the Big 5 Banks, devs had no access to production as part of the security policy. In at least one case the result was a SysAdmin's (whom had previously worked for Morgan Stanely) solution which allowed for unlogged redirection of pages (also a violation of the security policy).

That same gig one of their "outstanding successes" was .Net app (not in source control) that needed to be rebooted every 45 minutes to keep from crashing. It was quietly rewritten and retired within 6 months of launch.

Re:DevOps is a security violation (1)

Opportunist (166417) | more than 2 years ago | (#37582516)

This sounds more like a blunder on the IT-Sec's department than anyone else's.

Security is important (hey, I can't speak against my trade, can I?), but the moment it keeps people from being able to do their work, they will start looking for workarounds around the security system. And this is about the WORST thing that you could possibly have as an IT-Sec. It's akin to an inside job, only without the intent to actually cause trouble.

But intent doesn't really matter when the data is lost...

Agile - no "deep cross-departmental integration" (1)

ex-googler (2090078) | more than 2 years ago | (#37578344)

The wiki page on "devops" sums up part of the idea to involve "deep cross-departmental integration"

While there is a lot of agile practises around Google, including "stand-ups", iterations, scrum, planning poker, and whatever, there is hardly any "cross-departmental integration". Communication is as poor as at any other huge behemoth. Just like we've read about multiple internal teams fighting each other at Microsoft and Nokia, so do many teams at Google create similar products, both externally and internally.

In fact, there are multiple projects and initiatives to cut down on duplicated projects. Think about that one for a second.

architects of technology vs architects of product (1)

Anonymous Coward | more than 2 years ago | (#37578468)

the software industry in silicon valley by and large does not have a role 'architect of product'. we have architects of technology. the product comes together as various teams: tech pubs, qa, tech support, professional services, it, ops, manufacturing, marketing -- all contribute their efforts to take the technology delivered by the typical development team and transform it into a product. DevOps is one attempt at mitigating or filling this gap to varying degrees depending on the folks involved. Now what is interesting is, "where does DevOps" report into? CIO? is so, where does Development report into? and now we are back to ownership...

DevOps? No thanks. (0)

Anonymous Coward | more than 2 years ago | (#37579274)

Yeah, I'm a longtime sysadmin, and yeah, I'm against this whole DevOps thing. Why? Because over the years I've learned that while developers can write some awesome code, they're generally shit when it comes to managing production systems. They leave development tools scattered about the system. They install Linux web servers with X11 enabled. They solve problems by writing code first, rather than analysing the problem properly. Don't get me started about the large Drupal installation I just overhauled, where the DevOps team went and coded the crap out of everything, rather than using the tools that Drupal and the Drupal devs provide.

In short: No DevOps are allowed anywhere near any of my systems.

DevOPS sounds like how things should be done (0)

Anonymous Coward | more than 2 years ago | (#37581696)

I do realize some businesses have a completely seperate IT department that is responsible basically for hardware and only hardware, and some other group that develops/sets up the software required to run the business. The developers then are expected and end up treating the actual servers, the storage, the network, as "black boxes" that the stuff just runs on. This has several immediate problems:

          1) If the software has peculiar timing requirements, or gets put under heavy enough load, this whole "black box" thing falls apart, as either CPU cycles, perhaps RAM, storage speed or network speed becomes inadequate for the job. These may not fail gracefully, and diagnosis can become difficult when the develoeprs and IT are so isolated from each other.

          2) Various other fails, I've heard of a few examples (on The Register) where some place loved virtualization, they'd implement a primary VM and a failover, but nobody bothered to realize they were BOTH running on the same physical machine! Or, something makes "backups" but they are actually just sitting on the same storage array.

          Honeslty, if some project is going to be large enough to need at least one dedicated machine, then those who develop it should have carte blanche over what kind of hardware it is running on. If they wish to call this DevOps and pretend it's a new thing, then so be it. I can say, when I worked at a university surplus department, we kept university IT far away from the systems we ran our internal use software on.

Cult of the Dead Cow (0)

Anonymous Coward | more than 2 years ago | (#37582024)

Moo00o0oo0o000o

The "it's not my problem" problem (2)

Opportunist (166417) | more than 2 years ago | (#37582504)

It can be summed up like that: If the problem will arise in an area he isn't responsible for, and if implementing it that way regardless is cheaper, faster or less problematic for his department, he will implement it that way.

Running the traffic through the internet instead of the LAN? Why not? It's not going to be the development's problem (since that's operation's problem), and it's faster to do it that way, so it's done that way. A user interface that makes users beg to break the programmer's fingers and make him a consultant? Not the programmer's problem, it's the user's problem, and if he can implement it faster with a shabby interface, it will be done that way.

The problem isn't that the people wouldn't know that these problems will surface. Often they know that quite well. The problem is that this is not a criterion, while the time they spend on solving the problem is. Hence, whatever lets him get it faster off his table will be the way it is done.

Sure I'll comment... (1)

Foolomon (855512) | more than 2 years ago | (#37584938)

...and I'll even try to remember to check back in case people respond.

Assumption: the trading application being critiqued is the same one that was there when I was an IT consultant at Morgan Stanley. I left in August 2000.

I know the application well. It was developed by a department headed up by Vinny (whose name is withheld because...I'm senile and don't remember it). I worked in the department that wrote the messaging infrastructure that was used by every application on the sell side of the firm.

If the application is the same one then the mere fact that it was still in use when Mr. Fried left is a testament to the application's effectiveness. Would they do it better now if they were writing it from scratch? Of course. But Morgan Stanley has 3,000-4,000 IT staff in its ranks so they could easily do so if the application were as bad as he says. And the messaging infrastructure...well, I have no love for the original authors (no hard feelings Steve and Arthur...I'm lying of course), but that subsystem was extremely robust, predated TIBCO and Talarian, and provided more functionality that those two products until after they were on the market for several years.

don't care (1)

Nyder (754090) | more than 2 years ago | (#37585532)

cults are stupid.

if you want to name yourself something stupid, i don't care. Stupid people do stupid shit because they are stupid.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...