Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Who Owns Deployments - Dev or IT?

Cliff posted more than 7 years ago | from the custody-battles-are-always-nasty dept.

Businesses 152

txpenguin asks: "I am IT manager for a small software company. We host several generations of our applications in a fairly complex environment. Our systems are very much inter-dependent (clustering, replication, heavily loaded, and so forth), and bad changes tend to have a domino effect. Additionally, it seems that there are always those who need to be 'in the loop', but aren't aware of changes which affect them. There is a constant battle between IT and Development regarding who should handle the deployment of new code releases and database schema changes to production systems. Dev doesn't understand the systems, and IT does not know the code well. How do you handle this at your company? What protocols seem to work best? Can there be a middle ground?"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered


Who owns developments? (1)

Mipoti Gusundar (1028156) | more than 7 years ago | (#17221018)


Re:Who owns developments? (1, Insightful)

Anonymous Coward | more than 7 years ago | (#17221372)

We're talking about actual development here. That is, we're not talking about taking example code from Sun's website, changing a few class names and calling it a new and finished product. We're not talking about taking snippets of code, unattributed, from various incompatibly-licensed open source projects and combining them into a single crashing pile of shit.

We've seen shit like that the few times we've dealt with Indian software firms, and frankly their development practices are just plain unacceptable. And I'd hardly call it "development". It's more a case of "let's-fuck-up-because-we're-clueless-and-then-try -to-trick-our-clients-with-Sun's-example-code".

Middle ground (5, Insightful)

Southpaw018 (793465) | more than 7 years ago | (#17221036)

These kinds of things where there are two opposing sides always have the same answer. Unless one side is teh debil or something.

You have to compromise. That's it. Middle ground. There are no other solutions to or ways around this problem. As you describe it, each side has access to and knowledge of half the problem. Half plus half is whole!

So, meet with the guys in Dev. If you want to be beaureaucratic and official about it, create a "deployment team" consisting of an equal number of members from each side that will sit down, discuss, and supervise all necessary changes to production systems. Hell, send someone to a project management class if you need to.

Now, the obstacle you're likely to hit is office politics. People won't want to listen to others and/or won't want to give up their turf or allow others on it. Too bad. To place how serious this issue is in overcoming the political terms: everyone in both departments needs to be cooperating or unemployed.

So there you go. Just like any other relationship, business or otherwise: sit down and talk it over. Problems solved!

Re:Middle ground (3, Insightful)

popeyethesailor (325796) | more than 7 years ago | (#17221152)

There's no need for a compromise.
Developers write code & documentation.
Installation/Deployment guides *are* required documentation.
IT takes the software, and follows the guide.

Business applications are mostly consolidated on a few servers. IT guys know dependencies, time windows, batch runs and the whole shebang. A dev team has no business doing all this.

Re:Middle ground (4, Funny)

Schraegstrichpunkt (931443) | more than 7 years ago | (#17221232)

There's no need for a compromise.
Developers write code & documentation.
Installation/Deployment guides *are* required documentation.
IT takes the software, and follows the guide.

And in Magic Happy Land, that actually works without a problem.

Re:Middle ground (1)

popeyethesailor (325796) | more than 7 years ago | (#17221866)

And in Magic Happy Land, that actually works without a problem.

But it doesnt have to. That's why in a good shop, there's actually a process for rolling-back the install, contingency plans, business continuity and disaster recovery. It's not a trivial process; even more important that there's a clear separation of responsibilities.

Re:Middle ground (2, Insightful)

ComputerizedYoga (466024) | more than 7 years ago | (#17222300)

err, then there's also testing.

"rolling back the install" is a low-grade disaster recovery scenario. Testing the install on a non-production machine, working out the install/upgrade kinks and maybe even having a team of testers or some scripted testcases to throw at it before you start doing anything on the production systems is disaster prevention.

And any doctor, sysadmin, or person with a modicum of common sense (or at least familiarity with some common-sense aphorisms) will tell you something about the relationship between prevention and cure...

Yes it works.. (1)

Baikala (564096) | more than 7 years ago | (#17224068)

It may sound highly bureaucratic to you, and yes, for a small change is like 5x times more slow but it works if you establish some policies.

I work for a fairly sized enterprise. There are 20+ engineers in development and only 6 of us in "infrastructure" (sys admins). Development teams simply don't touch the Test or Production environments, they have their own domain and servers which we help them to run on a you-broke-it-you-fix-it policy. All code, db scripts, catalog data, etc is managed by a Change Management Application, there are 3 people in change management that do the compilation of code and only them are allowed to upload executables (dlls, exes, jars, ears).

When the test environment admins receive a Change Order this package has already been checked for completeness (including deployment documentation and End User Manual deltas). If the test env. admins are not happy with the deployment documentation or if any script fails for something that is not infrastructure related we simply reject the package to development. Of course development has a chance to rebate the rejection but we usually have evidence for the failure prepared by then.

Re:Middle ground (3, Informative)

mjpaci (33725) | more than 7 years ago | (#17222048)

My company (~2,500 ppl) has moved to this model. However, we have dedicated "deployment engineers" who take the developers' documentation and follow it to a T. If there is an issue, they roll back. At first nobody liked the idea, but as time progressed it has started to work well. The deployment engineers were taken from the server teams and the DBA team and formed into a new group of about 10 people.


Re:Middle ground (1, Funny)

Anonymous Coward | more than 7 years ago | (#17222266)

>IT takes the software, and follows the guide.

And then IT takes the lotion, and puts it on IT's skin.

Re:Middle ground (2, Insightful)

Doctor Memory (6336) | more than 7 years ago | (#17223956)

Installation/Deployment guides *are* required documentation.
IT takes the software, and follows the guide.
And note that these are guides, not step-by-step instructions. They should say things like "Load the database schema update script (app_schema_updt_1_1_19.sql) into the database". The actual mechanics of doing this (making a backup, bringing an additional transaction log file on-line, starting table-level auditing, whatever) are left to the people who actually know the system (and the department procedures). Part of the development process is circulating a draft of the installation guide to IT for comments. IT is free to ask "Why?" or "How?" or "Where am I supposed to get the disk space for that?". I always made a point of inviting someone from IT to our team meetings whenever we discussed deployment requirements, and that always worked well. IT got a heads-up as to what was coming down the pike, and often the IT person had good insight into alternatives and what resources were really available (e.g., "XYZ division got their own server, so all their information in the production system is static. They wanted it updated hourly, but the update locks those tables for over five minutes, so we're just doing it noon and midnight for now." Need I mention that our app was reporting against XYZ division's data, and this was the first we'd heard of it?).

Re:Middle ground (3, Insightful)

gbjbaanb (229885) | more than 7 years ago | (#17222250)

Neither. You do not have to compromise if you're the boss and you require stuff to work. Office politics and willy-waving over who's more important should be a secondary issue to making the stuff work.

So. you have a certification team (or quality team, or test team) who's job it is to certify that what dev has goven them works as dev said. These guys install it on their own separate systems that mirror the business (on a smaller scale) and test it out. Bugs get reported back to dev who get to fix them and so on. Eventually it'll get rolled out to IT who will have a reasonably good expectation that it'll all work.

However - even in the best of cases there will be exceptional circumstances, and its at this point that IT will get dev members to come and fix up issue that arise on the live system. IT should be first contacting the cert team who will determine the bug (hopefully with a bit more inside knowledge to reproduce it on their systems) and will then get dev to issue a patch, which goes through the standard release process.

Of course, if you want to let dev team hack about (which is probably why you have such a complex system in the first place), and IT to twiddle with their setup then fine - expect it all to go arse-up.

I like to think of these environments as always having a 'customer' that they deliver to. If they provide a poor service, the customer has every right to complain. So, Dev's customer is the IT guys, IT's customer is the Business, and Business answers to real, paying customers. Such a chain of responsibility does focus people's attention on what they are trying to achieve for the company.

Re:Middle ground (0)

Anonymous Coward | more than 7 years ago | (#17222692)

I like to think of these environments as always having a 'customer' that they deliver to. If they provide a poor service, the customer has every right to complain. So, Dev's customer is the IT guys, IT's customer is the Business, and Business answers to real, paying customers. Such a chain of responsibility does focus people's attention on what they are trying to achieve for the company.

Ha! I worked at a Big American Bank where the Business Units had a a dedicated development department and the IT guys were obstructionists who had no customers, so I worked for the production guys and the business unit. So we end up with shit like - the system (modifiable $$$$ software, think SAP but for banking) doesn't run fast enough but the production dept thinks that the software should be written differently because the manager wrote software on a system with 4K of core memory, so they won't give us bigger machines. The business unit thinks we are idiots because we can't get the software their revenue depends on running decently. And the production guys _claim_ ownership, but aren't accountable since they don't have an "if A then B" 400 page troubleshooting manual covering every future unanticipated problem (mostly unusual data problems - stock/financial "products" are ever changing). So I get to do my job, their job and they get to do nothing except bitch about how our system "should" do this and that.

My feeling is that any dev team should support what they write, because the production (IT) department can always cry lack of documentation - no need for the dev guys to do their job and have a deep understanding of the system and write docs and then still end up doing production support.

Re:Middle ground (1)

Tim12s (209786) | more than 7 years ago | (#17222424)

Why is installing an internally developed component any different from deploying and managing Microsoft Exchange or Apache?

IT deploy, Dev must develop and document.

Small companies may try to save costs by skipping documentation.

What about QA? (3, Insightful)

trcooper (18794) | more than 7 years ago | (#17222580)

In my company QA is the bridge between development and production. I'm a team lead (dev) in a company which has a suite of web applications. Each application has a lead assigned to it, who handles the development and documentation of a product through their team. We do several deployments of software each week, and if our leads had to hand-hold through each of them we'd be hamstrung for time and working more night hours than we'd like.

When we have a RC I'll branch the trunk, and request that QA perform a Pre-Production build. Developers will work with ops to get this running properly on the pre hardware, as this can be done outside of maint hours. We'll then do several builds of the branch until it's gold, and then tag off the branch as X.vv.zz.

While a major release is in QA the lead focuses on creating/updating the operations document which addresses day-to-day maintainence issues and tells operations how to troubleshoot the app in the case of a problem. They also produce an implementation plan which identifies the groups/persons needed to deploy the application, and the steps needed to be taken, using what they've learned from the initial pre deployment. Once this is done, and QA has promoted the app, a dry-run is performed to try to catch any missing steps. The implementation plan is handed to QA, who coordinates with IT/Ops to resolve any conflicts and schedule the deployment. Ops/DBA's then physically performs the deployment following the steps given in the plan. In a major release situation, you may have a team lead or platform manager coordinating the actual steps on a conference bridge. But for minor releases we've been able to just have our operations teams do the full deployment with verification by QA and the product's customer service group.

We also have a twice weekly meeting where any upcoming production changes are discussed between IT/Ops, QA and Dev. Release documents are put on a calendar, so if an issue comes up on another product we can go to this and see what may have caused it.

Dev and QA also meet weekly to discuss the progression of products through or into QA. Any issues with testing or problems with builds not being stable can be addressed.

It took us a while to get to this point. We had previously been in a situation where dev would handle the build and deployment process, and it was had for many of the leads to let their projects go, but now we can see the benefit, not only for the company, but also in the fact we don't have to be doing releases at 12AM on tuesdays anymore. It takes a lot of work across departments, and definitely is a long road, but something that needs to be done.

Re:Middle ground (0)

Anonymous Coward | more than 7 years ago | (#17222700)

The real question to ask is who wants to support the system ? (carry pager etc)

If its the dev team then get them up to speed on the operation side and backoff.

If its the IT team then get them up to speed on the system side and backoff.

I'm in dev. My app is complicated. I have no wish what so ever on carrying a pager, so I educate IT to the best of our abilities and backoff. Sure I'm still called on if things go badly but for the most parts no hassle from the guys.

Also I always get support involve in testing (esp. system testing) since its a sandbox for them to learn about the system & app.

Re:Middle ground (1)

Mage Powers (607708) | more than 7 years ago | (#17223582)

That creates more complex wordy logic, if you want to get them to work together, remind them that its thier job to work together, and if they don't, they're not doing thier job.

Re:Middle ground (1)

Fastolfe (1470) | more than 7 years ago | (#17223628)

Now, the obstacle you're likely to hit is office politics. People won't want to listen to others and/or won't want to give up their turf or allow others on it.

I work in a large IT organization, and my experience is similarly negative, but exactly the opposite: Production Support wants nothing more than to have zero liability and zero work. Consequently, they think of every possible way they can push responsibility onto the development teams, including giving access to production servers. Stability isn't as important as being able to point the finger at someone else when something fails, so there's really no incentive for them to take ownership if they don't have to.

Granted, not all of Production Support works like this, but enough does to demonstrate that the whole "treat your organizations like mini-companies" thing doesn't always work. Maybe we're too big for a socialist model to be effective and we need to set multiple IT departments in competition with each other.

Dev owns the deployments (1, Insightful)

BadAnalogyGuy (945258) | more than 7 years ago | (#17221062)

IT should only be involved in the maintenance of your company's network. The developers need to own the entire product, and if that means bringing in some database experts who can handle deployments, then that's a must. If you are a consumer of your own product (and a lot of companies do that as a way to be one of their own wins) then IT should handle the network right up to the point where your network starts hitting the product network.

Of course you will need to have separated the networks in order for this to be any use at all. Putting your production servers on the same network as your business servers will be catastrophic if anything were to ever happen. In the worst case you'd lose both and be totally dead in the water.

The problem is not "You've got peanutbutter in my jelly", but rather that you either need to hire devs who can do network stuff themselves or you need to bring a network admin onboard who can work solely on the deployment system. If you can't afford that, then you will need to use the current IT guys you've got, but they need to know that they are to keep the two networks separate.

Re:Dev owns the deployments (2, Informative)

MyLongNickName (822545) | more than 7 years ago | (#17221420)

The developers need to own the entire product,

So what about someone like me in the banking industry? You have any idea how long and hard the auditors will bitch if there isn't a separation of duties? Technically the developers are not to have ANY access to the production environment above that of an end user. (operative word being technically)

Re:Dev owns the deployments (1)

Peeteriz (821290) | more than 7 years ago | (#17221538)

I work in banking as well, and here, the developers are not allowed ANY access to the production environment, period.
    An end user is an officer of the bank that is trained and authorised to transfer real money to/from customer accounts. No IT-developer has the right to even view the customer account balances. Some from IT-production technically can look, but Internal security guys do try to log and audit the records that they look at, and viewing a politician's/musicstar's bank account is considered a fireable offence.

Re:Dev owns the deployments (1)

simm1701 (835424) | more than 7 years ago | (#17222660)

We had a similar setup in a previous department I worked in.

A problem occurred when there was a bit of a mix up of which logs came from where.

You see we had populated our entire test database with well known names, sports people, movies stars and well known film characters. The first two overlapped in places and film characters gave rather useful mappings (these were set up in relations from other fields)

After checking one days "database access logs" an internal auditor almost had the entire dev department on disciplinary charges for looking at famour peoples data we should not be looking at. After stopping himself from laughing our manager stepped in and slowly, patiently and using words of two sylables or less explained the concept of test data to this auditor demonstrating his point by retreiving data from influential customers such as William Shatner, Christopher Lee, Patrick Stewart, Yoda, Darth Vader and Obi Wan Kenobi (yes we are an IT department) - he nearly started frothing at the mouth on the first few accounts then left rather sheepishly after the film character accounts were pulled up. Didn't hear anythign else from him on the matter...

Perhaps auditors/compliance officers should also be supplied with the specs of dev->production deployments

Re:Dev owns the deployments (0)

Anonymous Coward | more than 7 years ago | (#17222914)

You should have just let the auditor bring the matter up for disciplinary action. You might have had to deal with one less idiot for good.

Re:Dev owns the deployments (1)

BadAnalogyGuy (945258) | more than 7 years ago | (#17221578)

The question seemed to be related to a webservice application development team more than an in-house development team, so my answer was to that.

The development team owns the product. That doesn't mean that they don't have their own IT guys who handle deployments onto live servers, just that those IT guys are specifically doing their job as members of the dev team.

In more traditional businesses where developers are mostly developing applications for internal use, you are right. Developers need to hand off their work to a deployment team (in IT) that can integrate the new parts into the existing whole. This usually entails an intermediate server and test team that signs off on the updated system.

Re:Dev owns the deployments (0)

Anonymous Coward | more than 7 years ago | (#17221452)

In most cases, Dev groups are unfamiliar with the bigger picture within in the organization and are focused strictly on their own world and development cycle. Yes, they often know processes which they have modeled in software, but understanding business reasons for the processes is not usually a priority. Plus, Dev often is thinking about the next development cycle of the software and less interested in maintaining the existing system once it's installed.

It's best to have a mixed team responsible for deployment. IT has to keep the system working long after Dev walks away "done." End support responsibility is a burden Dev doesn't want.

It's also possible that it's time to break the traditional Dev/IT paradigm. Many companies do that from time to time, to shuffle personnel and responsibilities, keeping groups somewhat more fluid. There are pros and cons to this, but more pros from the executive perspective.

This dev says, "we do not own it" (1)

AHumbleOpinion (546848) | more than 7 years ago | (#17223892)

The developers need to own the entire product

No, whoever supports it and interacts with the customers should own it. If it is an internal product then that may be IT, if it is a product for outsiders then that may be Tech Support. I've worked in both environments, and while my ego as a developer says I am at the top of the food chain and should be in charge of everything, I realize that things worked pretty well when others owned the deployment and interacted with customers. It was an interesting environment when Tech Support owned the distribution. We would create a build, do our own testing, and then deliver a text file with the CRCs of files. QA would take a wiped system and check out the source and all development tools from version control, build, and verify CRCs matched. An unexpected high number of problems were detected by this simple step. Putting barriers between developments and customers is probably a good idea.

Speaking as a developer (4, Insightful)

mwvdlee (775178) | more than 7 years ago | (#17221080)

IT should own the deployments.
Assuming the dev department does their job well, a deployment would not require any of the dev department's skills.

Re:Speaking as a developer (3, Informative)

arachnoprobe (945081) | more than 7 years ago | (#17221098)

I think thats a good comment. That would also force the Dev Team to write a good documentation and spec.

Re:Speaking as a developer (1)

walt-sjc (145127) | more than 7 years ago | (#17222294)

That would also force the Dev Team to write a good documentation and spec.

This works in theory. In practice, Management / the customer will be asking why the project is taking 3 months longer than planned - IT says Dev hasn't completed the docs. Dev will say the system is done and IT is just whining. Management dictates that the software get installed anyway, and tell Dev to get you the docs soon (don't hold your breath.)

Been there, done that.

It comes down to Money. If company income is on the line, best practices get tossed mighty quickly. About the only time this actually works is in Very large companies where projects can afford to be late, and where development is part of IT.

Re:Speaking as a developer (1)

arachnoprobe (945081) | more than 7 years ago | (#17222714)

But then, thats a Management (tm) problem, if they don't follow their own rules, they didn't understand them in the first place anyway. Fire 'em, get some better ones.

BTW: you should write the specs first anyway....

Re:Speaking as a developer (2, Insightful)

Fastolfe (1470) | more than 7 years ago | (#17223896)

I have to agree with the post you're replying to. I work at a major telecommunications company in a large IT department, and "needs of the business" trump "correct" every time. Projects are always due-date-driven, not quality-driven. In theory, if a deployment team should do deployments, but if they have to rush to meet their due dates, you can bet the developers are just as much on the hook and are going to be the ones up in the middle of the night. Eventually someone asks, "Why don't we just make the development team responsible for this permanently?" Unless you can respond to that question in a way that directly translates to getting the work done faster or cheaper, there's just no point in trying.

There is a HUGE difference between companies that sell software, and companies that produce software for internal use. With both companies, it's the bottom line that matters. But when you sell software, the quality of your software is directly tied to your revenue (monopoly situations notwithstanding). It's in your best interests to do things "correctly" in these situations. But if you're just producing software for internal use, you're not making any money from selling that software. There's no reason to strive for quality, and you focus instead on costs. It is preferable to have defects and poor process, because it is cheaper to deal with defects and poor process than it is to design and implement everything correctly. Everyone hates this except for management.

If you want to have a good software development experience, work for a company that's in the business of producing software.

Re:Speaking as a developer (1)

ben there... (946946) | more than 7 years ago | (#17221168)

That assumes that there aren't any quirks in the IT departments' design of the company network, systems, or policies, that any quirks are already documented, that the dev's system doesn't have any bugs, and that the dev's testing environment is completely equivalent to the production environment.

Short of all that, they're going to need to work together on something.

Re:Speaking as a developer (1)

I Like Pudding (323363) | more than 7 years ago | (#17221818)

That assumes that there aren't any quirks in the IT departments' design of the company network, systems, or policies, that any quirks are already documented, that the dev's system doesn't have any bugs, and that the dev's testing environment is completely equivalent to the production environment.

This is why we have something called a QA department

Re:Speaking as a developer (5, Interesting)

zrq (794138) | more than 7 years ago | (#17221188)

IT should own the deployments.
Assuming the dev department does their job well, a deployment would not require any of the dev department's skills.

Absolutely agree.
If the developers do their job correctly, then a release should include a full set of install and migration instructions for IT to use.

If IT do their job correctly, they should test the install on a separate system before deploying it live.
If the install does not work 100% first time, don't tweak it, send it back.

If the developers complain that IT didn't follow the instructions correctly, then the instructions were wrong.
Send it back to the developers to write better instructions.

If all goes pear shaped on the live system, IT should have a full set of (tested) instructions on how to rebuild the system from scratch.
If the developers can't supply those instructions, then you don't know what you have.

Ok, I know this is nice in theory and difficult to acheive in practice, but both teams should be working towards this as their goal.

Mod Parent Up (2, Interesting)

A. Lynch (17937) | more than 7 years ago | (#17221292)

This scenario has always avoided the "pissing match" that inevitably occurs between Dev and IT, in my opinion. Clearly defined roles.

In a previous shop, we treated our in-house developer code releases like any other vendor release. Just because some code was written by the guy down the hall, shouldn't mean that I can't ask for the same level of documentation/support I get from another Tier 1 vendor.

Re:Speaking as a developer (3, Insightful)

SomeoneGotMyNick (200685) | more than 7 years ago | (#17221356)

If the developers complain that IT didn't follow the instructions correctly, then the instructions were wrong.
Send it back to the developers to write better instructions.

That's not always true, Mr. Nick Burns [wikipedia.org]. Sometimes IT has a permanent bug up their network port and refuses to learn a small amount of Developer's vernacular to share in the process. Likewise, Developers should not have to know how to speak 100% "IT" to write instructions. There is a common ground. IT personnel are paid for their experience, and ability to adapt, and not to simply follow instructions.

Re:Speaking as a developer (1)

zrq (794138) | more than 7 years ago | (#17221550)

There is a common ground. IT personnel are paid for their experience, and ability to adapt, and not to simply follow instructions.

Yep, apologies if my post sounded too confrontational.

You are right, not everyone in development needs to understand "IT", and IT should have a fairly good idea of what is actually going on underneath the hood.

However, someone in development team should have tested the release before they passed it on.

  • In which case a good starting point would be to write down the steps they used to install their test system
  • IT test the instructions by installing it on their own test system, and ask for clarification if it isn't clear
  • Development should not be allowed to login and 'tweak' the system, they have to reply in text
  • Repeat until IT have a fully working test install, and a repeatable set of instructions showing how they got there
  • Then install the release on the live system and archive the instructions

As part of the process your IT and development teams have to learn to talk to each other, and you end up with a documented set of instructions for installing a replacement if all goes pear shaped on the live system.

Re:Speaking as a developer (2, Insightful)

walt-sjc (145127) | more than 7 years ago | (#17222524)

Good IT people are ALSO programmers. Check out the SAGE job descriptions... Even for Junior System Administrators, one of the "desired" skills is "Programming experience in any applicable language." Beyond Junior level, it's a "required" skill. I wouldn't put a junior person on a major deployment project other than at a mentoring level (which should be done - how else are they going to get beyond "junior"?.) I think it is a travesty that some educational institutions are pumping out degreed IT people that can't write one line of code.

IT should be able to work around most deployment issues, and ensure that any minor fixes needed to the code / process are communicated back upstream. After all, this is the "real world" where deadlines are real, and money is at stake. A top notch IT team is critical to the success of a huge portion of the modern business world. There is very little room for incompetence at senior levels.

Re:Speaking as a developer (1)

eharvill (991859) | more than 7 years ago | (#17221618)

If IT do their job correctly, they should test the install on a separate system before deploying it live. If the install does not work 100% first time, don't tweak it, send it back.
I would tend to agree with this as well. The problem when developers deploy code or applications into production is they (in my experience) WILL try to fix bugs/errors/etc in PRODUCTION if things go tits up during the install. They don't like to back out the change and do more testing in their DEV environment. Even if they do get the code working, you really aren't sure what you get stuck with in production. And then it's the IT support guys who get the 3AM phone call when something is wrong and also the blame for whatever downtime might have occurred. I have, and will, always vehemently object to any developer having access to a production environment that I have to support.

Speaking as an IT coordinator (2, Interesting)

negyvenot (582011) | more than 7 years ago | (#17221318)

I could not agree more. We have a fairly large projects where the devs do the deployments and I can tell you its all a big mess. Since the devs have the right to do deployments, naturally they can make small changes to the production environment invisible to the operation team. Since there are quite some incidents occuring on the production environment, the dev team tends to fix the problems on the production environment on the spot because "oh, its just a matter of fixing this and that", therefore the acceptance environment is not used for testing as it should be.

Although I know it would put considerabe workload on operation team to take over deployments, I'm sure there would be less incidents and more stability on the production system. sure the deployment process would slow down updates, but would give a chance for a more transparent and controllable process with clear ownership of responsibility, etc.

Now that the project is for more than half a year in production, I see no real chance to change the deployment ownership, but still I wish it could be done.

Re:Speaking as an IT coordinator (2, Informative)

cerberusss (660701) | more than 7 years ago | (#17222060)

Since there are quite some incidents occuring on the production environment,
Well, it obviously doesn't matter enough for your customers to have those projects 24/7 online, otherwise things would be different. So why bother?

In a previous job, we had three groups: development, system administration and application management & testing. Development would put the deployment files in a share and then sysadmin would take over, deploying it to acceptance and production if fiated by application management. If something went wrong, development would create new deployment files in a share. Sometimes when deployment was quick 'n' dirty, a developer would walk through the deployment with a sysadmin.

However, it took a very strong head of sysadmin. His manager would try to order quick 'n' dirty things, but the head of sysadmin would take it up higher to the boss. I heard once that when he got refused by the boss, he walked straight out of the door saying he'd quit. Boss ran straight after him :-)

Re:Speaking as a developer (1)

Scarblac (122480) | more than 7 years ago | (#17221546)

Yes, theoretically. However, that would also imply that IT makes all the specific requirements of their setup known to Dev.

At least, it sounds like the deployment environment is rather tricky. If it's sufficiently tricky to have implications for development, then that should lead to explicit, testable requirements.

Re:Speaking as a developer (1)

Peeteriz (821290) | more than 7 years ago | (#17221614)

Tasks to deploy to the live environment should 100% match the tasks to deploy to the acceptance test (or staging, or whatever) environment.
  If you don't have a proper test environment, then the problem is in this fact, not in dividing Dev/IT responsibilities.

Re:Speaking as a developer (1)

nsupathy (515587) | more than 7 years ago | (#17221638)

Yeah agreed. But most of the time the situation is not ideal. Every project as usual is way out of its deadline and rushed and hence these documents are comfortably forgotten. You ask development to write the release document about the production environment which they dont understand/know very well and that create problems. What should happen is a session involving both parties going through a high level implementation plan followed by series of meetings before the release. Communication is the key. (I am from IT)

Re:Speaking as a developer (1, Insightful)

Anonymous Coward | more than 7 years ago | (#17222178)

Assuming the dev department does their job well, a deployment would not require any of the dev department's skills.
As an IT guy who has done deployments of internally developed programs, that's a big IF. I had several tough times of such situations but one stands out...had to deal with an dev dept that wouldn't touch anything they deemed "done", it was an IT issue and they would just tell the managers that it was already done anything going wrong now must be something IT did. So, we collected everything wrong with the program from exe's that when closed still ran in the background hogging 30 - 50 percent of the CPU, images in the program were actually getting served from the desktop PC of the guy who put them in across a WAN to 300 sites, coding in IP's instead of names when network resources were under change and some setup as dynamic, a list I could go on about. In the end the trouble tickets from the helpdesk ended up going to the Dev team and stopped coming to IT until they could show that it wasn't there problem.
Would be nice if there was a fine seperation, but the proper checks were never done in the first place. My piece of mind is back now though as the current companies dev team accepts input from IT and often promptly works on a fix for issues and actually seem to enjoy the feedback.

Re:Speaking as a developer (1)

Rycross (836649) | more than 7 years ago | (#17222218)

Yep, thats how we do it at my place of work, at a very large bank. I wrote some scripts that create an installation package, which we developers run whenever we want to do a deploy. We then hand the packages off to our deployment team, who installs them on the server. We usually have a developer or two who are around to help debug configuration issues when they're working on a deploy. Also, I keep in pretty close contact with our deployment team to make sure that they are aware of changes in configuration, and that they have everything they need to successfully install the application on their side. I guess you could say one of my job responsibilities is to be a liaison to the deployment team.

But in our case, we also need the deployment team for auditing and tracking purposes. Given that this is a bank, we deal with some pretty sensitive customer information that is subject to a lot of regulations, meaning that we can't just have the developers muck about on production servers.

I'd say the system works pretty well. Give some developers the responsibility to keep in touch with the IT people who will be deploying your application, and make sure that you guys work out any issues before they come to pass. Write scripts and installation documents. Document any configuration changes and make those available to IT. It may not seem like it, but IT is part of the team too.

Speaking as a frontline grunt (1)

BenEnglishAtHome (449670) | more than 7 years ago | (#17222800)

I work for a pretty big place [irs.gov] and while I agree with you, we've found that it's nice to have some IT folks who work closely with dev to usher new products to the field. We have a testing lab where dev hands off their work to a mixed IT/dev team (mostly IT) who then upgrade a sample network with the new product. They test against every expected configuration of hardware and software the field is likely to be using. If there are no problems, there's a joint sign-off, the testing team writes deployment instructions (down to the "click here" level) for field techs, and the software is put on a download server and/or pushed. If there are problems, the devs get pulled in for a quick fix and update to the docs. If they can't pull that off, the Lab then sends the software back to the devs and through channels informs the customers of the delay.

It's my impression that this is different from many other large organizations where pre-deployment testing is strictly a dev thing. Technically, that's true of us but the reality is that the testing is done by frontline folks and others who are rotated into the lab and charged with thinking up new and innovative ways to break everything. I hear it's way too much work and a lot of fun at the same time.

One extra feature - every two weeks the IT and dev folks from the testing lab hold a conference call with the front-line techs. They very frankly tell us what's coming, what they're working on, what fixes they've developed recently, etc. The front-line techs are allowed and encouraged (and they take advantage of the opportunity) to vent, complain, suggest, ask for help, etc., directly from the devs, the testers, the customer reps who spec'd the project, the technical analysts, and each other. All this happens in an open, honest forum where there are no reprisals for plainly saying "This stinks and here's why..." even when the executive in charge of said stinky project is on the call.

For a big place (well over 100,000K employees), this system works pretty well.

Production Operations (0)

Anonymous Coward | more than 7 years ago | (#17221164)

Middle Ground = "Production Operations" group. Familiar with the code and your application as well as the IT infrastructure.

The way we do it... (1)

cs02rm0 (654673) | more than 7 years ago | (#17221174)

...or at least, should, but we're close enough, is to just provide releases at certain points and when we make a release provide one script to update from (at least) the previous version.

That script (which probably calls other scripts in turn) should run backups, install new files, update the database schema, import any new data, the lot. Sure it's not trivial to produce such a script and they can end up taking on a life of their own but it saves the devs from having to talk to the IT monkeys.

Re:The way we do it... (1)

walt-sjc (145127) | more than 7 years ago | (#17222680)

If your IT department is staffed by monkeys, I suggest you get a non-zoo job and work at a REAL company. While I do know an unfortunately large number of incompetent IT people that shouldn't be working with computers or networks at all, none are monkeys.

Who does it with other apps? (1)

onion2k (203094) | more than 7 years ago | (#17221190)

When your team rolls out an application update that was created by an external company, Microsoft for example, do their coders pop in and lend a hand? No. The dev team should write an application effectively as a third party external to the rest of the company. If they're writing things that the IT dept can't roll out without their help then they're not doing a good enough job.

Re:Who does it with other apps? (1)

BadAnalogyGuy (945258) | more than 7 years ago | (#17221662)

But if the developers are working on a product (and it sounds like that is the case, what with "hosted applications" being used in the writeup), then we need to turn your analogy on its head a little.

You wouldn't expect Microsoft to give you raw code for you to build and release into your own environment. You would expect that they develop, test, and release a final shrink-wrapped product to you and your IT guys would take that and install it. If Microsoft doesn't do their job of releasing a binary to you, you wouldn't even consider them as a vendor.

For product developers, the product is the whole package, not just the code tree. If they are working on distributed applications and hosted services, then they need to be right in the guts of the application server because the application server is part of the product.

Production Services (2, Informative)

bihoy (100694) | more than 7 years ago | (#17221196)

This is why many companies start a Production Services team. Generally this means the hiring of a Build and Release Engineer or Manager who has an IT background and an understanding of Software Development.

The alignment of the Production team varies. At some companies they report to the development organization (e.g. to the Manager or Director of Software Engineering) and at other companies they report to Quality Assurance.

I would suggest that you check out the site: http://cmcrossroads.com/ [cmcrossroads.com]

Re:Production Services (2, Insightful)

MoralHazard (447833) | more than 7 years ago | (#17221694)

You are 100% correct. I don't know what this guy's problem is--are IT and development are the only two departments allowed by law in his jusrisdiction? I mean, it's normal for an "Ask Slashdot" question to be totally stupid, but this one is pretty bad. I wish I knew what company it was, so I could avoid them. Someone always needs to lead a deployment project, and to be responsible for both the quality of the application AND the quality of the installation it's running on. This dude doesn't need a whole department, but he does need to have access to the time and knowledge of both the people writing the code and the people running the servers. A company that hasn't figured that out, yet, is courting trouble.

I take issue with other details of the poster's question, too:

1) Why is their production environment so unstable? If the server software (OS, middleware, support apps) is unstable, why the hell are they running it in production? If it's the specifics of their setup that are causing the problems, fire the IT director and get somebody in there who know how to engineer production systems. And then tell them to come up with a plan to fix all of it, give them a budget and time and people, and make sure they fix it.

2) If people aren't "in the loop" regarding changes that effect them, your mananagers need to be sacked. A large part of a manager's job is to keep issues that don't concern you from bothering you, and to make sure that you ARE aware of issues that do concern you. A sysadmin or software developer should have someone above him who sees the big picture, goes to inter-department meetings, and stays in the loop so he can keep his people in the loop.

3) Any idiot could tell you that the developers need a staging environment that replicates the production runtime environment as closely as possible. That means it includes whatever feature may bear on the application operation/health/efficiency, like load balancing and replication. The sysadmins set it up, the developers write-test-debug their code on it, and when they hit a release candidate, the deployment project manager checks out the release, installs it on the staging systems, and runs it through the QA process. If it doesn't work properly, the project manager sends it back to the developers with notes and they fix it. This *should* guarantee that the developers are aware of whatever production-environment issues exist--if not, fire your sysadmin because he lied to you when he said the staging environment was as identical to the production environment as possible.

Having lived through the kinds of startup environments where these issues crop up, I would guess that the current clusterfuck of affairs is jointly the fault of a lot of people. Management doesn't understand these issues--they think they can hire a bunch of sysadmins, a bunch of programmers, and appoint a director of IT and shit will get done. The people lower down fail to impress on management the need for proper processes, and they get lazy and don't want to be as careful and thorough as they should.

Of course, many companies do achieve a buyout without ever fixing their issues--but nobody will ever have a successful growth past that without realizing the right way to run production systems.

Re:Production Services (1)

Rycross (836649) | more than 7 years ago | (#17222246)

Thats my main job description, and I'm mainly aligned with the development team (I report to the production support team). Good advice, especially when you work on large products that involve a lot of people.

Re:Production Services (1)

Fastolfe (1470) | more than 7 years ago | (#17224030)

At a minimum, a company needs a semi-independent Production Support organization. The problem I have with ours is that nobody ever took the initiative to decide, at a high level, who would get what responsibilities. Consequently, Production Support tries to do the minimum amount of work they can while achieving the minimum amount of liability in the event something goes wrong. Ultimately all they do is set up and upgrade servers, and click the button to deploy software to production servers. The development team assumes ownership of all other roles/responsibilities, allowing Production Support to shrug and point the finger when the app breaks after they click "Deploy".

Don't let teams negotiate this stuff at the low level, because they don't know what the hell they're doing initially, and only care about minimizing their work/headaches in the long term. Divide responsibilities meaningfully to your business at a high level and require them to stick with it.

enviroments (1)

Biggest Banana Tree (980484) | more than 7 years ago | (#17221228)

This is why you should have at least 3 enviroments, dev, uat, prod (or similar) the dev's can carryout the promotion between dev and uat and in doing so iron out any bugs with their code / the promotion process. The entire promotion to production should be fully handed over to the production / IT team. At the end of the day you dont want developers anywhere near the production system.

Re:enviroments (1)

Peeteriz (821290) | more than 7 years ago | (#17221508)

In my workplace, the usual doctrine is that the transfer to UAT environment is also done only by the production team - in that way it ensures that 1) the installation instructions are clear and complete (or the tests would fail) and 2) you always transfer to live the exact same thing that was tested (instead of something with a 'tiny bugfix' added)

It Depends (1)

Elvis77 (633162) | more than 7 years ago | (#17221236)

It depends on the the makeup of your organisation:

Do you do ITIL?

Do you have SOX obligations or something similar if outside the USA?

Are you agile?

Can you automate your releases?

What technologies do you use?

My organisation is a large user of IT but we are not an IT company (we're a University). We have a network of about 8000 Computers (Solaris, Linux and Windows) and are heavy users of Oracle products.

We have a focus on workflow automation at all levels in the business and have around 200 IT staff. Our developers prepare releases but a dedicated team does the migrations. We use a workflow system to automate migration requests etc.

We use the releases to the user acceptance test areas as a test of the release package - Is everything there? Does it compile? etc. Some of our releases are fully automated ie PHP applications but our ERP releases are almost completely manual. We are currently running a project to increase the level of release automation.

Our process works because we treat the release package as a deliverable; we test the release mechanism along with the code in the acceptance test environment and we keep an audit trail of what is released and when.

In summary, we programmers and other IT people only want to only release tested and approved code; and to achieve this with good processes. I think a good process is a process that achieves this goal with the least amount of effort and overhead.

Don't Ask Us . . . (1)

Dausha (546002) | more than 7 years ago | (#17221282)

You work for a small company. Any advice presented by /. will fail to solicit the correct response---except this one. This fight is between department heads, and should be resolved between them as businessmen. Failing that, whoever controls both managers should make the decision. This is not a software issue, but a business decision outside your control.

Ommm... (3, Insightful)

MarkusQ (450076) | more than 7 years ago | (#17221306)

In art classes they teach students to draw the space around the objects they are trying to depict. It's a useful skill in many areas.

Rather than imagining that there is this atomic transition point that one side or the other must own, look more closely at what happens when changes are put into production, zooming in until you have enough detail that every piece naturally belongs to one team or the other.

Then look at how this would play out in the real world, to find the "frothy" or "tangled" parts (well, IT should do this, then Dev should do that, then IT should do two more things, then it's Dev's turn again). These parts should be sorted out by requiring documentation (or scripting) to flow one way or the other, so that the process can be performed by one group without the direct involvement of the other.

In short, the problem here is the granularity of your question.


Integration Deployment (1)

problemchild (143094) | more than 7 years ago | (#17221320)

Most projects of a size to really worry about this stuff would have an Integration/Deployment team. Essentially they are the "middle ware" of the Application Dev cycle. Dev don't know about the realities of real implementation and have all sorts of bad habits which don't scale to production and IT/production don't know about the deeper technicalities otherwise they would leave and get jobs in Development!!
Your Integration Team should be able to keep your dev/project "real", get them doing your Dev environments before the project goes into deployment otherwise its a real PITA. Good deployment teams should also produce a finished result with documentation which should be to a standard that the IT/Production can truly use. Best of luck with your project!

John A

As An IT Manager... (1)

CowboyBob500 (580695) | more than 7 years ago | (#17221350)

...you should make a decision and stick to it. It doesn't really matter which team is in charge, so long as one of them is and that the other team knows it.


Re:As An IT Manager... (1)

gbjbaanb (229885) | more than 7 years ago | (#17222302)

you should make a decision and stick to it. It doesn't really matter which team is in charge, so long as one of them is and that the other team knows it. ....said like a true PHB.

From a Dev Guy (2, Informative)

BishonenAngstMagnet (797469) | more than 7 years ago | (#17221410)

I work in my city (population approx. 200,000) government's IT department as a developer. As someone else mentioned, the best way to handle code deployment is over a three-tiered system. At DIT, we have Dev, Stage, and Prod for the Internet, Intranet, and Applications sites. Developers should be given full read/write access to dev, in which they do all of their work. In no way should the world, nor the client, see dev. Upon completion, we promote our work to staging (via sending a promotion request, or doing the promotion yourself, if you happen to have access to staging like myself). At staging, the client (for us it's 99% internal work, so someone else within City Hall) can review the work, request additional changes, etc. After the staging process is done, a request is sent to the guys in ops (IT) to promote to production.

Basically, to answer your question: IT should handle code deployment, but developers should be involved in the process.

Re:From a Dev Guy (1)

Fastolfe (1470) | more than 7 years ago | (#17224136)

If not more. We routinely have 4+ tiers. One for developers to experiment. One or more test tiers for developers to integrate all of their code in preparation for a release. (Sometimes multiple releases could be going on simultaneously.) Separate stage and production tiers (similarly if not identically configured) allow for pre-release testing and the release itself. The stage tier, if configured identically to production, also allows for stress/performance testing. If you have lots of similarly-configured production tiers for other applications, you might be able to share a single stage tier between them, but that then requires coordination and scheduling. A second stage tier may be desirable for a large release, or a middleware upgrade, so that you can manage your current release while preparing for your next release (allowing you to back-out and still have a stage tier to work with to resolve defects in the older release).

Once you get to this scale, though, it becomes advantageous to start automating your provisioning. When you need a new set of servers/tier, submit a request, get an assignment out of a pool, do your thing, and release it when you're done.

Dev is IT (1)

Y Ddraig Goch (596795) | more than 7 years ago | (#17221460)

in our company. Once the developer(s) have a stable release, passes developer unit testing, it goes to QA (Quality Assurance) for systems testing and User Acceptance Testing. Once the user signs off QA asks dev for deployment procedures/scripts. Once Deployed the end user is asked for final sign off on the deployment. Software development is a TEAM effort requiring developers, DBAs, System Admins, QA, and end users. The only one of these that may not be IT is end users.

Re:Dev is IT (1)

tcolvinMI (922327) | more than 7 years ago | (#17222448)

We use a similar setup in our company. Ideally we have three environments, Development, QA and Deployment. Obviously, all development is done in DEV. Once we've completed the project and have done our own internal QA in Development (which we pass all projects to another developer to test), we send the changes to QA, where they are tested again. Once everything has been signed off by QA, we cut a deployment environment, which consists of the signed off QA release, plus any deployment scripting that needs to be done. Any documentation needed for deployment is generated at this time. Once our deployment environment is completed, we send it to our technical staff, who actually installs the release. One thing we've tried to be consistent on is how releases are installed. Major releases are generally the same, so a lot of the documentation from a previous major release carries over. Same with minor releases. Usually our minor releases are structurally the same, so the documentation doesnt change much.

There are still a lot of issues with what we do in our company. From reading other comments, I fully agree that any release should be automated. Installing a release should not be too complicated, where there are 75 installation steps in order to complete the installation. It seems to me you would want an automated installation with logging, that logs any errors in copying files, as well as any errors as a result of scripting. So when you finally complete the deployment of a release and pass it to whomever is installing it, you hand them a CD and say, "Hit next a few times and here's the log file in case there are problems." There are a couple of additional things here to remember. 1)Always take a backup. No matter how failsafe you make a release, there is always the possibility of a problem occurring and should always have a way to revert back to a previous version. 2)It seems to be that any IT dept/Technical Dept is a bridge between every department within a software company and would need someone who has a good background in all things IT, including development. If you dont have this and cant afford this, my suggestion is to stay up to date with what is happening in the application cycle as a whole. In order to support something, you have to understand how it works and what the possibilities of failure are, if there are any. Im not saying that IT people have to code, but they should at least understand the technical processes behind whatever is being released.

In the end, Development is just another part of information technology and whether you have someone who is well rounded, who understands technical as well as development, or someone who works with development to understand the pieces needed to support an application, the goal is the same for every department involved in the application and thats design, develop, deploy and support the application, in all aspects.

We fight with deployment all the time. Since we're a small shop with limited staff, our processes arent perfected and I dont believe they ever will. But we get better with each release. Consistency and teamwork is the key. Without either, deployments are going to be a pain for all parties involved.

I work for a large HMO, and here's how we do it. (1)

hrieke (126185) | more than 7 years ago | (#17221472)

Each week we hold a meeting (call CCB - Change Control Board) where the people who do the development talk about what their projects that need to go in production in the upcoming week.
All software & hardware changes will be discussed; what the original problem was, what the solution is to do, what changes to make, what effects to expect, what the rollback plan is, emgency support, and who's all signed off on the changes & tested them. It's then approved or denied (for lack of planning, support, user sign off, etc.)

Works really well, even if it's bureaucratic.

config management (1)

wiswaud (22478) | more than 7 years ago | (#17221536)

in bigger companies, it's a CM team that handles these things. neither dev nor it can do the deployment. CM will testify that the release has been tested (which can be done by a testing group in dev, or a test department), and they will have a handle on everything that is installed right now and has ever been. They have the final word on whether something should be deployed. They build the packages going to the clients, if need be. And you can always go to them and ask "where's the code that's running right now?".
they'll also keep track of what documentation corresponds to what. you usually can't trust the developpers for things like that.

you need a CM team, even if they're not full-time CM.

IT needs to understand the apps (1)

jtharpla (531787) | more than 7 years ago | (#17221590)

In our organization, we have a small group of admins who focus support on just internal business systems, including a number of in-house developed web apps. Our philosophy is we support the whole application, including deployments. When something breaks at 2 am, we're on call and anyone on the team is expected to be able to respond, even if the problem is on the application side of things. The developers are just an escalation point.

This means we are involved in a lot of details of the development of these applications. We maintain the configuration of all systems used to develop the applications short of the developers' individual workstations and actual code on development. So developers work with us to make the changes they need on the development systems, then document and request these changes and any new code get applied to QA. Finally they submit a request to production using the same set of documented changes--this request is reviewed by their project manager, then approved. In each case, we have the ability to review the changes and sometimes get involved if the requested changes may cause issues for the servers or open a security hole, etc.

The real key to a system such as this is documentation. When something breaks, we document how we fixed it. When we build a new server, we document the steps we used to load it. These documents we freely share with the developers and encourage their feedback. At the same time, the developers are expected to document their application, any scheduled tasks that must happen, any routine maintenance tasks that must be done, etc. This is on top of the aforementioned documentation of configuration changes and code deployments.

In practice, this system doesn't always work...especially as the number of systems and applications has grown over time and thus the complexity of supporting these, both for the developers and for us admins. But in general it does, and even when things don't quite mesh between the two teams, it's usually much easier to resolve such conflicts because of the overall framework we have in place.

Neither owns the deployment (1)

punker (320575) | more than 7 years ago | (#17221652)

I work for a small company, and I am in the convenient position of being in charge of both ops and dev. I came up in very much of a slash role (admin/developer), and I know both sides very well. In my experience, the real answer is neither operations nor development "owns" the deployment. The company "owns" the deployment. Usually there are several groups that have a part to play in making a deployment successful (dev, QA, training, ops, marketing, etc). The main thing is cooperation. We have a process that lays out the things that need to be done in a deployment along with the group responsible. This lets everyone do what they're best at. It takes a bit of work to get this, but it's much more important to have a successful deployment than to get an ego boost for "winning" the argument.

No question (1)

80's Greg (457939) | more than 7 years ago | (#17221656)

...that IT should own deployments. That being said, this assumes that DEV has made strict guidelines clear instructions, and foolproofed their release. There also needs to be a fully tested system in place for rolling out changes and rolling them back.

I work for a small software company that provides software for doing exactly this, and run into these questions and scenarios at every client I go to. Even when the instructions for deploying a release are 20 pages long and extremely complicated and labor intensive, you still don't see development applying it. My job is to make it a simple hand off process where DEV packages the release (in our software) and hands it off to the deployment team, who in turn executes the job against the production environment. If there are problems deployment can roll the changes back and let DEV debug any issues.

Stuff shouldn't interact (1)

dpbsmith (263124) | more than 7 years ago | (#17221712)

Sooner said than done, of course... but if applications are so interdependent that the combined system is fragile unless everything is just exactly the right version and installed just so and configured just so and located in just exactly the right place in the directory and has all the other configurable settings of everything else in the system just so... then it wasn't well designed.

In the Good Old Days an application was a single file, you copied it to your system, and ran it. Over the years--I tend to blame the loosey-goosey early PC culture for this--applications wouldn't run unless a slew of other things were installed and configured properly, and all of a sudden everything needed to be Just So. And we fell into the black darkness of "installer programs." Which generally had the property that they would work fine on virgin systems, but often failed if any other application had been "installed."

I used to work for a Fortune 500 company where things were seriously out of control. One of the ways you knew it is that people were always sending out memos saying to remember that version 10.8.5 of application X would only work if application Y was 6.2.3b and the firmware was rev 8.1.9c and the OS was rev 7.1b-2.... I worked in a department where the department head thought everything was OK because everything on _her_ PC was hand-installed file by file by her developers... and everything worked. But an ordinary layperson trying to install the company's products in the documented way frequently failed.

What I'm saying is that you may not have an organizational problem, but a technical problem. If things are so complicated that neither IT nor development knows how to make them work, maybe things need to be made less complicated.

Like I said, easy to say, hard to do...

IT...However... (1)

haplo21112 (184264) | more than 7 years ago | (#17221722)

...The IT person(s) who will be hadling deployment should be involved at all stages of Dev. They should attend all development meetings, and be part of all discussions that will affect deployment.

Perhaps I'm missing something (1)

hey! (33014) | more than 7 years ago | (#17221774)

Why would anyone want control of a balky, high profile and critical process? Glory? Future generations of IT managers speaking your name in tones of hushed awe?

Kidding aside, I've worked both ends of that stick. If anybody is goign to "own" a deployment it's going to be IT. But owning doesn't mean you don't share. IT and development both have roles in the deployment that, if you must insist, belong to one or the other. IT should make development aware of any deployment constraints, the development people should make sure it is possible to do the deployment under the conditions set by IT and provide an appropriate procedure, scripts etc. IT should take that procedure and test it, then put it into the context of a roll out plan, then do the plan.

The truth is in a sufficiently complex, sufficiently custom system, there is no clear line between IT and development. If you really care about doing it right, you form a team with members from each department, and you promise to horsewhip anybody who gets snooty about which department they come from. It doesn't wait for hand-off from development, development doesn't wash its hands once a procedure has been written. So make sure there are people from both functions in the game for the long haul.

make a middle ground job (1)

alen (225700) | more than 7 years ago | (#17221780)

we have someone in our company that is a middle ground between the dba's/IT and the dev's. He is a coder by training but he supports the code the developers write, deploys it, checks for problems after QA signs off on it, etc.

Why separate dev from it? (1)

Shivetya (243324) | more than 7 years ago | (#17221812)

the same people who write the code where I work also take care that it deploys properly. Sure someone else pushes the buttons and writes some user information but until its in the field at all locations the developer is actively involved.

code goes from development, q&a, field test, deployment.

once the code reaches q&a it does not change unless q&a rejects, but afterward the developer(s) are responsible if problems arise during install and work with support to fix it. they never lose ownership.

this might not work in well disciplined shops as it would be easy for the support staff to just bury developers in all sorts of stupid issues. This also may not work in pc centric development groups which seem for some reason to have a different idea on development from mini and main framers. (really, we make jokes about the pc centric groups - if something takes forever to get done, even simple stuff, we say "it moves as the speed of X" (x being the name of the pc development group)

just wondering when did IT and development become separate entities? wouldn't that make the job all that more difficult?

Re:Why separate dev from it? (1)

dctoastman (995251) | more than 7 years ago | (#17223408)

just wondering when did IT and development become separate entities? wouldn't that make the job all that more difficult?
When all the people who ran out and got A+, Network+, MCSE, and ITT degrees thought that they "knew computers" and that IT was just networking and system administration.

I'm not saying that A+, Network+, MCSE, or ITT grads are useless and/or dumb. Quite the contrary, I have known many people in that category that had a wealth of knowledge in those specific realms (networking/system administration). But IT isn't just that. IT encompasses development, networking, and system administration. Basically, this is a nomenclature issue. Their IT department is effectively the development department and operations (networking and administration) department. Someone just renamed the operations department to IT to feel more important and "put the code monkeys in their place".

In other words, your problem isn't "Who's job is it?", it is "Who's being a douchebag?". Once you find that individual and neutralize him, everything else will be resolved nicely.

IT for the win (1)

Yonder Way (603108) | more than 7 years ago | (#17221926)

I work for a very large software company and we're going through this battle right now for an internal application.

The developers are fine at making improvements to the application interface and providing new features, but completely and utterly clueless when it comes to system architecture, security, using hardcoded hostnames, etc. etc. The more we close the developers out of the production deployment, the better this runs. We also have to increasingly review their work to make sure they aren't doing things like hardcoding hostnames, or writing data to local disk instead of the rdbms (writing data to the disk kills horizontal scalability, high availability, etc)

The relationship has turned quite sour and there is a big wall between both sides with a liaison sitting on top relaying messages back and forth. It's an entirely unhealthy working relationship but at least the developers aren't sneaking things into the production environment anymore that have no business being there.

neither (0)

Anonymous Coward | more than 7 years ago | (#17222206)

If you are a SaaS company, you should be striving for a system that makes deployments so streamlined that they are able to be done by the system itself.

We use subversion to control our source, and the moment we tag a release for testing MSBuild scripts take the tag out, compile it, unit test it, deploy it to the testers environment, and resolve the bugzilla bug created for the release. When testers are finished testing, this bug gets marked verified and the staging published release gets copied into a stable folder, where at 3am the following day it will be updated on the server and any upgrade scripts that are required get run.

If new systems are to be added, IT builds the system, gives it an address, installs the required programs, installs the 3am update script and performs the initial checkout. The system will then come online at 3am the next morning (when it runs all the scripts required).

If it is an emergency the IT staff (or the developers) can manually run the update script, causing the system to be fully up to date and online immeadiatly. The only issue is that this could bring production down for a few seconds (all production systems running the same network must be running the same version [ok, that is a generalization, this isn't always required, it depends on the changes], so if the update script is fired off, it must be run on every production system at the same time: production will be down for the duration of the update from the first system to the last).

Speaking as a Release Manager... (4, Insightful)

Horza66 (1039328) | more than 7 years ago | (#17222252)

Plenty of other posters have pointed out that you sound a like an operation that is a bit small for the full Software Development Process. However if you're asking I suspect you're a growing company, in which case you need to get a Process in place, and soon or you will experience the full agony of a chaotic IT environment. (NB That's where I work now - I've worked sane places too) Fairly typical Process: 1. Dev receive Requirements and Defects from the Business, and code to them, unit testing their code (). 2. Code is delivered to Operations with a 'Release Note' or equivalent covering how to deploy the code to Environments 3. Operations deliver (deploy) the code to test environment(s). Link and Acceptance testing is performed - does it meet Requirements? are key defects resolved? Plus regression testing - does it break the existing system? Test sign it off if it clears these tests. 4. Operations deploy the code to the Production system on sign off. You inevitably end up with tensions in the Business vs IT, plus the divisions between the priorities of Dev, Test and Operations. Sounds like you are at the stage of not having any well-defined roles/teams for these responsibilities. I'll detail the Operations breakdown too. Operations: As others pointed out this breaks down into various teams. DBAs, Sysadmins, Change Management, Release Management, Operators, depending on your site. Operations are responsible for the stability and smooth running of the Production system - they must accept change, but control it. Since I work there, and you specifically address the subject, I'll detail Release Management too Release/Change Management Usually end up the Gatekeepers on changes. They'll need to be familiar with the whole system, and resistant to the pressure they'll receive from all sides. They need to know what versions of code are where, and be able to reject bad code when it turns up, but be flexible enough to make sure Test have something to test. They need to be experts on everything your IT does. No jobsworths here, and good generalists are rare. Since you'll inevitably go through a period of chaos if you are growing I'll mention that staff turnover here is very high - unless you get in contractors, and pay highly for them. The Change Management role, sometimes covering the Release role too, is to track changes and know who they impact, and be able to prioritise changes. If Release and Change roles are separate, CM is closest to the business. Hope that helps.

we run into this ALL THE TIME (0)

Anonymous Coward | more than 7 years ago | (#17222364)

I work for a company that provides software for deployment/configuration management http://www.certalertsoftware.com/ [certalertsoftware.com]. Whenever we engage with a new company we always have to do the dance for who owns what and where the line is for ownership. What we see more often then not is that IT owns the deployemnt - until it is broken. Then it goes back to Dev.

Some of the other comments that talk about the Agile approaches with rapid iterative builds and deployments are interesting. We have run into a few larger companies (F500) that are using Agile or derivitives. In these cases the dev and IT guys seem to have made peace with each other. I would guess - sometime after Agile was forced on them ;) However, organizations that are EARLY in the Agile adoption or change are seriously challenging to deal with. There is still tons of turf wars and property disputes.

It depends on the deployment (1)

Todd Knarr (15451) | more than 7 years ago | (#17222576)

I think it depends. Where I work we do multiple deployments: into the common development environment, into the QA system, into an internal user-test system, into a customer-test system and finally into production. The rule is that devs do the deployment into the CDE to work out the kinks. Devs and QA do the deployment into the QA so QA knows what's going on. Devs work with the IT staff doing the deployments into the two test systems so that IT knows how it goes and devs can work out any final issues that come up or that IT may have with the procedures. IT does the final deployment into production since they're responsible for those machines.

You can't just throw deployment on IT, it's the devs who wrote the software and know what needs done and who need to see what they missed so they can add procedures to cover it or fix whatever problem in the software caused the deployment glitch. You can't leave it to the devs to do production deployments, it's production fercryinoutloud, devs shouldn't be messing around directly in production except in dire emergencies. It's got to be a combination.

Separation of Duties (0)

Anonymous Coward | more than 7 years ago | (#17222658)

There is no reason for developers to have hands-on involvement in operations. They may be involved in some sort of advisory capacity, but they certainly should not be doing installation and configuration of systems they developed. If there is a problem with the deployment, the devs should be working it out on non-production systems, preferably without live data. If, for some reason, there is an absolute need for debugging on live systems, someone from IT and the security team (you do have a seperate security team, don't you?) throughout the process.

Devs Deploy, Sysadmins Fix The Deployments (0)

Anonymous Coward | more than 7 years ago | (#17222722)

SA's aren't developers, so they shouldn't be taking on any of their responsibilities. It's the dev's responsibility to deploy their code properly and ensure it won't cause any conflicts. If there's a problem as a result, it's the dev's fault not the SA's. You can tell i'm biased, but the end result is devs get to control things and SA's can continue to work on important things until the devs break deployments again and the SA's have to muddle through a lot of bad undocumented procedure to figure out wtf went wrong.

Im a Dev and I do everything (1)

usidoesit (956958) | more than 7 years ago | (#17222866)

It scares me to think of the sheer brainpower that would have to be possessed by the candidate they'd hire to do deployments with no prior dev knowledge of the apps... hey wait, that's me! I'm downstream from all the devs, who document nothing. Speaking as a former dev, to document everything to the extent required for deployment personnel is very costly and the antithesis of agile. I wouldn't want to read your stinking docs. Go ahead, jar everything to oblivion and hand me the keys.

Root owns. (1)

Qbertino (265505) | more than 7 years ago | (#17222958)

Root. Root owns it. Root keeps everything that doesn't come from us running. Root, Chief Senior Root, Senior Root and 2nd Root keep a Binder and a Booklett with all procedures and Accounts written down, in case they all die in a planecrash. Dev builds the stuff deployable and is assitant to Root on deployment and irons out the glitches as Root orders. During Developement it's the other way around. Dev tells Root what it needs, Root delivers. If the System goes down, Root takes blame. Dev owns Root, but Root owns everything. That includes exclusice Root access (hence the name) ... That thing standing on your desk is called a monitor. You use it by looking into the glowing end. That thing with all the buttons is a keyboard. That round thing with the few buttons is called a mouse. Pick up "Idiots Guide to Managing more that 2 Computers in a Work Enviroment" for further details.

BTW: You asking that question here means only one thing: Your companies IT-frastructure needs a complete reedo. Trust me, I've seen it a million times. Stop right now, no need going an further. Fire anybody who doesn't basically understand the above (including yourself, if needed) and have somebody come in and do a setup for you. OSS geeks are OK if they are articulate and competent. Check their clothes - if they are unkempt and less than quality casual don't deal with them. They should have a few years of admin experience up their sleave and be good workers. Which means they should be able to give precise estimates of all factors and need no other than 8 hour days in a classic 5 day workweek to fullfill their estimates. If they can't do that, they are not professionals. ... Are you? .... That'll be 200$ geek-buddy-price for consulting services. Thank you. ... :-)

Now how come I strongly suspect you guys are Windows (Junk)shop?

Re:Root owns. (1)

usidoesit (956958) | more than 7 years ago | (#17223178)

Mostly agree, however for a guy who hasn't actually divorced and handed his life responsibilities downstream to some other shmuck to deploy, 8x40 hour work week is unsustainable. You need the late shift occasionally. Precise estimates? Heh. Get real. We estimate continuously all the time otherwise we couldn't do anything, just not about the things mgmt cares about. The need for estimates is created by the fictional role of "uninvolved bystanders", also known as Management, or Customers. Mgmt's role is to create loops and count beans WRT those loops, but not for me to spoon feed them their own estimates...

This is a policy decision. (1)

DaveV1.0 (203135) | more than 7 years ago | (#17223054)

The CIO or whomever makes the IT policies should be the one to say who "owns" the deployments. That is what management is for, making these kinds of decisions. In a way, the person who should "own" the deployment is the Project Manager.

This decision should be enforced by the Project Managers (PMs) overseeing the projects. If your company doesn't have PMs overseeing the projects, then it may want to start assigning them.

The PMs are the ones who are supposed to be keeping the changes on track, on time, and on budget and that the application does not experience problems. They are the ones that are supposed to be letting people know about the changes. They should also be making sure there is proper documentation for the deployment and the software. They should make sure that IT and Admin can understand the documentation and that what is in the documentation can be implemented. The PMs should be meeting and let each other know the status of the various projects so that information can be passed down to whomever needs.

For the who needs to know problem, go to those who think they need to know and get a list when they think they should be notified about something and make them justify it. And, "Because I want to know" is only valid if that person is one's superior.

By the way, I work as production support in my company and I own the deployments for my application. My job is making sure the application stays up. That is my primary concern. Next comes what the users want and then what the developers want. Unless I approve, or a director demands it, the deployment does not happen.

don't reinvent the wheel (0)

Anonymous Coward | more than 7 years ago | (#17223230)

don't reinvent the wheel, use ITIL or some other tried and proven framework / methodology.

Staff Rotation (1)

swillden (191260) | more than 7 years ago | (#17223320)

Unless your IT department is under-educated, you should have a number of IT guys who are moderately capable software developers themselves, and at least some of your developers should have some system administration skills as well.

Given that, one good solution is obvious: For each development project, put someone from IT on the development team. The development organization is free to assign that person some development tasks, but their *primary* job is to ensure that development is taking IT's concerns into account in the design and implementation of the application, documentation and maintenance and support planning. Obviously, it needs to be a fairly senior IT person who understands how things work and knows who to talk to whenever any IT-related questions come up.

Then, for each deployment, the development organization should provide one of the developers to IT to support the testing, deployment and cutover to production, plus the first few weeks of operations. This developer should be one who understands the code fairly well, and knows who to talk to whenever there's anything he/she doesn't understand.

Later, rinse, repeat. Over the course of a few years, assuming you can retain your staff, you should end up with IT and development organizations who both have a deep understanding of one another's jobs and work together very smoothly.

Security (1)

sublime (38124) | more than 7 years ago | (#17223364)

From a security standpoint, developers should not deploy, install, manage, or administer any production servers. It is a conflict of interests and is covered by separation of duties.

That being said, I've worked in many environments. In my current position we have four teams.

1) Development
2) System Administrators
3) Application Administrators
4) Database Administrators

Development develops the applications, System administrators manage and maintain the hardware/os, app admins manage and maintain the applications, and the dbas manage and maintain the databases.

    Development writes the applications and hands them off to the application administrators. These guys install the app in our QA environment and go through some testing. Once it's tested internally, we install it in UAT and give our customers a chance to test. If there are any problems during testing, the release goes back to development. Once everything has tested clean, we have a change control meeting with the DBAs, App admins, and System admins to discuss all of the moving parts and work together to put everything into production.

I think we have the best of both worlds. Development is out of the loop after they hand off the product, but we also have Application support folks which work closely with development and know the applications and their requirements.

I'm in the same exact situation - On the IT side (1)

HavokDevNull (99801) | more than 7 years ago | (#17223412)

It should be IT deploying to production period (I'm the Sr. System Admin and deploy all apps to production). Yes we know our own systems and we know the application to an extent. If done right "code monkeys" have done their job and the application passed QA'ed and testing, the application then belongs to IT. In my experience the vast majority of DEV's don't understand System Administration (there are exceptions to the rule of course) and no it's not about us loosing control but boils down to accountability. Just yesterday I had to fix and (try) explain to a dev why they could not deploy the code to a QA box because they kept on getting a CRC error in thier package. Now would you trust a production deployment to that person? What would happen when you have to account for service levels for a customer and the dev's kept scratching their heads trying to figure out what went wrong and then call a system admin into the frey.

Project Manager (1)

fuzzybunny (112938) | more than 7 years ago | (#17223502)

The answer is: "neither". This is why you have a project manager (I hope.) Someone neutral who tracks progress and assigns tasks to dev/IT/whatever.

At least I hope you have a project manager. If not, assign one who's capable of working outside his normal role, either dev or IT, but they'd better know what they're doing (cost tracking, that sort of thing.)

There can be no middle ground (1)

kunakida (886654) | more than 7 years ago | (#17223766)

Software going into production systems belongs to IT. End of story.

IT should treat Development just like any other software vendor.
If vendor representatives are there at the production site, it is only to offer help at the behest of the customer
(which in this case is IT). Nothing goes into the systems directly from the vendor without being vetted by the IT guys.

If the product has bugs, it needs to be treated by development at least as seriously as with any other important customer.

What this means therefore, is that IT has to have enough people on staff that can understand and maintain the deployed system.

The usual problems here are that

1) a lot of IT staff don't want to learn a new system (specially one that is in-house and doesn't give transferable skills) -> The answer here is to suck it up and replace them with someone more willing to learn the required systems.

2) development doesn't treat IT as a real customer (it's only IT) and releases shoddy code that needs frequent updating -> The answer here is for IT to do a formal acceptance test and reject the software if it is not ready for prime time (with high visibility to upper management). Ideally, these formal acceptance tests should not be scheduled too often, so that deployments can be better spaced out to force development to actually plan and design what is going to be in each release before they toss it over the wall.

3) IT doesn't see itself as having a stake in the deployed software. Since the software comes from development, and IT had no choice in its selection, any success makes development look good and does nothing to make IT look good. IT treats the situation as forced babysitting. -> the answer here is for IT to treat it like any other deployed system upon which they are dependent for success. Any bugs, or problems in the system should be reported to the "vendor". Anything which IT thinks could be done better should be sent back to the vendor as a requirement (to be assessed by the product manager to be sure).

If the company is too small to have very many people, then you can dual-role some people with the required skills,
but the job roles have got to be kept separate even if in the same person. You need people who can change hats and role-play effectively here so that oversight boundaries are not erased.

In software, as in so many other venues, good fences do make good neighbours.

Gradual transition... (1)

TBone (5692) | more than 7 years ago | (#17223780)

Here at our shop, there is a gradual transition of ownership. I'll replace IT with Operations for us, because we're actually a tech company, and our IT department is the internal computers-on-desks group, where as the Operations team is the group that manages all fo the customer-facing money-making production sites.

Development owns their development environments. We provide as-can-do support for them, during business hours, when nothing else is going on. From there, a build manager (development does not do code builds eventually destined for Production) starts building code, and it deploying it with our pre-production team. In the first QA environment, Development and Pre-Prod work to make sure the release notes are complete, that all environmental and configuration changes are documented, so that the code will deploy to a downversioned environment. Development has significant, but not full, access to this environment. There is at least 1, and in some cases as many as 3, additional QA environments for each customer we have. Dev loses access at each step, until we get to Production, where they have a normal user-type account that isn't in any of the group IDs that the application install runs as, but still has access to view logfiles and such. In Production, Operations owns the install and the application.

There are a few assumptions that lead to "Dev owns Dev, Ops owns Prod". The biggest one is that, for each new release, the Release Notes HAVE to be complete. If we can't bring in a monkey off the street (OK, a technically oriented, slightly trained monkey...) to install the new code from the directions, then it doesn't get installed.

In addition, for the customer teams which run smoothest, we have several meetings each week. Just an hour, but we get people from Dev, QA, Operations, DBAs, Project Management, and Client Services in a room, and just go over the current issues. Sometimes the meeting is fast and over in 30 minutes, sometimes it runs long, but it gives everyone a chance to talk about what's going on and keep in the loop of current and pending releases. It seems that projects that have more problems don't have as much communication between teams have more problems in the transition from Development to QA to Production.

So, the answer you're probably lokoing for is that there needs to be a transition from Dev to Production. Dev needs to document, document, document. Dev can't be involved in the actual release of code. And the more often you can sit down with everyone involved, the more smoothly things will run.

They who are on call should do releases (1)

nick_urbanik (534101) | more than 7 years ago | (#17223884)

Where I work, there are nine of us in the server team, looking after 350 production servers. Only we do any releases. We are all skilled programmers. Only we are on call.

Politics is often a problem, but our team refuses to release important software that hasn't been tested properly, since otherwise we have to support the mess at 2:00 AM.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account