Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Should Developers Have Access To Production?

CmdrTaco posted more than 4 years ago | from the can-of-worms dept.

IT 402

WHiTe VaMPiRe writes "Kyle Brandt recently wrote an editorial exploring the implications of providing developers access to the production servers of a Web site. He explores the risk introduced by providing higher level access as well as potential compromise solutions."

Sorry! There are no comments related to the filter you selected.

For me (5, Insightful)

enderjsv (1128541) | more than 4 years ago | (#33370324)

Whenever an error occurs that I can't replicate in a dev environment, I'm always SO tempted to hop into prod and start adding in some output statements.

Yeah, it's probably a good thing I don't have access to prod.

Re:For me (1)

xaxa (988988) | more than 4 years ago | (#33370426)

Round here, it needs to be the other way round.

My manager likes to stick to the processes, so we (the developers) fill in forms before doing anything (or asking to have it done).

The sysadmins hate processes, so they just change whatever they feel like on the production servers and make some excuse for why it didn't need any discussion. Unfortunately, since we work quite closely with the users, we get the complaints when it all breaks :-(.

Re:For me (0)

Anonymous Coward | more than 4 years ago | (#33370562)

The way I work it here to be in line with compliance is through roles. We are a tiny shop, so I need my developers to help troubleshoot when production breaks. The Application Production Support role people get accounts for prod, but the Developer role people do not. Thing is, it's mostly the same people filling both roles, but allows for a clean break as we grow. Auditors love roles.

Re:For me (2, Interesting)

x2A (858210) | more than 4 years ago | (#33370608)

Eugh yeah I hate that. So what I try to do is code in such a way that if a bug should occur, the whole thing stops working, that way there's no point in my /not/ fixing it on the production server! I'm a freakin genius! No of course I'm joking, but a recent project has hit some problems where I've been able to explain and the client has actually been able to understand the challenges of trying to reproduce an intermittent undiagnosed problem without touching the production code (ie, is just not worth the time trying to do) and lets me fiddle with the code. Usually tho it's enough for me to be able to add logging code where it's needed and there's no end-user-visible effects. There've also been problems that have languished, but as soon as I've had the go-ahead to try resolve it on the live system and resolved it quickly and without interruption, so they're getting more okay with letting me do it that way. Sometimes I'll just fix a problem and not tell them, to avoid all the hassle. At the end of the day, I know better than them (which is why they come to me) and sometimes you do just have to make a judgement call. BUT, it's not a massive project with many developers, and in those conditions obviously you need to retain more order.

rm -rf /^H^H^H^H^H^H^H^Hoops wrong window

Re:For me (4, Insightful)

IICV (652597) | more than 4 years ago | (#33370482)

That's why you shouldn't have access to prod, but you should be able to either A. get a clone of prod made fairly quickly or B. already have one running so you can mutilate it however you want.

Seriously, hardware is cheap and people are expensive. Minimizing person-time is worth a bit of hardware gluttony.

Re:For me (4, Insightful)

FatSean (18753) | more than 4 years ago | (#33370618)

Test environments are essential, but they do require people-time to keep them matching production.

Re:For me (5, Funny)

stillpixel (1575443) | more than 4 years ago | (#33370546)

User: There is an error on page X
I tweak that page code on the production server after looking at the error log. Me back to User: An error really? Have you tried pressing F5?
User: Oh.. hmmm I guess I must have done something wrong. Sorry for bugging you!
Me: Hey, no problem.

Install Good Logging Practices (2, Informative)

FatSean (18753) | more than 4 years ago | (#33370588)

With some fore-thought and some discipline an application can be developed with very robust logging techniques. It takes development time, but there is nothing cooler than asking the production guys to turn the logging detail up for a few packages and seeing tons of data in the logs. It's not perfect as you can't log every variable at every moment but it certainly does help.

I understand some shops can't or won't modify the logging levels on production servers.

Re:For me (1, Insightful)

Anonymous Coward | more than 4 years ago | (#33370622)

I specifically refuse access to PROD so that there is never a question of whether I touched anything. I provide installation scripts and instructions to the DBA/SysAdmin to install my changes in an orderly, documented process.

Re:For me (1)

gorzek (647352) | more than 4 years ago | (#33370704)

I make changes directly in production on a pharmacy benefit management system. If I screw up, people can't get their pills.

It's fun to live dangerously. :-p

Re:For me (3, Insightful)

idontgno (624372) | more than 4 years ago | (#33370796)

If I screw up, people can't get the correct pills.
It's fun to make other people live dangerously. :-p

FTFY. Well, for certain values of "pharmacy benefit management system". If your production hacking can botch scrip fulfillment, please say what company you're working for so I can try to avoid it like the plague it is.

Re:For me (1)

gorzek (647352) | more than 4 years ago | (#33370982)

Production hacking very rarely breaks anything, fortunately. The system has been developed for 20+ years so it has a lot of fault tolerance built into it by now. The worst possible case is a break in script fulfillment, which might happen for a couple minutes out of an entire year.

I'm sure it's going to give me gray hairs, though.

Re:For me (1)

rwven (663186) | more than 4 years ago | (#33370836)

While it's not always a good idea to allow devs unrestricted access to prod, sometimes it IS needed in order to solve problems.

The fact is that most Infrastructure teams that generally manage the production environment simply don't have the technical chops to debug and ferret out issues that are only appearing in production.

If the infrastructure teams were "team players" it'd be one thing, but they tend to be overbearing, territorial types that insist on debugging by proxy which takes 100x as long and often flat out fails.

Without some semblance of prod access, at least for emergencies, it's just a world of hurt waiting to happen.

Should my dick have access to CmdrTaco's ass? (-1, Troll)

Anonymous Coward | more than 4 years ago | (#33370328)


One thing I do know (0)

Anonymous Coward | more than 4 years ago | (#33370336)

Is that marketing definitely should not have access to production. They ... they wiped out the site a couple months ago, while trying to update an image.

Short answer (5, Insightful)

Issarlk (1429361) | more than 4 years ago | (#33370346)

LOL! No.

Yes. DBA's are no longer needed... (1)

bodland (522967) | more than 4 years ago | (#33370350)

All we do is run scripts and get in the way of developers trying to "git'r done!".

Re:Yes. DBA's are no longer needed... (1)

swanzilla (1458281) | more than 4 years ago | (#33370844)

All we do is run scripts and get in the way of developers trying to "git'r done!".

Well, that, and apparently confuse your roles with those of systems administrators.

What a silly question. (3, Insightful)

Score Whore (32328) | more than 4 years ago | (#33370360)

No. It just encourages sloppy development practices.

Would you want to drive over a bridge that wasn't actually designed and engineered, but rather they just piled some stuff up and will fix it if it collapses? Or have a surgeon chopping you open with the idea that they'll figure it out as they go? So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?

Re:What a silly question. (0)

Anonymous Coward | more than 4 years ago | (#33370498)

> So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?

I interpreted the question to be whether developers should have access to log files and data(-base). I would hardly think about serious developer would start modifying anything on the fly in a production environment. Sometimes it's just really hard/impossible to figure out what a problem is if you just go by the description of the customer and or helpdesk. We have this power struggle all the time in our company. If you have to ask for the data/logs, it only helps if you actually know exactly what you are looking for. And usually it takes way too long and way too many emails to finally get the correct set of data from the sysadmins.

Re:What a silly question. (5, Insightful)

jameson71 (540713) | more than 4 years ago | (#33370560)

On the other hand, I wouldn't want the surgeon to have to give instructions to a trained monkey on how to do the the surgery because the surgeon does not have access to the production patient.

Re:What a silly question. (3, Insightful)

jellomizer (103300) | more than 4 years ago | (#33370774)

Bad analogy.

It is more like having a Cardiologist diagnosing the problem then telling the heart surgeon what to fix.
We don't give our fixes to a trained monkey we give them to System Administers or implementation specialists.

I would say that I am a good developer. However I am only an OK System Administrator and I often when my code is done and working well. Deploying the code offers a new set of issues to work threw.

However If I give to people who are good at taking my code and implementing it, the process runs much smoother without much problems.

Re:What a silly question. (2, Insightful)

Desler (1608317) | more than 4 years ago | (#33370852)

We don't give our fixes to a trained monkey we give them to System Administers

There's a difference?

Re:What a silly question. (2, Insightful)

rtfa-troll (1340807) | more than 4 years ago | (#33370902)

Ahh. Well then I don't advise you to visit any medical colleges or hospitals. 'cos, whilst the doctors that will be treating you aren't going to be exactly trained monkeys, they definitely won't be the ones that developed the procedure. In fact, most of the time you will find that they are pretty much following the documentation.

Re:What a silly question. (1)

atisss (1661313) | more than 4 years ago | (#33370666)

That wouldn't be correct comparison. Good developer would correspond to experienced surgeon who have just done surgery on you and is coming by often to see if you have some complications. In case of complications he always keeps scalpel available and fixes whatever is necessary.

In my project I'm the only dev with access to production, and having it really helps resolving some issues instantaneously, as it would otherwise take significant amounts of time to replicate issue on development machines. Having an experienced person available to fix problems when they occur saves my employer a lot of money, as each minute of downtime is pretty expensive. Essentially the dev must have good knowledge of System Administration.

Re:What a silly question. (5, Insightful)

dkleinsc (563838) | more than 4 years ago | (#33370706)

So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?

Because if there's a problem, there will be an expectation that they need to intervene to resolve their failures.

To play Devil's Advocate here, there are some semi-legit reasons why developers might get production access:

  • If there's a serious production failure, developers are often called upon to assist the admins, because while they aren't admin experts they generally have some administration skills.
  • If there's a bug that makes it to production, the time it would take to fix the bug using proper procedures may cost more than doing a quick-and-dirty fix now and cleaning up using proper procedures later.
  • Diagnosing production-only bugs, which frequently require read-only access. For instance, developers may need read-only access to determine that their software didn't deploy correctly.
  • Helping admins properly configure their software.

Now, none of this should be done willy-nilly. The basic rule at my workplace is that a developer can do nothing that could potentially alter behavior without managerial approval and admin approval where appropriate. At the same time, the primary enforcement of that rule is trusting our devs, so very little of that is actually enforced technologically.

Re:What a silly question. (4, Insightful)

SatanicPuppy (611928) | more than 4 years ago | (#33370910)

I work for a big company where every significant business unit has it's own devs and own processes. And while I'm primarily an admin, I still deploy some dev-level stuff a few times a year.

And nothing, nothing annoys me more than going to a shop where the admin doesn't understand the tech, doesn't read the project requirements, and will not, under any circumstances, let me see his production hardware before he tries to deploy.

Because inevitably it fails and then he tries to throw me under a bus, and I have to get on a conference call and try to use my magic mind powers to debug a system I'm still not allowed to see.

Back when I was primarily a java dev, I used to have to configure Tomcat all the time, and it was the sort of thing were you needed to have some experience. And time after time I'd end up dealing with some windows admin who knew absolutely nothing, but was utterly convinced that I could never know more than him about a server technology.

In short, common sense and a reasonable respect for experience beats a hard and fast rule any day.

Re:What a silly question. (0)

Anonymous Coward | more than 4 years ago | (#33370746)

It might be excusable in a perfect storm where you deploy something to production, it fails in prod (but of course didn't on dev), and you can't rollback due to trollkin (again, perfect storm.)

Also, people are less likely to die than in the bridge and surgeon scenarios.

You can be as clean and tight as a nun's cunt with your development practices, but IMO provisions should still be made for debugging extreme unforeseen situations. (e.g. Flags to produce call traces and debug info, JTAG or breakout boards for embedded devices, a debug or scripting console for client standalone applications, and so on. Better to not need the tool and have it rather than need the tool and not have it.)

Re:What a silly question. (1)

cpscotti (1032676) | more than 4 years ago | (#33370880)

Hey! Don't use surgeons for that reasoning!
Whoever thinks Medicine is "Science" or that Surgeons know exactly what they'll do when they have your belly wide open: a) lives in a beautiful world where nobody dies, b) never seen a surgery in real life, c) knows nothing about the huge difference between math and medicine.

Re:What a silly question. (2, Insightful)

mini me (132455) | more than 4 years ago | (#33370896)

I do prefer to drive over a bridge that is well designed, but if that bridge is found to have design flaws, I prefer to have them fixed right away. I do not want to wait for the crew to build a second identical bridge next door in order to test their patches.

Backdoors (0)

Anonymous Coward | more than 4 years ago | (#33370370)

If you deny developers access to production servers, then don't be surprised if your web sites end up with some backdoors. Not saying it is right or wrong, but you're pissing in the wind if you think you can cut off the developers.

Re:Backdoors (1)

Score Whore (32328) | more than 4 years ago | (#33370410)

One would hope that developers have a greater level of professionalism than that. However if it turns out that they aren't mature and professional then they shouldn't be surprised when they end up losing their jobs and facing criminal charges.

Re:Backdoors (2, Insightful)

alanebro (1808492) | more than 4 years ago | (#33370448)

I don't think he meant malicious backdoors. I read that as backdoors to allow debugging/etc.

Re:Backdoors (1)

roblarky (1103715) | more than 4 years ago | (#33370476)

I completely agree with this, but I also have an undying commitment to my customers and when it's my name on the application and as the primary POC, you better believe I'm going to do everything I can to support my end users in any way I can.

Yes, I could've lost my job or got in the end, I would never condone it or defend it as reasonable, but I just couldn't stand to be shackled and ultimately cost the business more money by having an entire department sitting around with their thumbs you know where because of a stupid process.

Re:Backdoors (1)

roblarky (1103715) | more than 4 years ago | (#33370434)

Yep. I did this for a few of my applications. When the business would call and had a major problem that needed to be resolved immediately, I would use it instead of the 1+ hour of time and 5 additional people involved.

Was it "right"? No, but when the Production support/engineers are worthless, what is one supposed to do?

Re:Backdoors (1)

rtfa-troll (1340807) | more than 4 years ago | (#33370948)

er.. design your application so it gives debugging feedback you can use? Put in assert statements so that you spot when something is wrong? Make the debugging facilities and instructions for using them so that the support people can give you useful bug reports?

Sorry, was it a trick question?

Re:Backdoors (1)

iPhr0stByt3 (1278060) | more than 4 years ago | (#33370500)

That may very well be a true for a small shop, but in a small shop you may not actually have dedicated systems for a complete Dev cycle anyway (Dev, Q/A, Prod) and if you do, it may be acceptable to let the devs have access to production systems (they may even be the only people capable of maintaining them anyway). In a large environment review processes should prohibit backdoors - if not, then you need to fix the review process. But somewhere in-between are those businesses where it can be difficult to balance system integrity and security with development access.

Re:Backdoors (0)

Anonymous Coward | more than 4 years ago | (#33370744)

MS windows is a large dev project, yes? It's rife with backdoors.

Re:Backdoors (1)

hal2814 (725639) | more than 4 years ago | (#33370962)

Mister Potato Head! Mister Potato Head! Back doors are not secrets!

no. (2, Insightful)

pdp1144 (599396) | more than 4 years ago | (#33370374)

It is my experience that giving development access to production gives you a production environment that looks like it has been vandalized. Although meaning well and trying to make the best application as possible; they need their own development lab, and their own staging / production lab.

Uhm... (1)

drolli (522659) | more than 4 years ago | (#33370378)


If needed there should be a mechanism to automate bug reports in a meaningful way, as most professional software has.

Not just no.... (0)

Anonymous Coward | more than 4 years ago | (#33370386)

hell no.

Read-only, if that, and nothing more. (4, Informative)

Junior J. Junior III (192702) | more than 4 years ago | (#33370394)

If you want to have control over your production code, you need to have assurance that it is not changing in an uncontrolled fashion. Allowing developers to have access to production locations makes it all too easy for this to happen. Read-only access allows developers to see the running code and perform file comparisons which can be useful in troubleshooting. They should never need more than this.

And in some cases, even read access can be risky -- I've seen production web sites with resources linking back to development server URIs. It's a good idea to firewall your production servers in such a way that it is not possible for them to reach resources on development servers. This shouldn't prevent developers from being able to read the files on the production server, though.

Re:Read-only, if that, and nothing more. (4, Interesting)

SatanicPuppy (611928) | more than 4 years ago | (#33370602)

See, to me this is more an issue of devs not obeying the rules. They damn well shouldn't be changing production code, and they damn well shouldn't be linking code from other servers.

Either your devs are a bunch of barely trained lunatics, or they're breaking the rules in a vain attempt to get things done in a timely manner.

Most times, when I see devs screwing with production it's either a "hero" coder who is way too good to use best practices, or a situation in which the environment is so hostile that the "best" solution seems to be breaking the rules.

I once did some contract work for a company where the Q&A and testing process took a minimum of two weeks for the most trivial changes, and where the admins on the production servers refused to deploy things like security patches without a testing period that ran close to a month. The devs there had a hundred tricks for sneaking their code into production, and linking production code to the development servers in an attempt to meet their productivity goals.

Fucking nightmare. Once we ironed out the Q&A thing, and split the admins into two groups (one who maintained, and the other who upgraded and approved changes) the whole process evened out and the devs stopped screwing around on production.

Re:Read-only, if that, and nothing more. (1)

GigsVT (208848) | more than 4 years ago | (#33370960)

I once did some contract work for a company where the Q&A and testing process took a minimum of two weeks for the most trivial changes, and where the admins on the production servers refused to deploy things like security patches without a testing period that ran close to a month. The devs there had a hundred tricks for sneaking their code into production, and linking production code to the development servers in an attempt to meet their productivity goals.

Fucking nightmare.

Wow you worked for Linden Lab?

Re:Read-only, if that, and nothing more. (1)

hsmith (818216) | more than 4 years ago | (#33370786)

My favorite is, developer writes the install instructions.

Works great in dev. test. uat. production staging.

Doesn't work in production, because something is configured differently.

Of course, it is the developers fault to not account for it!

Everyone agrees... (5, Insightful)

SatanicPuppy (611928) | more than 4 years ago | (#33370414)

Everyone agrees that developers should never have access to production...Unless they're the developer, in which case it's different.

Its a good practice to keep them separated, but in the end its just a pissing contest. The server admins don't want some filthy dev messing with their stuff, and I can appreciate that.

However, admins often lack appreciation of some dev-specific issues, and their ignorance can lead to problems down the line.

In the end, its the best practice to have everyone work together sensibly, than throw down inflexible rules that cause more trouble than they prevent.

Re:Everyone agrees... (1, Insightful)

Anonymous Coward | more than 4 years ago | (#33370700)

Its a good practice to keep them separated, but in the end its just a pissing contest.

No, it's not a pissing contest. It's a legitimate best practice.

However, admins often lack appreciation of some dev-specific issues, and their ignorance can lead to problems down the line.

I know a lot of sysadmins, and they are all developers of something as well. It has nothing to do with ignorance.

Comments like these act like we're some caricature of a human being.

Re:Everyone agrees... (2, Insightful)

SatanicPuppy (611928) | more than 4 years ago | (#33370750)

I work both sides of the fence myself, so I have a better than average appreciation of the actual risks and the actual benefits.

It's my experience that the Berlin wall-like separation that many companies enforce between dev and production causes it's own set of problems, and that, instead of treating everyone involved as if they are involved, they just impose even stricter separation, which causes other issues, and makes the whole process glacial and inefficient.

Re:Everyone agrees... (2, Insightful)

Aceticon (140883) | more than 4 years ago | (#33370954)

Everyone agrees that developers should never have access to production...Unless they're the developer, in which case it's different.

Developers should not have access to Production and I say this as a developer.

Having a way to check logs in Production, maybe read the databases yes, more than that, no.

Two reasons, one "good" and one bad:
- If people have access to Production willy-nilly, sooner or later they will break it. This can even be true for Support people only they're the ones getting calls at 5 AM to fix it, so they soon learn not to do it.
- Forcing a proper release procedure avoids creating expectations on non-developers (read managers/clients) that minor changes are fast to do and put in prod. Nothing like a Release Procedure in place to instantly kill any expectations of the "you can do this change in 5 minutes" type and create some space for actually doing UAT testing of even those seemingly innocuous changes that sometimes end up poluting a database with garbage beyond the point of recoverability.

I work in finance and I've seen more than once cases of seemingly innocuous fixes done in Prod that brought down the systems for several hours, pretty much paralising the business and costing millions in lost business. This more often happens is seemingly non-mission-critical systems (where Prod access is more likelly to be open) that turn out to be required for know critical systems to keep on working.

No, if you can afford for them not to (2, Insightful)

dmomo (256005) | more than 4 years ago | (#33370418)

So, yes.

Uhm, no. (4, Insightful)

HerculesMO (693085) | more than 4 years ago | (#33370422)

This is why there is a change control process, and a testing environment.

If you're doing it wrong, you're asking for trouble.

No correct answer (3, Insightful)

Motard (1553251) | more than 4 years ago | (#33370430)

There's no correct answer to this question. It depends on the size of the organization and the nature of the system. I've worked in different companies that have been on either sides of where I thought the line should be. The line is drawn in a very different place for a 20 employee company than where it is in a 20,000 employee company.

Re:No correct answer (2, Interesting)

RyuuzakiTetsuya (195424) | more than 4 years ago | (#33370636)

Bingo, and I don't think the line is drawn at the size of the company but how mission critical the application is. Granted, in a 20 person firm versus a 20,000 person firm, the developer's probably also the administrator. But OTOH, if your business is 20 people big and one of them is a developer, it's easy to assume it's probably IT based and as such, some sort of administrative control is probably a good idea in general. Think about it. Well, sure, 20 people but how many machines? Half rack? rack? 5 racks? A whole datacenter?

Re:No correct answer (1)

hedwards (940851) | more than 4 years ago | (#33370826)

More than that it also depends upon how it's dealt with. It's a completely different situation if you've got proper auditing, revision tracking and revision control to if you're allowing the files to just be copied without such measures being in place. Not to mention how the accountability when a mistake is made gets handled and how clear the delegation of authority is.

Jesus no (2, Informative)

BigJClark (1226554) | more than 4 years ago | (#33370432)

The day developers can write code that compiles the first time, then yes, otherwise, jesus, no.

I work as an Oracle DBA for a mid-size company, and I provide a day-old cleaned copy of production in a different environment/box, and it does the trick.

Re:Jesus no (1)

herring0 (1286926) | more than 4 years ago | (#33370658)

We use DataGaurd and part of the process for opening the standby for use sanitizes any sensitive data and resets all the passwords to the development passwords. This has been an excellent resource since devs can see exactly what is on production at any point in time.

Then you just close it up and it reapplies all shipped logs for DR and you're good to go.

Love it.

Re:Jesus no (1, Troll)

unformed (225214) | more than 4 years ago | (#33370728)

Yep because code that compiles is guaranteed to work properly.

Stick to what you know best, the database. Let us do what we do.

Re:Jesus no (1)

rednip (186217) | more than 4 years ago | (#33370732)

The day developers can write code that compiles the first time, then yes...

I use an IDE (Eclipse), and the builds I do always compile, so, like, thanks for the access! What I think you mean is that the day when developers can guarantee the expected results...

Personally, I like the QA process, as it gives me 'cover' if something expected happens.

Re:Jesus no (0)

Anonymous Coward | more than 4 years ago | (#33370912)

Personally, I like the QA process, as it gives me 'cover' if something expected happens.

That makes you a sociopath too.

Re:Jesus no (1)

klui (457783) | more than 4 years ago | (#33370742)

Getting stuff to compile first time is easy. Getting it to work properly the first time, that's something else.

Re:Jesus no (1)

EricWright (16803) | more than 4 years ago | (#33370864)

As a developer, I agree 100% with this. However, my company doesn't provide a day-old cleaned copy of production, so I still need read-access to debug what piece of data is causing catastrophic failure in the customer billing process.

When I have all the tools I need to do my job, then no, I don't need any access to production.
When I don't have all the tools I need to do my job, I need to have the ability to look at production.
I should never be given the ability to touch production.

überdev (3, Funny)

Anonymous Coward | more than 4 years ago | (#33370872)

I am one of the few people who can run correct code the first time round. I am also proficient enough in OS matters to be able to circumvent access to locked down resources. So I don't care what this post says, I'm doing it myyyyy waaaaay.

Hmm (2, Insightful)

mark72005 (1233572) | more than 4 years ago | (#33370436)

I think it's helpful in analyzing real-world data and getting an idea about real system loads, testing issues to see if they are in the wild today, etc. For a good developer, it makes life much easier.

In a very healthy development ecosystem all this data is replicated and there is never any need for a developer to touch prod. In the development ecosystems that exist in the real world though, most are very unhealthy, frustrated by ham-fisted security, process flaws, red-tape, inconsistency, and incompetence ranging from scattered to mostly cloudy.

The answer is, do you have the class of developer that knows what not to do and desires to play nice, or do you have the usual.

Developers need access to production DATA only (2, Insightful)

mikein08 (1722754) | more than 4 years ago | (#33370440)

As a developer I can tell you that it's impossible to test programs properly and thoroughly without access to production data. However, developers should NOT be granted access to production logins/sites - production data should be copied into development work areas so that developers have an appropriate "sandbox" in which to work/test.

Re:Developers need access to production DATA only (0)

Anonymous Coward | more than 4 years ago | (#33370828)

Exactly. We snapshot production's database every 2 hours and post the dumps on a local intranet page that our developers can download and restore to their local system. We utilize Hudson to automatically build packages that have all of their dependencies listed so you can aptitude install the frontend app on any new system and have everything pulled in and configured. Logging is all done to syslog which is fed back into a tool that the developers can track down issues with individual users by looking up their session ID and pulling the history for that specific user interaction.

That's for an app that's getting pounded by 10k users. You obviously have problems as the data set gets bigger or the load goes up, but you can always scale things back as you get larger and have more staff to implement a proper Dev->QA->Staging->Production cycle.

And the concensus is ... NO (2, Interesting)

mr_stinky_britches (926212) | more than 4 years ago | (#33370450)

And the concensus is ... NO

Who let this question through? It doesn't even seem controversial. I am not aware of any good reason to routinely give developers access to production.

The argument made is a sound one. (1)

jd (1658) | more than 4 years ago | (#33370480)

I dislike blog postings on Slashdot as a rule - they can get a Slashbox like everybody else - but the arguments made in the article are well-reasoned if somewhat short on detail. How do developers troubleshoot in a production environment? The article acknowledges that troubleshooting in production is necessary and mentions the installing of software, but installing software alone changes the environment (generally a bit of a no-no for debugging, due to Heisenbugs) and debugger hooks can pose a potential security threat (a big no-no for sysadmins). Further, there was no discussion as to whether developers should be the ones troubleshooting - first rule of testing is that you should never rely on programmers to test their own code. They're way too close to it. Either have testers or have programmers test other programmers' code. It is the only way to ensure that there's proper coverage of sufficient corner-cases.

No (1)

SquirrelCrack (522382) | more than 4 years ago | (#33370484)

As a general case, I say that no, developers should not be given access to production. While giving us access to production might seriously speed up the resolution of an issue, in my experience, it always eventually introduces more problems than it fixes. It also tends to create an environment where testing is devalued because it creates the perception that any issues can be quickly resolved in production. This encourages management to compress timelines and causes the dev team to waste a lot of time fighting fires.

The best environments I've worked in have a fully replicated "break fix" mirror of production that can be used to test and rapidly deploy emergency production fixes outside of a normal test cycle if absolutely necessary.

Biggest issue for us... (2, Insightful)

i.r.id10t (595143) | more than 4 years ago | (#33370490)

Biggest issue my cow-orkers and I have is that the sysadmin *claims* that the dev box and production box have the same packages, configuration, etc. but in reality, they don't. Most often we find out when we ask for production stuff to be copied over to the dev site to test errors, etc. and just loading it - which works on the live site - generates errors on the dev site.

Production (1)

CFBMoo1 (157453) | more than 4 years ago | (#33370528)

I use error handling to give me as complete a picture as possible of stuff on production. I don't want anymore access to production then I absolutely need.


Anonymous Coward | more than 4 years ago | (#33370556)

WTF?!?! Your freaking INSANE to even consider it! Devs should have access to two things, Jack and $hit and Jack skipped town.... I can't even begin to express the problems that one would have even with compliance with allowing the propellerheads access to production.

Yes and no. (1)

RyuuzakiTetsuya (195424) | more than 4 years ago | (#33370580)

Yes, certainly developers should have access to their production machines.

No, they shouldn't be allowed to do anything they want with them.

Troubleshooting application breakdowns are much easier for the developer to do. Thus, the access should be limited to logging data, etc. Unless the admin worked on the application itself, diagnosing those kinds of issues through someone else can be extremely difficult at best.

Depends on the size of your organization (3, Insightful)

jaymz2k4 (790806) | more than 4 years ago | (#33370582)

If you are a small software shop then I can see reasons for allowing your small technical staff to have access to production. It's all well and good saying that only the admin of that server should have access and there's a full rollout procedure in place to be followed only on certain days, certain times; but even when I've seen that sort of structure in place there are times when it's useful for the developers to have access to production. Nothing is perfect and we'd all love to have multitude's of staging servers, replicating the typical load and uses of production but for a hell of a lot of (non critical I'd add) systems that just doesn't happen.

There simply is no one rule fits all. Sometimes I wish we had extremely rigorous rules & regulations in place - I'd probably get to go home a hell of a lot earlier. I'm not suggesting you start chucking exceptions all over your checkout code on live but I think you should asses your own situation (and staff for that matter).

... am I the only one? (2, Interesting)

merlinokos (892352) | more than 4 years ago | (#33370610)

I work in an environment where the devs fix bugs before adding features, so the code is stable almost all the time. I have less than 1 callout a week that's caused by something a dev has done to the code.
We hire the best devs, and work in an environment where fixing bugs is more important than adding features. The result is that our devs get full access to production, and even offer to provide support in order to ensure that they're the ones that are woken up if something they've broken falls over OOH.
I've been at my current company long enough that I'd forgotten there were places where devs and ops didn't trust each other.

Have, Yes. Need, No (1)

xero314 (722674) | more than 4 years ago | (#33370612)

If you are running into any issue that requires the developers to have access to production then you have much bigger problems than access control. Developers should need access to development servers only (which really should just be there local box or a set up identical to the supported configuration if you need to test things like clustering or different platforms). Developers should not even require access to testing environments. If you have valid contracts and adequate testing then the only issues that should get to prod are environmental issue, things that can be handled by administrators.

On the other hand, denying your developers access to anything, be it production servers, IM access, youtube, is just asking for them to circumvent the system. So your developers should never need access to production servers, but I wouldn't waste time trying to lock them out of it, or else they will work around those locks if it turns out that they do need it (because your process failed).

Re:Have, Yes. Need, No (1)

Dog-Cow (21281) | more than 4 years ago | (#33370952)

If you never run into issues that require the developers get some kind of access to production, you've never worked on a system more complicated than "Hello World".

Horrible font (1)

IICV (652597) | more than 4 years ago | (#33370614)

Is that font illegible to anyone else? I had to turn Readability [] on, it was so bad. Who the heck thought it was a good idea?

Re:Horrible font (0)

Anonymous Coward | more than 4 years ago | (#33370850)

The font your system should be used is Helvetica, this is a extremely common font. I suspect your problems are your own, it looks fine to me.

Not unless you want magic fixes (0)

Anonymous Coward | more than 4 years ago | (#33370644)

Sysadmin: It's working now; what did you change?
Dev: nothing.
Sysadmin: (sigh)

Yes.... (1)

tdyer (1399659) | more than 4 years ago | (#33370672)

I am a developer. Our environment team is practically retarded. I have to go on-call during DR tests because it is too complicated to restore an image and double click an icon. God forbid they have to install a App Server, or configure 35+ JBoss instances (default is 1 instance per box) to start, tune for memory usage or performance or both, etc. Just last week they decided to upgrade the os to 2008R2. no need to test anything right? Sure all of the code is 32-bit but that wont have any implications will it? Trust me I would much rather not have access to prod, but as the saying goes "Better us drunk, than them sober"

Read only (1)

seniorcoder (586717) | more than 4 years ago | (#33370678)

Developers should have read-only access to production. In this way, they can investigate what is happening but should on no account have any ability to alter anything.

As a developer: read-only access (2, Interesting)

Todd Knarr (15451) | more than 4 years ago | (#33370688)

Speaking as a developer, I want/need read-only access in production. All too often I need to dig out information while troubleshooting, and most commonly I don't know what all bits I'll need when I start. If it were easy to identify exactly what I'd need to find the problem, I usually already know what the problem is. The hard ones are the ones I can't replicate in development and I only have a starting point, something that won't identify the problem but might help me narrow down where to look next. In those cases the only place I can look is production (since I can't make it happen in a controlled development environment) and I can't give the admins a list of what I'll need (because I need to dig through logs and config files before I'll know what I need to look for next). And if we've gotten to this point, it's probably a priority problem impacting production so it needs to get fixed Right Bloody Now.

OTOH, while I may need to look at production, I don't need and don't want the ability to modify production except by going through the admins. This, of course, also requires admins who can follow basic instructions like "Look at config file FOO. Find the line in section X that starts with Y. It's value should be XYZZY followed by the number 1. Change that 1 to a unique number for that machine/instance. Repeat this for every machine/instance.". But all too often the response is "That's too complicated. Can you just give us config files to install?". And of course when I ask for the current config files, so I can be sure I'm not overwriting any other modifications to them (which may have happened since the admins control them and do modify them), I get "We can't do that, they've got production passwords in them.". Now all I can do is throw up my hands and go "Whatever.".

Re:As a developer: read-only access (1)

tdyer (1399659) | more than 4 years ago | (#33370798)

A-fucking-men. My favourite bug was a OS patch that had been applied to DEV, but not QA or PROD for time zone changes. For reasons only known to management, half our production QA and PROD environments are UNIX and half windows. DEV is all windows. We quickly figured out that nobody had been applying the patches to UNIX since management hadn't hired anybody who knew UNIX. Turns out it was a Windows patch that was missed. fucking sysadmins.

Re:As a developer: read-only access (2, Funny)

EricWright (16803) | more than 4 years ago | (#33370894)

You gotta know how to talk to admins. Tell them they also can be replaced by a very small shell script.

Have it. Love it. (1)

BillCable (1464383) | more than 4 years ago | (#33370714)

I develop mostly internal apps for a very large company, but even for the external ones I'm the one who moves files to production. It's not that way for every department, but for ours it works. Better than waiting half a month to get a type-o fixed.

So, those in charge of production are superhumans? (1)

bahface (979106) | more than 4 years ago | (#33370724)

It would be nice if I worked with support people who knew what they were doing. I don't have access to certain environments but if something goes wrong I'm supposed to fix it somehow. But then again, I work in a robot clone environment where software development is some sort of alien concept. I need a new job.

Absolutely not (1)

waddgodd (34934) | more than 4 years ago | (#33370726)

Under no circumstances are any units in a company to have even contact with each other, much less share work product. This leads to unacceptable things, like collaborations over lunch and generally helping each other out and making a more efficient company. If we have a more efficient company, that may mean we have to lay off even more employees, and this cannot happen in this economy because we'd then pass reporting requirements for layoffs and be subject to higher FICA taxes.

No! No! A Thousand Times No! (2, Insightful)

OldeTimeGeek (725417) | more than 4 years ago | (#33370764)

It's not necessarily a case of the admins versus the developers, its more of practicing good data governance.

Our developers used to have direct access to all of the production databases. This was bad enough, but because of this the organization permitted them to directly "clean up" databases (meaning they wrote to tables directly), we had data that was being changed without the ability to really know who did it. The DBAs hated it and the developers were extremely uncomfortable doing it but it happened anyway. We eventually had a real process audit and the auditors had a field day.

Needless to say we changed. I hope.

Virtualization (1)

Kjella (173770) | more than 4 years ago | (#33370776)

Before I would have said "at least read access", since in my experience the bug reports are usually very inadequate and you need to know exactly what the user was seeing and any settings/configuration made in production. Write access was already rather iffy before, and now with most servers being virtualized the best way would be a fast track to create a new clone of production for the ugly cases. We used virtualization heavily at a client I was at, they originally had two environments, test and production. We did a major upgrade, and at most we had 5 environments:

1) The old production, ready to be resurrected in case of OMG problems
2) The old test, used to verify upgrade results (not old prod as we didn't want people making changes there by accident)
3) The new production, obviously
4) The new QA, where the customer was doing regression testing
5) The new test, where we kept working on the next delivery

Being able to suddenly scale up to five environments - eventually down to two again - was brilliant. The cloned it, changed a few IPs and away it went...

Yes, but audited (1)

GryMor (88799) | more than 4 years ago | (#33370810)

The author/owner of an application should be on the hook for keeping it running and for it's failures. To separate these responsibilities creates perverse incentives and encourages fire and forget development with no thought to future maintenance and troubleshooting. At the same time, to discourage the practice of 'keeping things going by kicking them', access should result in a detailed audit trail, which would be necessary anyways for regulatory compliance.

This doesn't work very well without other arrangements in place, namely, a standardized version control and deployment system, hardware as a service and fleet maintenance systems and a hetero-generous service based architecture.

Of course they should. (1)

Zangief (461457) | more than 4 years ago | (#33370822)

Super simple question: yes, they should have read-only access.

Unless you are concerned about privacy issues, but then you probably solved those for your sysadmins too, so no biggie.

Risk vs Productivity (1)

Maxo-Texas (864189) | more than 4 years ago | (#33370824)

In my experience, programmers with production access cause more visible problems but have substantially higher productivity (6x to 8x).

My company has previously had unbelievably tight controls put in place as a result of SOX which added a 45 day overhead to any change (except emergency changes) regardless of size (which means small projects were no longer approved- only home runs).

Now we are going to SAP. All that is gone for now-- productivity is off the charts. I'm sure they will start locking it down after we get the first production environment settled but it is nice to be productive again.

Restore and replicate (1)

funkysoulbrother (1766214) | more than 4 years ago | (#33370860)

I found it's best to have the admins restore a copy of Prod to test or dev then reproduce it there and fix it. Updating directly to prod or debugging against prod should always be a last resort.

I would hope not.. (1)

indraneil (1011639) | more than 4 years ago | (#33370886)

I work with world class developers and an equally competent team of operations folks. The amount of disconnect between the 2 sets of folks is amazing. The developers black box stuff out of their consideration (e.g. setting up load balancers, with or with out affinity, not littering certificates all over the place, the amount of privileges a service needs etc.). The operations folks ignore other aspects (a cache that's hard to build could be lost after a process recycle, not version controlling their ad-hoc queries/sql jobs etc.)
Even if I take out considerations of giving developers access to customer sensitive data, the mere fact that most developers assume that a complete clean reinstall is as trivial as going back to a previous VM image (wrt time considerations) makes me pause and not provide them access. Add to the fact that developers talk in logical terms (regardless of scale) while operations talks in physical terms (actual machine names, drives etc.) and watching them communicate is like watching 2 blind men describe an elephant to you.
Our team makes it mandatory for developers to request for clean concise information from operations who procure it on their behalf. Yes it is slow, yes, it makes the developers having to batch their queries together but I can't imagine doing it any other way right now.

Ummmm, no. (1)

kperrier (115199) | more than 4 years ago | (#33370892)

Not only no, but hell no.

Sysadmin's take (1)

ihatejobs (1765190) | more than 4 years ago | (#33370950)

I'm a sysadmin who used to work at a web development company. As a one man team I managed ~40 servers, 8 of them being in production web servers hosting 200-300 sites each.

Web developers should not be allowed anywhere NEAR a production server. The last time I let one onto one, I spent the next day and a half fixing what he broke.

On the flip side, sometimes developers will just flat out need access. In this case, at least in my experience, a clone does the job just as well. You just need to have a couple servers sitting around specifically for development use, and then have a way to clone machines to this hardware in short order. In my years of experience I have yet to come across a problem that absolutely needed to be tackled on a production server.

No, but not for the reason you think (1)

spiffmastercow (1001386) | more than 4 years ago | (#33370964)

I worked at a small company where I was the sole developer, and had access to the production system. I was able to make changes and roll them out quickly, and only once or twice did I screw something up (and I was able to fix it right away). The problem is that users started coming to me instead of the sysadmin when they had problems. Then the sysadmin/tech support guy got all butt-hurt about it and declared that he would no longer support anything I wrote. As a result, I ended up having to spend way too much time teaching users how to use their computers (half the time it didn't even have to do with any code I wrote) and didn't have enough time to perform my primary job function.

I'm sure you'll say I should have just refused to provide tech support. But when you work for a small company where half the employees are either family members or personal friends of the person who signs your paycheck, that's not always a possibility.

Security Violation FTW! (1)

pentalive (449155) | more than 4 years ago | (#33370966)

Developers with access to either the test environment or the Production environment is a violation of the doctrine of "Separation of function"

(learned recently in my BS-Information System Security program)

Hope you are never audited! (4, Interesting)

TarPitt (217247) | more than 4 years ago | (#33370974)

I worked as an IT auditor for a very big public accounting firm. Reviewing IT controls was a key part of the financial audit (and more so now with Sarbannes Oxley).

If I found developers had access to production, it was automatically a "no reliance" finding.

This means the financial applications are inherently untrustworthy that the financial auditors would have to review original source documents for validation.

"No reliance" meant the audit became much more expensive as a result.

Also - if the auditors can't rely on the financial reports, should management?

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?