Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Fast is Your Turnaround Time?

ScuttleMonkey posted more than 6 years ago | from the never-fast-enough dept.

Software 418

petrus.burdigala writes "I work for a mid-sized commercial software company (~20 Mloc) and we are frequently challenged by our supervisors to get fixes around the clock. Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment), which I consider not so bad. But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic. So I wanted to get feedback from my peers: are we doing that bad? It takes months for other software vendors to issue zero-day exploit fixes, are our customers being unreasonable?"

cancel ×


Sorry! There are no comments related to the filter you selected.

FIRST TROUT! (-1, Offtopic)

Anonymous Coward | more than 6 years ago | (#21341617)

I am a fish!

Turnaround time (5, Interesting)

fyngyrz (762201) | more than 6 years ago | (#21342095)

We generally get fixes for real bugs out within 24 hours, unless the problem is traceable to the OS, the only factor really out of our immediate control. Even then, we do a quick evaluation to see if we can replace the OS function. Over the years, we've replaced quite a few of them, but rarely within 24 hours.

But we know our code backwards and forwards; I wrote the majority of the current codebase myself, and I can generally get to within a few lines of the problem just by a bug's description... the rest is a matter of minutes and testing. This app is very large - comparable to Photoshop in terms of feature count - but it is also very stable after 15 years of whack-a-bug and a continuous drive to make the internal structure as orderly and regular as possible.

It is my observation that the more programmers you have involved, the slower your turnaround time (for everything from bugs to features) will be. Likewise the larger the entity, the slower it will generally move. Almost every layer of management and corporate compartmenting disease will contribute to slowing down the process.

For the apps that I use that I have had the experience of reporting bugs, it is my general experience that bugs often are never fixed at all. One browser, "Omniweb", truly my favorite in terms of features, has bugs that make it essentially unusable for me. Crashing, slowing, lockups and so on - really serious problems. I've reported them, they never were fixed, in fact the software was never updated. Eventually, I just went back to firefox. Then as Leopard came out, after years of doing nothing, they released a "Leopard version" in which, perhaps, I might find those bugfixes if I looked... but as I say, I have moved on and no longer have any enthusiasm for the product. Slow bug repair (or ignoring them) is synonymous with telling your customers you really don't care what kind of experience they have with your software.

Apple, with all their emphasis on customer experience, does this too. They've had bugs in hand for very long periods where they simply don't address them. If your bug isn't something they think will affect a lot of people, it isn't likely to be fixed. I've not yet purchased Leopard, preferring not to catch early-adopter syndrome bugs myself, but when I do, I would not be the least bit surprised to find you still can't refresh a remote share that's been changed by the remote OS; that the wifi differs hugely in compatibility between PPC and Intel hardware; that mail still hoses the sent mail box based on the return address; that shell fonts are poorly rendered; that shell ANSI compatibility is still broken; that the OS still provides locked-up beachballs at the most inconvenient moments; that the OS still puts the wrong things away on the HD when RAM gets tight, and consequently becomes massively unresponsive... Basically, Apple doesn't have good control of their OS, are unable to respond to bugs in a timely fashion, so much so that they triage out bugs based on report counts, and the common patter is that Apple provides a great customer experience. So while my own experience is that bug fixes are important and can be quick in turnaround, here's Apple showing us that you can make a complete thrash out of the entire bugfix issue and still come out smelling like roses. So is a few weeks too long? Probably not, if you have a good marketing department. :-)


Anonymous Coward | more than 6 years ago | (#21341619)

.| |
stop! goatse time! []

There are no stupid questions... (-1, Troll)

Anonymous Coward | more than 6 years ago | (#21341629)

Only stupid people who ask moronic questions on slashdot. Suck my balls, dude, and get some education before you start working.

Definition (5, Funny)

ShawnCplus (1083617) | more than 6 years ago | (#21341633)

aren't our customers being unreasonable

It may just be me but I think that's why they are called "customers"

Parent is right. (5, Insightful)

iknownuttin (1099999) | more than 6 years ago | (#21341781)

It may just be me but I think that's why they are called "customers"

Sometimes, customers are unreasonable and if they are, they should be treated with respect and the problem explained to them. Yes, they may be incredulous, but if you hold your ground (if they're being unreasonable), treat them with respect, they will come around.

The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.

Re:Parent is right. (0)

Anonymous Coward | more than 6 years ago | (#21341847)

The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.
A blanket insult to an entire group is flamebait.

Re:Parent is right. (-1, Troll)

Anonymous Coward | more than 6 years ago | (#21341983)

The truth hurts, doesn't it? Go run to your hugbox and cry more.

That works both ways. (4, Insightful)

khasim (1285) | more than 6 years ago | (#21342043)

Maybe the customer is being unreasonable.

Maybe the developer is being unreasonable.

It isn't possible to determine which from either person's viewpoint. You will ALWAYS think that you're right and that the other person is unreasonable.

Which is why you need criteria for bug escalation. Generating an incorrect response on 1 type of transaction for 1 specific scenario that may pop up once a year is far less important than a bug that corrupts the entire database.

And if your product is considered "mission critical", I would expect a data corruption bug to be fixed within 24 hours. Even if it is nothing more than rolling back the recent patches and re-issuing the previous version.

Re:That works both ways. (1)

log0n (18224) | more than 6 years ago | (#21342111)

Yeah.. I was going to say.. I interpreted the FP as saying they are the customer, so they have every right to be 'unreasonable'. They bought something (seems to be mission critical) so they should have every right to expect it works for them. That's how the customer/provider relationship works, especially one that provides dedicated and one-off services like this seems to indicate.

Of course, 48 hour turn around may not be possible.. that's another issue.

Scratch the "always". (1)

iknownuttin (1099999) | more than 6 years ago | (#21342189)

You will ALWAYS think that you're right and that the other person is unreasonable.

For the exception of the "ALWAYS", I agree with you. I have never had the attitude that the customer is ALWAYS unreasonable. Honest.

Re:That works both ways. (5, Interesting)

Laglorden (87845) | more than 6 years ago | (#21342237)

The problem is when the customer is being unreasonable, the "support" (or more likely sales) just agrees to everything they say and "sure, we'll fix it" because they don't a) know any better b) they wan't to sell, not take the conflict c) they're stupid and just passes this backward "fix this, NOW!"

Then you're going to have a bigger problem! It's the same thing in any kind of relationship, just bowing and scraping and always saying "it's my fault" is going to cause bigger problems in the future than just saying "nope, we're not gonna fix that. or "sure, well fix it, but not now, you'll get your patch when it's tested properly, in the meantime, do this instead"

Re:Parent is right. (1, Informative)

Anonymous Coward | more than 6 years ago | (#21342137)

How we handled this in one environment was simple. All normal updates followed process to the tee or else.

But a process deviation authorization process also existed. In essence, the VP of engineering and marketing, the customer and legal had to sit down and sign a binding contract to get pre-release. The contract basically said "Without duress or obligation we both agree that we are not responsible for ANY issues or fitness for use and are not obligated in any way to mitigate. You cannot disclose any issues you may have with a pre-release product as it is of alpha-beta test quality and acknowledge this." Participated in a few too. Usually only done when the customer had some competative advantages by accepting the added risk.

Essentially an agreement that if say, not being tested enough it wiped out a 10TB DB tough O. If not signed, they had to wait until Q/A said cut, wrap and ship.

Re: Definition (1)

ShaunC (203807) | more than 6 years ago | (#21342175)

It may just be me
Nah, it's you--;

Sorry, couldn't help it.

Not Too High! (5, Funny)

Hanging By A Thread (906564) | more than 6 years ago | (#21341635)

How much of that 48 hour deadline did you waste reading /.

Get back to work!

Re:Not Too High! (5, Funny)

Fred_A (10934) | more than 6 years ago | (#21341973)

I don't know about you but my turnover time can be pretty fast especially if customers call me in the middle of the night. I'll even put my pillow on my head for free.

None (4, Funny)

EmbeddedJanitor (597831) | more than 6 years ago | (#21342027)

I made them believe it was a hardware problem!

Small Startup Experience... (4, Insightful)

Black-Man (198831) | more than 6 years ago | (#21341653)

You have to serve the client who is paying the bills - and we had a very vocal one (Nik*). We had a running joke about the release d'jour. But it wasn't a joke. We literally would push a new build to them every day which contained minor bug fixes. It was maddening! But no one had the balls to stand up to the 800lb gorilla, so the madness continued. As a side-note, they were acting as a beta tester and anyone in the software business knows what that can mean.

The Real Meaning of Bad (5, Funny)

soloport (312487) | more than 6 years ago | (#21341995)

Our running joke used to be:
Marketing: We need it real bad!
Engineering: How bad do you need it?
Marketing: <puzzled look>
Engineering: Careful what you wish for... OK, Ops. Ship it!

Re:The Real Meaning of Bad (5, Informative)

clsours (1089711) | more than 6 years ago | (#21342093)

If they ask for something within 48 hours and know what that means, then they deserve what they get.
If they ask for something within 48 hours and expect something usable, it is up to you to educate them.

Re:Small Startup Experience... (5, Funny)

Anonymous Coward | more than 6 years ago | (#21342163)

I work in a small company doing support (amongst other things). I've been there longer than most of the developers due to high staff turn over.

A common scenerio for me goes like this:

1 - client reports a problem.
2 - spend 2-3 hours on phone with client identifying what's really going on.
3 - spend an hour or so in SQL Profiler/Delphi debugger to find the root cause.
4 - half an hour documenting what the problem is, causes & how to replicate in order to hand it off to a developer in an off-shore office

... wait a week or so for the client to get really cranky ...

4 - have my supervisor ask me Monday morning if I can look at it because the client is cranky & the developer is sick/has quit/has family crisis/there is a public holiday in that country.
5 - fix the damn thing which only takes a few hours to code & test & delivery after all the hard work was done in step 3
6 - wonder why the hell I am still with this company

Pfft (1)

porkThreeWays (895269) | more than 6 years ago | (#21341655)

3 hours right here. Maybe you just need to rub your hands in some comcast coax for extra speed (k thx lame comcast commercial ftw!)

Hmmm. (4, Funny)

Stanistani (808333) | more than 6 years ago | (#21341665)

What was that exploit again?

1 to 2 weeks (4, Informative)

duplicate-nickname (87112) | more than 6 years ago | (#21341669)

For high priority bug fixes, it usually takes 1 to 2 weeks to get a patch out once we determine that a patch is needed.

It depends on the priority (0)

Anonymous Coward | more than 6 years ago | (#21342195)

Low priority issues just get dropped. Management may order us to start working on a high-priority issue a month after it was raised. For simple problems a fix could be released another month later. Many issues have to be escalated to our supplier, which usually takes a few months. Hey, we just released a bug fix to our customers two whole years after receiving it from our vendor. Security issues such as exploits are deemed low-priority.

how long is a piece of string (5, Insightful)

LiquidCoooled (634315) | more than 6 years ago | (#21341681)

It depends upon the nature of the problem and the competency of the developers.
If you know enough of the code tree you can tell when first reproducing and examining the failure whether it is a one off mistake or a larger procedural fault.

Single instance stupid errors (doh! moments) can be rectified and put through testing fairly quickly, however if your initial examination uncovered a larger problem then obviously the process will take longer (if at all - consider workarounds).

If the original dev/test team has been replaced over time this becomes a more difficult issue and every bug must go through complete verification simply because the extent or ramifications of the code modification will not be known.

In some instances we have had fixes out of the door the same day an issue was noticed, in others months go by before a final fix is put in place.

ditto, but more Re:how long is a piece of string (2, Insightful)

arete (170676) | more than 6 years ago | (#21341937)

I love the parent's subject-line analogy.

I'd add, it depends on product, the complexity of the codebase, the extensibility, modularity, readability, and extensibility of the codebase (eg, if it's highly modular it's easier to test a fix that's limited to the module/plugin)

I'd suggest that weeks sounds too long for an in the wild update without a security patch - or published workaround limiting your exposure. (eg, "use this method to restrict the IPs that can access it to trusted ones.") But that isn't me saying you're developing too slow, it's me saying that if it's going to take you that long you need to either find alternate solutions or create a architecture that allows you to quickly make access-limiting patches and layered security.

Actually, I'd make that more broad - if they want faster response to patches, what they need to do is to invest a lot on a highly modular, pluggable architecture so you can MAKE rapid changes. It's really a question of how much investment they want to make.

We routinely do same day fixes to certain kinds of things... but certainly the complex things take longer. And I think we're pretty unusual in that regard.

four or five WEEKS? (3, Interesting)

tannhaus (152710) | more than 6 years ago | (#21341685)

I can understand a week, but honestly...if you're leaving your customers vulnerable for over a month, they might start looking elsewhere

Exploits should be a high concern for any company

Re:four or five WEEKS? (0)

Anonymous Coward | more than 6 years ago | (#21341729)

if you're leaving your customers vulnerable for over a month, they might start looking elsewhere

Tell that to everyone using Windows. =P

Re:four or five WEEKS? (1)

daeg (828071) | more than 6 years ago | (#21341743)

He didn't say it was a security bug. It could just be a mis-spelled name in an application menu for all we know.

Re:four or five WEEKS? (0)

Anonymous Coward | more than 6 years ago | (#21341793)

If it takes his team five weeks to change a name in an application menu, he's in the wrong career. I recommend marketing.

Re:four or five WEEKS? (4, Funny)

Tozog (599414) | more than 6 years ago | (#21341947)

In that case, 5 weeks is not enough time for a marketing team to decide on a new name.

Depends on the scope (1)

teknopurge (199509) | more than 6 years ago | (#21341693)

of the issue. My team has had to get things out the door in 4-6 hours before. Just make sure you have people that have intricate knowledge of the app so that you can properly scope any changes, which allows you to perform good QA.


Re:Depends on the scope (1)

SnarfQuest (469614) | more than 6 years ago | (#21342003)

of the issue.

part of the sentence. Is SlashDot deleting some

Maybe CowboyNeal (2, Funny)

SnoopJeDi (859765) | more than 6 years ago | (#21342081)

does business with this company.

It is common practice on Slashdot (1)

QRDeNameland (873957) | more than 6 years ago | (#21342193)

to begin a post with a sentence fragment which is a continuation of the subject line.

As a Web Developer (0)

Anonymous Coward | more than 6 years ago | (#21341699)

If a bug in our widely used web application is discovered at 12:00 I am often expected to put a fix on production by the time I leave at 6:00.

That said, I would hardly call our patches bullet-proof, though we usually manage to not screw anything up.

Web based (0)

Anonymous Coward | more than 6 years ago | (#21341701)

My company sells web based software (hosted). I have found, fixed and pushed out to 30 servers in as little as 15 minutes. I love our system.

Re:Web based (1)

AuMatar (183847) | more than 6 years ago | (#21341815)

30 minutes? SO much for QA. Care to give me the company name, so I never hire you?

Re:Web based (5, Insightful)

Gregb05 (754217) | more than 6 years ago | (#21341941)

*15 minutes.
It's bad enough that they directly state they're not really testing patches with a 15 minute turnaround, but the fact that they're making mistakes that can be fixed in 15 minutes speaks loudly as well.

Re:Web based (0)

Anonymous Coward | more than 6 years ago | (#21341957)

Could I have yours? I usually like my employees to be able to read at a 6th grade level.

Re:Web based (2, Interesting)

jgrahn (181062) | more than 6 years ago | (#21342245)

30 minutes? SO much for QA. Care to give me the company name, so I never hire you?

Sometimes (just sometimes) it's obvious what the bug is, and it's obvious that testing is meaningless. Would you want to hire a company which does meaningless things to please you?

Re:Web based (1)

thewils (463314) | more than 6 years ago | (#21342063)

The fact that you can do this in 15 minutes means I'd hire you. The fact that your company is set up to allow this to happen probably means that I wouldn't want to be one of your customers though.

In this day and age, quite often it's the company that can respond quickest who gets the business. You can't afford a three week integration testing period whilst fifty trucks per hour show up at a gate waiting to be processed.

Good on you for responding to your customers.

We don't do "box" software (5, Interesting)

netsavior (627338) | more than 6 years ago | (#21341705)

I work for a bank so we don't do box software, but our patches have to meet FTC standards and Federal bank standards.

It is uncommon, but not unheard of to have an 8 hour fix. In cases of customer data vulnerability, legislation has been made such that if we are aware of a problem, we have an automatic injunction against us continuing to do business unless the problem is resolved. So when we have a security flaw, our bank stops working untill it is fixed. So yeah 48 hours would have people fired for sure.

Compliance/security are the only two things that can spark a release with less than 72 hours notice though.

Re:We don't do "box" software (1)

lb746 (721699) | more than 6 years ago | (#21341881)

This is true of lots of other types of business's as well. I work in E-Commerce, and we do weekly minor fixes, and 24 hour serious fixes. 4 weeks for a patch even for our IDE/platform/development environment would send us elsewhere immediately. I can see 4 weeks for maybe a serious flaw in say a video game, but a month for something people are using and paying you money to use, that's just ridiculous. Just like the parent poster said, in banks they lose business the second a flaw is known to anyone, including themselves. The same could be said for a lot of business's if that flaw was exploitable in such a way.

Lack of information (4, Insightful)

Wellington Grey (942717) | more than 6 years ago | (#21341727)

But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.

Well, we really can't answer that question with knowing how big the problem is. If it's an embarrassing typo on a dialog box, then 48 hours is reasonable. If it's a windows vista security patch, then 48 days would be unrealistic.

-Grey []

Re:Lack of information (1)

hcmtnbiker (925661) | more than 6 years ago | (#21342001)

If it's a windows vista security patch, then 48 days would be unrealistic.

Not to sound like a M$ fanboy or anything, but they're average time to fix is 48 hours. The other 46 days is just how long it takes them to cycle it out. I assume that if you're cooperate or some other important customer it is possible to get the patch faster, but I honestly have no idea.

Can't compare (5, Insightful)

Sloppy (14984) | more than 6 years ago | (#21341735)

It depends on what you're maintaining and how complicated it is. I've gotten fixes out in 2 or 3 minutes. That doesn't mean I'm fast and you're slow, though. "How fast is your turnaround?" is like "how long does it take to write a computer program?" It's hopelessly vague.

Re:Can't compare (4, Insightful)

Shagg (99693) | more than 6 years ago | (#21342151)

Exactly... "How long is a piece of string"?

In addition to all of the other comments about the scope of the problem, number of resources, etc, you also have to take into consideration what you're changing. Obviously there are huge differences between patching the avionics system on an airplane vs a banner ad on a website. I've given estimates anywhere from hours to months before. There's no such thing as "X is the right amount of time for a patch" without a lot more details.

One thing you can always do is try and work with the customer to make them aware of the issues. You can tell them that it's possible to get a patch out to them faster, but you will be skipping a lot of the QA in order to do so (depending on what flexibility you have with the standard company process). The risk of it failing would be theirs. If they're OK with that, then you might be able to expedite things. It all depends on what you're patching and how important it is to them.

Depends... (1)

Foofoobar (318279) | more than 6 years ago | (#21341739)

Depends on alot of things: the size of the shop, the size of the project, the size of the fix, etc. Do you have an MVC framework? Do you have a centralized database architecture?

The better engineered the project is, the faster development will be. The worse engineered and less centralized, the harder it will be. Also, the larger the size of the project, the more time it will take. Also the size of the company adds extra time as well; smaller shops can fudge on steps to get a fix out but larger shops have alot more customers to support and therefore can't fudge on QA.

Management strategy (4, Interesting)

gurps_npc (621217) | more than 6 years ago | (#21341771)

There is a management strategy that basically says aim for the stars.

Yeah, your turn around time seems good and yes, the customer's request is beyond industry norm.

That might mean one of three things:

One: Customer is being foolishly optimistic.

Two: The entire industry is bad about turn around time, and can, if pushed improve it to 48 hours.

Three: Customer needs it really quick and is hoping to get it quicker by asking. They know 48 hours is well beyond the norm, but are hoping you can do it anyway, because the more time it is unpatched the more they are screwed. They know that if you don't ask, you can't get, so they are at least 'asking'.

Me, I think it is a combination of all three. Customer is being a bit optimistic, the industry is bad about turn around time, and also the customer knows it is a bit optimistic but is making the request anyway in hope you will provide amazingly good service.

Unrealistic (4, Insightful)

gweihir (88907) | more than 6 years ago | (#21341785)

With a little simplification, you have four parameters: Difficulty, quality, speed and available resources. Whenever you fix three, the fourth follows (with some unvertainity). It is well known, that there is a limit on how much you can improve the speed with more resources. So there is an upper limit on speed already. The second problem that difficulty is unknown when starting such a task. There is no fix for that.

So if these people fix speed and available resources, and difficulty is fixed by the task, quality is determined by these factors. Period. There is no arguing with hard, real limits. If they do also want to specify the result quality, then they have to leave speed open. Again, there is no way around that limitation. In fact they should be happy if the team manages the required quality at all in reasonable time. Not all teams do.

Maybe thisn will be an argumentation that is inderstandable for people with a business background. Engineers should already know this.

Software engineering is engineering. Engineering tasks in general have minimal time requirements. Look at structural engineering: Nobody would try to design and build a full-custom bridge in a week. Instead it takes up to a decade, depending on difficulty. And you can generally not speed things up by increasing the team size.

My jaded perspective... (4, Informative)

idontgno (624372) | more than 6 years ago | (#21341789)

Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment)

Not unreasonable, depending on the size of your release. (How many modules and how many LOC you're changing, the number of change requests or bug reports in the build).

But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.

I think they're smoking crack.

So I wanted to get feedback from my peers: are we doing that bad?

With your regular release schedule, I don't think so.

are our customers being unreasonable?

Yes. That's what they do. If they want a crash development program to get this "patch" out the door that fast, they seriously risk software which does nothing but crash. Really, if they want it that bad, they run the risk of getting it that bad.

You have to ask yourself and your "support team" (sounds more like marketing to me): "Do we wish to ruin a perfectly good reputation for quality and reliability in one hurry-up bashfest followed by weeks of agonizing on-line debugging?" Really, advocate any kind of work-around and risk mitigation response before being pushed into an overly-hasty release that will linger on your reputation like a dead skunk.

20 Minutes (1)

sconeu (64226) | more than 6 years ago | (#21341791)

Seriously... but only once.

We were on-site at the customer, and we were involved in a shootout to see who would get a major contract.

We were "pre-demoing" to a bigwig, and it turned out we had an incorrect concept of one of the requirements. Hacked out a fix in 20 minutes.

Needless to say, the customer was impressed (as were my bosses :P).

For production level code, though, I'd never do that, but it was a do-or-die sort of thing.

Oh, and we won the shootout (it wasn't even close, according to some of the guys who scored it) and the contract.

Difference between Patch and new version (4, Insightful)

techpawn (969834) | more than 6 years ago | (#21341801)

A patch (IMHO) is a bug fit to existing code. Given the resources we should be able to get a PATCH out in a week. However, if you need a new version of the software to address the issue. Then we're talking longer development/testing/QA times if which case 4-5 weeks would not be unreasonable. Bugs should be fixed as soon as they are spotted. If their is need for a whole rewrite then you may want to talk to your staff

Re:Difference between Patch and new version (1)

lb746 (721699) | more than 6 years ago | (#21341939)

Nicely worded. Too bad my last mod point expired over the long weekend.

Like everything in life ... "it depends" (0)

Anonymous Coward | more than 6 years ago | (#21341805)

If your revision controls system is in a constant state of flux and you break daily builds more often then the US breaks treaties. Then yeah you can expect to have some problems getting a patch out to a customer.
On the other hand. Most simple bug I've dealt with take less then a day. More complicated ones will take 2. Anything taking more then 2 days is usually a new feature and not a bug.

Pick 2 of 3 (3, Insightful)

CambodiaSam (1153015) | more than 6 years ago | (#21341819)

I know I'm going to end up baiting some developers, but I work for a specialized ASP and see a ton of third party software from a perspective few get...

Normally, the smaller the company the more agile. No surprise. They also get patches out faster too. Also no surprise.

When we look at vendors of equal size, the ones who are really quick at sending out patches are in that situation because their software is more buggy, and they have a *lot* of practice. It never fails.

In response to your question, I would suggest that you should look more at the frequency of patches and less at the duration. Sure, it might not be as fast as your support group wants, but if you start reflexivly sending out patches every time someone yells, then your overall product will suffer since you can't possibly do the proper QA to ensure THAT patch you just whipped up doesn't break something else.

That brings me to the age old choice:

Pick 2 of the following:

It depends mostly on (0, Flamebait)

antifoidulus (807088) | more than 6 years ago | (#21341823)

how hot she is and how much I've had to drink.....

Ok, who am I kidding, this is slashdot, it depends on how good the porn is :P

How much time do you spend on TPS reports? (5, Funny)

Joe The Dragon (967727) | more than 6 years ago | (#21341825)

How much time do you spend on TPS reports?

The last time I did one I forgot the cover page and my 7 bosses all bugged me about it.

It depends on the issue... (1)

Bert64 (520050) | more than 6 years ago | (#21341831)

What was the issue?
Some things are trivial and fairly obvious mistakes, that are often easy to fix without breaking anything, in which case it's not too hard to get a patch out quickly. Format string problems for instance, shouldn't be too hard to correct.
Often a little extra validation can correct a problem too, just put in a check for a value being zero before doing a division etc. Things like this are also easy to test, and don't break anything else.

Longer turnarounds arise when the issue is more of a design flaw than a simple typo, or when your patch includes more than just a fix for the reported issue. I think security patches should only ever fix the issue at hand, with any feature updates available separately. I don't want the process of patching my system to introduce new features that could bring with them new bugs.

Extreme Programming (2, Interesting)

obduk (1154583) | more than 6 years ago | (#21341841)

Ok, the name might suck, but the company I work at follows the Extreme Programming practice, a kind of agile programming. I have only worked there a few months, and had never herd of XP before, but am now converted. We work in pairs, which instantly adds a whole testing level. Deployments of code are done once every week, but sometimes more in an emergency. We write code test first, then run a build on our machines, then we upload it to a test environment where automatic tests are run. Finally on passing that, it moves to a stage environment where humans test the code, when they are happy a version number is noted, and that is uploaded to live. This means it can take a day for some code to be written, tested and deployed if required. It also means there is continual development, different departments can work on different versions, and then there is a weekly deployment of the latest stable code. It is a very interesting practice, and seems strange at first, but I would highly recommend it for certain types of companies. The company I work for took a few years to convert, and it was slow at first, but now it is an expert and even helps train other companies. It also builds its business upon being one of the quickest responders for code in our region.

Re:Extreme Programming (1)

KlomDark (6370) | more than 6 years ago | (#21342231)

What if you come to work with a bad case of the farts? That can't make pair programming any fun. Even worse is if it's your partner with the farts.

48 hours is reasonable (1)

evanbd (210358) | more than 6 years ago | (#21341855)

In the case of a zero-day exploit, 48 hours is an entirely reasonable expectation. As far as I can tell, it is far from the norm, however. For less egregious flaws, feature upgrades, or normal bugs, a slower turnaround would be perfectly reasonable (and what I'd expect). But for a zero day exploit... Well, I firmly believe software companies need to be held more accountable for those by their customers. If that's what your customers are doing, then good for them.

Depends on the team and the bug. (5, Interesting)

seebs (15766) | more than 6 years ago | (#21341871)

At BSDi, the initial patch (which did have flaws, but it fixed the problem) for the f00f bug was same-day, I believe; might have been next-day, depending on where you're counting from. (Contrary to popular belief, this didn't violate any NDAs.) Now, that was an emergency patch -- it took a while to come up with a patch that fixed the bug without noticable ill side-effects.

We had a better patch later, but the initial emergency patch was VERY fast.

On the other hand, if the initial bug report is "Sometimes the program hangs, no, I don't know when. Maybe every week or two." -- well, that's gonna be hard. Exploits generally have the advantage that an exploit is by nature at least somewhat reproducible, and the hardest part is often getting a reproducer. I've had it take six hours to develop a usable reproducer, and three minutes to develop a patch.

Release time depends hugely on process and procedure. IMHO, an ideal procedure would have some kind of way to get a Temporary Patch out into the field ASAP when there's an exploit.

How fast is your turnaround? (1)

ChengWah (955139) | more than 6 years ago | (#21341875)

Just remember - the cuntstomer is always right, even when expecting the impossible.

What kind of patch? (1)

orzetto (545509) | more than 6 years ago | (#21341879)

D'oh, it would be helpful to know which kind of patch we are talking about. Is that security? If so, how critical? Is that a theoretical case, or a gaping hole that is a dead giveaway to any script kiddie? Is the problem just annoying or mission-critical? Is that going to be used in a network? As a server service? On which platform? If it is a new feature, does it require re-engineering of the code base, or is it a drop-in feature you can be over with in half an hour?

Most importantly, is your code base well-architectured, or is it a filthy can of worms you cannot possibly modify without breaking something?

In my (tiny-sized) company, if the boss or other users of our in-house software need new features or a bug fixed, I have the habit of listening first to requirements and estimating time-to-delivery later, and I have a hunch most developers do just that. Sometimes it's half an hour, sometimes it takes three months. Last estimate I gave was "anything between two weeks and six months", when the boss gave me some very vague description of the next package we are to develop.

It depends... (5, Insightful)

gillbates (106458) | more than 6 years ago | (#21341883)

48 hours is tad bit tight. However, I've turned things around in a similar amount of time.

But, the old adage is true: you get what you pay for:

  • Granted, 48 hours is tight, but possible if you know the root cause, *and* the customer is willing to forego the usual QA process before delivery. It doesn't mean that you don't do QA, but rather, that you do it later and patch the patch, if necessary. In most corporations, this means that if the customer doesn't complain, QA doesn't get done for these "extra-special" releases.
  • Four to five weeks for 20Mloc is probably about average. As a general rule, a good team will average about 1 week per department the fix has to go through: field team -> engineering (fix) -> department review -> QA review -> ship. However, in some organizations, particularly the smaller ones, defects can get turned around in 1 to 2 weeks, especially if the customer works directly with the engineer/developer, and the developer is authorized to make releases to the customer. Be aware that your customers may have dealt with such organizations in the past.
  • 20Mloc is not really that big, provided that the project and code was well-organized from the beginning and the original developers are still on staff. But, if this is not the case, or you have a large developer base, where no one is actually an expert on the systems, or subsystems, you can add about a 50% overhead to your turnaround time. If the original developers are no longer there, add 100%.
  • If you don't know the root cause of the problem, you can't promise anything, and you need to inform the customer of this - you simply can't make any guarantees because you don't know the scope of the problem. Once the root cause has been identified, a 48 hour deadline is still tight, but is long enough to allow the key developer to build and do some rudimentary testing of the fix. Should the customer choose to accept it at this point (and you *must* make the point that it is their choice), they must be willing to forego the normal QA process, and should sign a statement of understanding to that effect.

When faced with unreasonable deadlines in the past, I usually voice my opinion once, and just do the best I can. Your higher-ups are probably already quite stressed at this point, and adding stress to the situation doesn't do anything for your career or theirs. Rather, if you make the point that you're doing the impossible, you might just have a little bit more bargaining power when it comes time for raises.

But on the flip side of the coin, if management doesn't learn, and you find yourself constantly asked to do the impossible, you might want to consider employment elsewhere...

Re:It depends... (2, Informative)

Weasel Boy (13855) | more than 6 years ago | (#21342073)

I work as a tech support engineer, and if I could mod the parent to +5 (Insightful), I would.

Most of my cases are resolved by explaining to the customer things that are unclear in the documentation, so it's unusual to decide within 48 hours that the customer is reporting a real bug. Once we agree that they are, then I can usually reproduce the behavior in a day. Once reproduced, then we do not consider 2 to 5 days for a fix to be delivered to be out of line.

Questions I can answer same day. Fixing bugs takes time.

Pick a policy, follow it. (1)

WPIDalamar (122110) | more than 6 years ago | (#21341885)

In general, the company I work for goes from end of development to GM in about a month, for boxed CD software that ain't bad. That assumes QA has been testing right through development. But it's going to vary widely based on the industry and size of the product.

I blogged about the opposite issue this morning. []

Essentially it comes down to choosing a smart policy and sticking with it. If your company's policy is "Spend 2 weeks QA on every release." and you don't spend that two weeks, expect that release to suck.

Patching Bugs (0, Offtopic)

ishpeck (160581) | more than 6 years ago | (#21341897)

My company can patch bugs within the same day the product's released. Never gone more than two days between a customer's bug report and a fix.

But we don't let bugs slip by. I know since I'm on the QA team. Never a bug. Never once... :P

Re:Patching Bugs (1)

Phu5ion (838043) | more than 6 years ago | (#21342187)

Wow, not a single bug? Either your developers are total ninjas or you are spending way too much time on /.

Depends (1)

olddotter (638430) | more than 6 years ago | (#21341901)

If you are talking about serious security vulnerabilities with-out a reasonable work around, then the customer has a reasonable expectation that you literally work around the clock to fix the issue.

If its a "normal" bug then I think several weeks or months is more the industry standard.

Offer to make it the customers problem (0)

Anonymous Coward | more than 6 years ago | (#21341903)

> are our customers being unreasonable

Not for security patches no. If the customer is being a pain, push it without QA and tell them they're welcome to take their chances outside the terms of any support contract. Then tell them how long your unit tests and such take to run and give them a realistic time for a supported patch.

Most PITA customers will opt to wait rather than risk downtime or data loss.

Size of the problem determines response time (1)

Shivetya (243324) | more than 6 years ago | (#21341913)

Let alone the importance of the problem.

That is why I am having trouble giving you a fixed number. I have been involved in fixes done in days and ones that take months. Context.

The real questions is

Does your management know how to prioritize tasks?

Then, do they know how who to put to to each task?

Finally, do they know how to accurately judge if the work is done correctly?

The problem I have always encountered is that a needed fix gets priority at first, then pet projects somehow get in there, and the deadline moves. Unless of course those affected have more pull than those assigning priority. I always found it amusing that our priorities are set by the people with most pet projects, wolves guarding the hen house.

What is the weight of water? (2, Insightful)

mcmonkey (96054) | more than 6 years ago | (#21341935)

You really have to supply some more detail to get any useful answer. And what is ~20 Mloc? About 20 million locations?

What kind of software? What classifies an urgent request? Do you make games, and an urgent request means your bug just made front page /.? Do you make internet-facing apps, and an urgent request means your customers just formed a spamming bot-net? Are in the medical/health care field, and an urgent request means folks might die?

I think a better question is, how do you classify bugs? How do you make that trade-off between fixing a bug ASAP and taking the time to make sure the bug fix is done right?

Who is involved in the decision process? Is it just the technical & regulatory folks? Do you pull in business folks to help gage customer impact? Do you pull in sales and support to see if they can push a work around before the final fix is ready?

Those are all better questions than, "How fast do you do this task of unspecified scope."

Re:What is the weight of water? (0)

Anonymous Coward | more than 6 years ago | (#21341959)

I'm guessing that's 20 million lines of code. An odd measurement of company size to be sure.

Nothing is unreasonable (1)

twifosp (532320) | more than 6 years ago | (#21341949)

Nothing is unreasonable when you tell your customers they can pick any 2 items from the following list:

You can have it fast.
You can have it cheap.
You can have it reliable.

If they want it in 48 hours, you should explain what that would cost.

If you want to streamline your process for patch deployment, I highly suggest you apply some six sigma or equivlant business process improvement strategy to it. Asking Slashdot is going to be counter productive. Your application is very different from anyone else's. Industry standard turn around time for software for a toaster is not going to be useful to you.

In other words, either you are working effeciently, or you aren't. It's that simple. If you aren't, fix the problems with your process. If you are, charge the customer what it would cost you to meet their needs.

Hot Fixes (1)

Hairy1 (180056) | more than 6 years ago | (#21341977)

If a customer has encountered a serious issue and is unable to operate, getting a hot fix out ASAP to them is sensible. Turn around time might be in the region of 48 hours. By hot fix I mean fixing the specific issue only compared to the state of the source when the last release was shipped - aka the maintenance branch for the last release.

In terms of period between each formal release; that will depend, but we try to have complete iterations (release or not) every three weeks. Ideally it should be two weeks, but we don't seem to do it on that time frame. Basically its one week of planning and design, two weeks of implementation.

Regardless, there is no 'correct' way - only the way that provides best value to the customer. If the customer can't use your software thats a *bad thing* - and you better fix it or they won't be your customer much longer.

Depends on your field (1)

Digital_Quartz (75366) | more than 6 years ago | (#21341979)

I would suspect this largely depends on the kind of application you're developing.

Major problems in "cool blog software", for example, aren't really a huge issue. If it takes them a few weeks to be solved, then poor bloggers will be without some magic-pixie-dust feature for a bit.

In the telecomm world, though, customers expect a root cause for a "critical" defect in less than 24 hours (and there's a definition for critical, although I won't get into that here). My company actually writes that into our support contracts, so we need to root cause in 24 hours in a very real and legally binding way (although we do not contractually need to provide a fix in that time; sometimes we do, sometimes we just give a workaround).

Financial customers are even less forgiving.

A critical problem in, say, the fire control on an assault aircraft, is expected to never exist in the first place.

Two prong approach (3, Informative)

khendron (225184) | more than 6 years ago | (#21342005)

I've had situations with customers who require a fix as soon as possible, because if the system is down they are losing money. When this situation occurs, we have two goals in mind:

(1) Get the customer up and running again as fast as possible. This is as often as not some sort of workaround that is not pretty, nor is it permanent, but it works. The workaround does get thorough testing (impossible within the time frame) but the customer is aware of this and willing to accept the risks.

(2) Get the customer a proper, version controlled, patch that they can install to fix the problem permanently. This can take weeks, most of that time being testing. If the customer is insistent we will ship them the proper patch before it is fully tested (again, making them aware of the risks) and continue testing so that we can send the customer some warm and fuzzy news later on (or, if we find a problem, another patch).

Make a policy. Stick by it. Make it reasonable. (2, Informative)

postbigbang (761081) | more than 6 years ago | (#21342009)

Show stoppers get immediate attention; whatever it takes. People are losing money because they're DOWN. Fix it now.

Criticals get next attention when show stoppers are out. 48 hours, depending on interdependencies and QA needed to make it work; it's not part of an official stable code tree until later.

Minors are in the next stable branch release; every whatever you can handle.

Nigglies are changed when the stable branch releases.

Don't deviate from your policy, and make sure the sales people KNOW AND UNDERSTAND what this indicates and implies. No exceptions; see above.

How Fast is Your Turnaround Time? (1)

ilikejam (762039) | more than 6 years ago | (#21342019)

Depends what chair I'm sitting on, and whether the armrests hit the desk on the way round.

a lot of that can be shortened (1)

Surt (22457) | more than 6 years ago | (#21342037)

It depends on the level of comfort you need to have with your release, and how quickly you can get there.
I also work at a mid-sized company in the enterprise software business. We do a lot of automated testing. For an urgent, customer blocking issue, I could potentially:

1) get really lucky and diagnose the problem instantaneously. Realistically, it will take me at least one hour to properly diagnose any serious bug that escaped our qa process.
2) Make a fix. Assuming this is a fairly trivial fix, this can be done in minutes.
3) Submit fix to automated testing. Autotest turns around in about one hour.
4) Make a patch. This takes about 2 hours, as it requires an autobuild of the whole project plus some version control stuff. It can be done in parallel with the automated testing if the bug is sufficiently urgent. It might be conceivable that we could bypass our standard patch mechanism to speed things up, but we've never had a sufficiently urgent bug to do this in my knowledge.

So we can turn around a patch in about 3 hours (plus fix implementation time) if it was nightmarishly urgent. A more realistic turn around would be next day, which would allow time for us to run a manual qa pass, and to do a more thorough job of documenting and testing the bug (which in the given scenario would take place after shipping the fix).

What makes this at all possible is the high level of confidence we have in our automated testing. Our auto-tests cover our application surface really, really well, so passing auto test will give you a pretty high comfort level in giving something to a customer.

the fastest response (1)

belmolis (702863) | more than 6 years ago | (#21342047)

I'm surprised nobody has mentioned the fastest responses, which may take essentially zero time. One is: "It isn't a bug.". The other is: "Already fixed. Update your installation."

That's a big "it depends" (3, Interesting)

QuasiEvil (74356) | more than 6 years ago | (#21342053)

I'm an embedded developer, and when my stuff goes wrong, it can *really* do bad stuff. I've literally pushed fixed firmware to a controller running in a production scan/sort environment within five minutes of finding the bug, because it threatened to completely bring down a huge sort operation (and by huge, I mean 1 million+ pieces that day alone). I've also stayed up all night tracking down a bug crashing a device used by one of our larger (hundreds of millions of dollars per year) customers. Those, though, are the exception, and are driven by the massive financial and PR consequences of not getting it done right now. Throw caution to the wind, code and load if you are reasonable sure what's wrong and the stakes of not fixing it are high enough.

The usual bug fix cycle depends on complexity, impact, and risk. High risk of breaking things and low impact? Generally gets scheduled for the next release (4ish times per year). Low complexity and risk but medium impact? Code today, regression test the rest of the week, push this weekend. On average, mission critical bugs can get fixed in 8 hours or less around here, small to medium stuff is put on a weekly(ish) cycle with *lots* and *lots* of testing, and large stuff gets rolled to the next major release, unless it just can't wait that long.

What type of product? (1)

alta (1263) | more than 6 years ago | (#21342061)

I don't think there's enough info to even answer this one. I think the answer is entirely dependant on the importance of the program you put out. If you write rules for DoD firwalls, then you should be measuring in hours or minutes. If you write screensavers to put on OLPC... You have a little more time. And there's the inbetween, like if you write the algorithm that counts how many M&M's go into each bag.

No offense (1)

moogied (1175879) | more than 6 years ago | (#21342067)

No offense intended.. But you should be finding these things on your own and creating patches. Not releasing it out for your customers to locate. Its as if I buy a truck from Dodge/Nissan/Toyota/Wherever and I get it home to find out that if I hit it off after driving for 30+ minutes, and try and nhit it back on, it will only go in reverse. I expect that fixed ASAP. Not 4-6 weeks.

46 days (1)

Maxo-Texas (864189) | more than 6 years ago | (#21342071)

Minimum turnaround.

Large product. Billions in sales go through it.
ITQA takes 10 days with no defect- 20 days with a defect.

Scheduling installation time, Sox forms, required paperwork overhead, project approval comprise the rest. Actual coding and testing probably bout 8 to 16 hours.

At a small privately held company, it would take us 1 to 3 days.

fast (1)

BigBossBert (836389) | more than 6 years ago | (#21342089)

Depends on the complexity of the problem. But with only 1-2 mloc, problems usually aren't to complicated. If the problem is both serious and affects only a small part of the software, patches can be issued in 10-60 minutes after reporting the problem. More often though, it can take 1 or a couple of days before we fix a problem. The key to fast response it the ability to deliver small patches that don't take a lot of effort to install. Our patches are installed on a central server, and then automatically installed on each of the clients whenever the client starts a program.

People don't know how to buy software/tech (2, Insightful)

danilo.moret (997554) | more than 6 years ago | (#21342109)

That's why there are companies who think a minor bug fix, or a small development, changing fonts or simple features, reconfiguring servers, restoring backups etc is something that doesn't need testing, concentration, at least little bit of planning and basic things like version control. So that's quite common in the industry: customers who think they are getting their product for less money because they can force every change as an emergency. They don't realize they are making development more expensive with hacks and constant build, tests and deploys overhead. Simple concepts from lean methodologies like Scrum should be taught to anyone who plans to spend more than someone's monthly wage on software. Everyone can benefit from a healthier development cycle and software will come out better and cheaper. But there are some clients learning to get the benefits of a constant release cycle and, as the poster said, they are getting the beta development cycles for free.

I was thinking about a joke on my subject on the lines of "people only know how to buy tech on Civ", but it's less important and I'll leave it on the jokes backlog.

What's your experience level? (1)

Malc (1751) | more than 6 years ago | (#21342115)

That's not a question that can be answered here. Why would you even bother asking? Every piece of software is different. Every company has a different set of talent, different size budget, different processes and different sets of expectations.

Discuss Turn Around Beforehand (1)

darkmeridian (119044) | more than 6 years ago | (#21342145)

You need to hire a lawyer. Part of my job includes crafting software licensing contracts. In your services contract, define levels of bugginess--mission critical all the way down to insignificant. Set acceptable turn-around periods for each level of bugginess: two weeks for mission-critical to months for bugs that you can work-around. (These are just examples; I'm not providing legal advice; hire your own counsel.)

The more you give, the more they want (3, Insightful)

SystemFault (876435) | more than 6 years ago | (#21342165)

From nearly forty years of programming (yes, since the IBM 026 keypunch days), I can tell you with absolute certainty that the more that you do for management, the more that they will want from you. It is not your responsibility to bear all the punishment for the lack of foresight and resource allocation on their part.

Consider this: What would be the managerial response if you asked for a cost of living salary increase and that you needed it within 48 hours? Do you think that they would be willing to work day and night to make that happen?

Working in panic mode is not professional behavior, and it certainly is not conductive to good engineering practices. Furthermore, it is detrimental to long term company survival. Engineers who support continued unreasonable demands have only themselves to blame for enabling poor strategic planning by management.

What if you had to fix a spacecraft (1)

cjonslashdot (904508) | more than 6 years ago | (#21342177)

It depends. E.g., if your software controlled a spacecraft, and there was a mission-critical failure, you would be thinking in terms of hours to find a fix - not weeks.

simple solution (1)

edwardpickman (965122) | more than 6 years ago | (#21342199)

Just change the version number and ship the patch. Then you can take all the time you need to fix it right.

continuous fixes? (1)

groovepapa (857180) | more than 6 years ago | (#21342211)

been doing some Agile/XP work recently complete with Continuous Integration. we write website code and not box code, so might be a special case here, but our CI server builds in about 13 minutes. we have a production branch of the code monitored by CI so if/when we make a patch, it's immediately tested and built and ready for deployment. so if it's just an easy fix, it might be as quick as 30 minutes that we could deploy it out to the production website.

I also recall an experience I had with an open-source project in which I requested a somewhat trivial feature. the author added it to scm, sent me a patch file, and it was in the next nightly build as well. so that took about 12-24 hours.

in light of these experiences, 48 hours doesn't seem unreasonable to me, but I also agree with previous comments - it's a highly relative thing to judge. the best you can do is keep your code maintainable and try to reduce red-tape, I guess.

All depends on your SLA contract (1)

Vicarius (1093097) | more than 6 years ago | (#21342217)

It seems your managers weren't able to negotiate a good contract terms, and that's why customer is able to dictate such harsh terms. Whether customer is right or wrong, let them know, that out of Quality, Time (Speed), and Cost they can pick only two.

How well is the code documented? (1)

miffo.swe (547642) | more than 6 years ago | (#21342225)

I guess the real problem for many is the quality of the code in question and the level of documentation. If you as many OSS projects have very well written code that as a bonus is extremely well commented a patch can be issued within hours for Q&A. If on the other hand have a big pile of poorly made code that lacks both sanity and documentation, well, it will take time.

Spaghetti code can make even the easiest fix break things in totally unrelated places making Q&A a nightmare. If it takes you a couple of weeks to get a patch released something is broken. Be it management or code, it still sounds like a long time.

My own experience is that many OSS projects for some reason has extremely short patch periods and still dont break stuff all over when releasing a patch. One of the differences is the lack of centralized management. How much of the time for releasing the patch is really just waiting for approvals and such from others?

Too many unknowns (1)

Coward Anonymous (110649) | more than 6 years ago | (#21342241)

The answer to your question is - "it depends".
It depends on too many unknowns that may take you longer than 48 hours to explain on /.
The overall size of your code base is a meaningless metric - it could be a beautifully architected masterpiece that is easy to tweak and modify or lots of little pieces with relatively few interactions or it could be a 20 Mloc ball of spaghetti.
What is the complexity of the specific code with the bug?
What is the source of the bug? Is it easily reproducible? Is the location of the bug known?
Is the buggy code relatively insulated from the rest of the codebase or is it central to the entire product (think my_very_specific_function_for_twiddling_diddles() vs. printf()).
If you fix the bug wrong how many other parts of the code could be affected?
What are the ramifications of screwing up the fix beyond embarrassment?

Unless you know all of these, it is arguable you are not prepared to answer if the customer is being realistic (as opposed to reasonable).
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?