Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×

Big Challenges for Vista Bug Hunters 213

Posted by Zonk from the game-over-man-game-over dept.
The New York Times is reporting on the final rush to bug fix Windows Vista. Even with massive numbers of testers and five years of work behind them, the folks in Redmond are pushing it to the wire in order to make sure it releases soon. From the article: "Vista has also been tested extensively. More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft. Such data supplements the company's own testing in a center for Office referred to as the Big Button Room, for the array of switches, lights and other apparatus that fill the space. (A similar Vista room has a less interesting name -- Windows Test Technologies.) This is where special software automatically exercises programs rapidly while looking for errors."
This discussion has been archived. No new comments can be posted.

Big Challenges for Vista Bug Hunters More

Big Challenges for Vista Bug Hunters

Comments Filter:

  • special software (Score:5, Funny)

    by macadamia_harold ( 947445 ) on Monday October 09, 2006 @07:13AM (#16362215) Homepage
    This is where special software automatically exercises programs rapidly while looking for errors.

    and this software, folks, goes by the name "internet explorer".

    • Re: (Score:2)

      by Aladrin ( 926209 )
      No, that 'program' is called Internet Explorer maybe, but that 'special software' is an automatic testing application, probably developed internally and woefully incomplete.

      • Re: (Score:2)

        by rs232 ( 849320 )
        that 'special software' is an automatic testing application, probably developed internally and woefully incomplete.

        What platform did they develop this 'special software' and why don't they rewrite Vista in it. What errors would you get if you ran it on itself.
        • I think this "special software" would be more accurately described as a series of macros and scripts (in the dumbed down, Microsoft sense of the word) with a simple monitor listening for errors being written to the logs. As we all know, Microsoft logs are often woefully underpowered at recording major system errors. Of course this is a hunch but based on my past experience with Microsoft (dating back to the early days of DOS 2.0)I think it is a decent one.

    • Re: (Score:2, Flamebait)

      by pdbaby ( 609052 )
      I know a guy who used to work on test suites at Microsoft who has since quit, given their awful attitude towards bugs in Vista, and got a Mac

      • Re:special software (Score:5, Informative)

        by suv4x4 ( 956391 ) on Monday October 09, 2006 @09:38AM (#16363347)
        I know a guy who used to work on test suites at Microsoft who has since quit, given their awful attitude towards bugs in Vista, and got a Mac

        You'll see this kind of attitude in every bigger software company. I've had personal experiences like this in Adobe and Macromedia with their flagship products. Features are dropped, specs constantly changing and inconsistent between teams.

        In some cases, you can spot the same feature implemented twice in source, with different interfaces, in different locations, and code linking randomly to one or the other, or even both (imagine updating this).

        The bugs to be fixed are selected first for how obvious they are (likely to occur) and not how critical they are. This is why it's common that bugs that can totally wreck operation and lead to data loss may be left, if the occurence is rare or unlikely.

        Everybody is in stress and the main goal is that you get the reviewed bug off your shoulders: if it's mildly reminiscent to something else, it's marked duplicate. If you can't reproduce it quickly, it's marked as fixed or not reproducible. Tricky bugs are marked "fact of life" or "deffered".

        Successful companies and their products grow, but the way the products and resources are managed does not scale. Instead, programmers are expected to churn a major release every X months, screw everything else, and keep the cash flowing, the investors happy.

        With Windows, we have a successful product that supports a huge ecosystem of applications (including legacy support), localization, usage cases etc. It's natural that in time, updates will become more rare, and will be much slower and more expensive to produce.

        The trend of software-as-a-service is not coincidental with this situation. In 5 to 10 years the base software we use might be so complex and tough to work on, that the only way it can be sustained is by small, regular payments, and the updates will be small, incremental, security/performance oriented. No more big releases, no more rushes to fix bugs in the last moment.

        This is the way evolution works. The other route is, of course, revolution...

        • Re: (Score:2)

          by dubl-u ( 51156 ) *
          The trend of software-as-a-service is not coincidental with this situation. In 5 to 10 years the base software we use might be so complex and tough to work on, that the only way it can be sustained is by small, regular payments, and the updates will be small, incremental, security/performance oriented. No more big releases, no more rushes to fix bugs in the last moment.

          I think the interesting part will come when we see large, old code bases that started in the software-as-service world. A few years back I s
        • Ask yourselves, does software become increasingly more complex because it does more complicated stuff, or because it tends to become a kitchen sink?
        • Three things that struck me a few months ago as I was installing dozens of little updates to my Gentoo system:

          1) Most open source apps do not suffer feature bloat. They aren't trying to entice new customers so they usually just do what they are supposed to do.

          2) Updates are often done just to improve or optimize. Companies who are paying their programmers to churn out sellable products often can't afford to optimize every little bit of the app. Volunteers who just want to software to be its best can.

          3) A

  • Time (Score:4, Insightful)

    by Kangburra ( 911213 ) on Monday October 09, 2006 @07:16AM (#16362239)
    This was a similar story for Windows ME, in the end the time to release became more important than the quality of the product. I would like to see Vista delayed until it's ready, even if that's not for six more months. In my view that would earn Microsoft more points than meeting a schedule and then needing to (service) patch it fairly quickly.

    my $0.02

    • Comment removed (Score:5, Informative)

      by account_deleted ( 4530225 ) on Monday October 09, 2006 @07:20AM (#16362255)
      Comment removed based on user account deletion

      • Re: (Score:2, Insightful)

        by h2g2bob ( 948006 )
        Well, I suppose it's not like they've got a reputation to protect...

        Though to be fair people will have a go at MS if it's late OR if it has bugs, so they can't win. That said, either way people will be forced to use it.

        • Re: (Score:3, Insightful)

          by Dan Ost ( 415913 )
          either way people will be forced to use it.

          I don't know. Few people upgrade their version of Windows
          unless they're getting a new machine. However, lots of people
          are discovering that a 3 year old computer is perfectly capable
          of doing what they need it to do and so doesn't really
          need to be replaced unless the hardware is failing.

          Even more interesting, for the first time ever, Apple's
          offerings are starting to be percieved as a real alternative
          that is, arguably, comparably priced.

          It will be interesting to see w

      • Re: (Score:2)

        by finity ( 535067 )
        That update during the install is a great idea. I remember one of those worms made installing Windows hell. You'd install, then it'd reboot, and if it was connected to the Internet it'd be infected within about a minute. Too fast to install the necessary updates. I hope Microsoft gets vista right...

      • Re: (Score:2)

        by suffe ( 72090 )
        So any really big bugs that remain present in the RTM build can still be fixed later.

        Bill, is that you?

      • Re: (Score:3, Insightful)

        by Ucklak ( 755284 )
        Getting it out the door is more important than if it's ready.

        At the time it's released, Mac will have another OS out but that's beside the point. That only matters to people that are `on the fence` OS wise and not a significant number. In the halls of the OS engineers, it matters as it proves what insiders at MS have said that "Microsoft isn't able to ship products anymore [blogspot.com]."

        When SP1 is released, there will be hoopla and hype that Vista will have even more features, be more stable, and even more secure.

    • Re: (Score:2, Insightful)

      by Hennell ( 1005107 )
      But they will probally figure that delaying it for 6 more months suggests its still not good enough. So even if its not ready they'd prefer the patch route.

      All a bit academic really, the adverage person will get it good or not and slashdoters will complain it crashes and is insecure. Its like the circle of life.
    • To be fair, Windows Vista at Release Candidate is still leaps and bounds better than Windows ME was after service packs. Plus, if you were even slightly intelligent, you'd buy Windows 2000 instead of ME.

  • Huh.. (Score:5, Funny)

    by Anonymous Coward on Monday October 09, 2006 @07:21AM (#16362267)
    Half a million installs, and 450k of them crashed.

    Color me unimpressed.

    • Re:Huh.. (Score:5, Insightful)

      by Angostura ( 703910 ) on Monday October 09, 2006 @07:56AM (#16362499)
      Depends how you interpret the figures. I have a stable, well configured Mac. Last week, I had a dodgy 3rd party app that crashed 3 times. Each time the Apple crash reporter asked me to send a report to Apple.

      If I had been running a beta version of the operating system I would have gone ahead and sent, on the grounds that it might have been a bad interaction between app and OS. In the event I said no.

      You need to know more about what is triggering the crash reporter.

    • Re:Huh.. (Score:5, Funny)

      by alohatiger ( 313873 ) on Monday October 09, 2006 @08:56AM (#16362943) Homepage
      Or maybe 499,999 didn't crash, and one of them crashed 450,000 times.
      • Ah yes, I once had a mainboard like that too...

        Wait a minute, that mainboard is still in my_MQ\ZY61?'DK7N.DP#+U^4:.$NO CARRIER
    • Half a million installs, and 450k of them crashed.

      "450k ought to be enough for anybody." - Bill Gates [wikiquote.org]

    • I wonder how long they ran them for?

      Some of them were probably trying to get it to crash, and that can distort the numbers.

      I know a lot of people on here have Windows XP systems that have run forever as long as updates are made, but what's the average uptime for a Windows system? It could be that almost all Windows machines crash at some time during the beta period timeframe, in which case the 450,000 crashes would be expected.

      I might expect most people to send the data to MS because you as beta tester wan

      • Re: (Score:2)

        by Z34107 ( 925136 )

        Uptime on the Vista beta boxes is likely to be low - on a test system, you're going to have frequent reboots, I would suspect. Besides, this is the home-user, desktop version - Longhorn Server is coming out later.

        My Vista install has crashed a few times - 90% due to DivX not liking Vista, a few due to Media Center not liking DivX, and the rest video drivers crashing. All but twice Vista RESTARTED the driver and booted my game again, meaning I only saw two bluescreens.

        All the bugs, minus the DivX relat

    • The crash reports are for both the OS and applications running on it. Anytime an incompatible 3rd party app crashes, crash data is gathered and you are given the option to send it to MS. In addition, it is very likely that a person will try to run that same incompatible application over and over in hopes it will work.

      At any rate, my personal experience with Vista is that it's as solid as XP once you run RC1 or later. Apps do crash (usually older ones), but the OS itself is very stable.

      And before you make a
    • I enjoy how the thought of buggy third party applications apparently never crosses Slashdot's collective mind. I know that 99% of crashes on my PC are third party applications that frankly probably ignored the documentation and did something stupid. Microsoft has good people working on finding programs that rely on undocumented behaviors and putting in various bypasses to deal with them, but you can't expect them to be able to test the hundreds of thousands of applications out there before the public beta

  • I wouldn't want to be the guy (Score:4, Funny)

    by Anonymous Coward on Monday October 09, 2006 @07:21AM (#16362269)
    "More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft."

    That is the kind of information that can get people fired...

    • Re: (Score:3, Interesting)

      by jonnyj ( 1011131 )

      The real world is probably worse than the statistics suggest.

      I tried to install Vista on three PCs, all of which passed Microsoft's Ready for Vista testing tool, but all three failed before they were able to sent any crash data back to Microsoft. Two installs hung due because Vista didn't like my SATA / motherboard combination. The other got its knickers twisted over my partitioning scheme. And that was before I got a chance to find out if any of my other hardware (printer, scanner, TV card, firewire,

    • Is that crash data in general, or crash data on Microsoft applications or OS parts? I ask because windows can send crash data for any application which crashes...

  • Statistics! (Score:5, Insightful)

    by shreevatsa ( 845645 ) <<shreevatsa.slashdot> <at> <gmail.com>> on Monday October 09, 2006 @07:22AM (#16362283)
    More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft.
    In other words, about nine out of ten systems using Vista crashed at some point. And that's counting just those who sent the crash reports. :-)

    • Re:Statistics! (Score:5, Funny)

      by ImaNihilist ( 889325 ) on Monday October 09, 2006 @07:26AM (#16362301)
      That's really what happened in North Korea, they just don't want to admit it. Some noob installed Vista on one of the nuclear control computers, and then it crashed, and boom.

      Now the world will be destroyed, and we'll find out it was really Steve Ballmer's plan all along...then he'll throw a chair at something.

      Begun the dark times have.

    • Re: (Score:2)

      by rvw ( 755107 )

      I think crash data can also apply to applications that crash. Still, if only 10% has a system crash, that's a lot!

    • And that's counting just those who sent the crash reports.
      Why would you install a beta version of an operating system and not send in crash reports?
      Perhaps it was because 1 in every 10 couldn't get it to install/boot/work

      • Re: (Score:2)

        by arth1 ( 260657 )
        Why would you install a beta version of an operating system and not send in crash reports?

        Yes, in fact, I would. If I were beta-testing an OS, I would make sure I did that on an isolated machine, until I was certain that it was ready for being hooked into a network. And an isolated machine won't be able to send bug reports.

        --
        *Art

    • Re: (Score:2)

      by jkrise ( 535370 )
      In other words, about nine out of ten systems using Vista crashed at some point. And that's counting just those who sent the crash reports. :-)

      Or it could be that 1 out of every Vista system crashed 9 times.... and the remaining systems went into BSOD before the reports got dispatched :-)

      Did you notice ALL the chairs AND Tables are taken? The developers seem to have learnt their lessons!

    • Re:Statistics! (Score:5, Insightful)

      by Tim C ( 15259 ) on Monday October 09, 2006 @08:20AM (#16362671)
      I don't know about Vista, but on XP the default is to submit crash reports for all crashes. That includes software you are yourself developing. Yes, you soon learn to switch that off, but at least some of those reports will be from developers writing code for Vista and submitting crash reports for their own software (or testers doing so).

  • 450,000 of 500,000 people report crashes ? (Score:5, Funny)

    by Anonymous Coward on Monday October 09, 2006 @07:28AM (#16362319)
    Obiously 50,000 users didn't test anything at all.

    Just wanted to thank god for linux.

    • Re: (Score:2)

      by ledow ( 319597 ) *
      Or turned off crash reporting, or a behind a strict firewall, or never finished installing the thing, or never managed to get their network card/modem working, or don't didn't use it on an Internet connected PC.

      When you look at the possibilities, it's almost certain that EVERY user experienced some kind of crash, however minor. Whether that reflects on the state of Vista, or the state of modern operating systems in general, I don't know.
    • Or better yet, 50,000 installs of Vista crashed so badly, they weren't even able to send the crash reports!

  • This surely is the solution to sift through it all (Score:4, Funny)

    by QuatermassX ( 808146 ) on Monday October 09, 2006 @07:37AM (#16362391) Homepage

    Surely Microsoft could use this to sift through such an vast quantity of code: http://www.google.com/codesearch [google.com]

    Just please don't start hurling chairs at my Karma!

  • Hmm.. exactly 450,000 machines eh?

    What are the chances? Damn.
  • "This is where special software automatically exercises programs rapidly while looking for errors."

    I for one would love to know more about the tools they use for automated testing.

    In my company, we have a build & testing server running compiler and NUnit [nunit.org] tests for all data-layer tests (complete tests like "load all of everything" and more specific tests like "authorise user with known bad credentials - expect login-failure") alongside NUnitForms [sourceforge.net] tests for the application-layer (random, frantic clic

  • This is where special software automatically exercises programs rapidly while looking for errors.

    Maybe they should try hooking WGA up to this thing.

  • Why try, and fail to, reinvent the wheel... (Score:3, Interesting)

    by rsidd ( 6328 ) on Monday October 09, 2006 @08:23AM (#16362689)

    Microsoft wanted a more reliable machine, improved memory management, a better filesystem, etc... Instead of throwing resources at doing these things from scratch, why didn't they just

    • Take Linux, or one of the BSDs (like Apple did)
    • Spend small amounts of money improving it (all that's really needed is improved device drivers)
    • Spend some money on improving Wine (it would be really easy for them, compared to anyone else who wants to do it), et voila -- near-perfect backward compatibility (certainly far better than Apple's MacOS 9 -> MacOS X or PowerPC->Intel moves)

    From every point of view it seems to make more sense. They spend less money, get a more reliable product that can run very nicely on existing hardware, get some good press for a change, and benefit from the work of unpaid open-source programmers all over the world. But it isn't their way.

    • Re: (Score:2)

      by drsmithy ( 35869 )

      Microsoft wanted a more reliable machine, improved memory management, a better filesystem, etc... Instead of throwing resources at doing these things from scratch, why didn't they just

      Because there is zero benefit to them in doing so.

      • Re: (Score:2)

        by booch ( 4157 )
        Because there is zero benefit to them in doing so.

        Uh, there's a substantial benefit to them doing so -- a huge cost savings in their Windows development costs.
    • But not Linux, because then they would have to give back to the community.

    • Re:Why try, and fail to, reinvent the wheel... (Score:4, Insightful)

      by I'm Don Giovanni ( 598558 ) on Monday October 09, 2006 @10:38AM (#16363945)
      yeah, because *every* OS must be unix-based because it's perfect in every way, can't be improved. The peak of OS tech was achieved 30 years ago. [face_rollseyes]

      As for Apple, I wish that they had succeeded with Copeland, so there would still be at least one mainstream OS that wasn't Unix or NT based. Apple chose NexT (the BSD version (there was also an NT version)) out of desperation, not because they so loved BSD or Unix.
      • Apple was never going to succeed with Copland. Copland was a disjointed mess of technologies. Apple's other choice before NeXT was BeOS, also based on UNIX. Because Apple was desperate for a new OS doesn't mean the fact that they went with a UNIX-based solution was a mistake or something they settled on with reservations. On the contrary, I'm surprised anyone is arguing such given all the open source software that now happily runs on OS X and how open the system really is.

        There is something to be said f

        • Re: (Score:2)

          by drsmithy ( 35869 )

          Apple's other choice before NeXT was BeOS, also based on UNIX.

          BeOS was not UNIX-based. Heck, BeOS wasn't even multiuser.

          Because Apple was desperate for a new OS doesn't mean the fact that they went with a UNIX-based solution was a mistake or something they settled on with reservations. On the contrary, I'm surprised anyone is arguing such given all the open source software that now happily runs on OS X and how open the system really is.

          Because releasing Yet Another Unix is a) not particularly interest

          • BeOS was not UNIX-based. Heck, BeOS wasn't even multiuser.

            It was most definitely based on and inspired by UNIX.

            Because releasing Yet Another Unix is a) not particularly interesting technologically

            It's better to ship something that already works then something that some techies might find "interesting."

            b) surprising from a company infamous for NIH-syndrome

            Steve Jobs was always progressive when it came to UNIX back in the 80s.

            a bit lame in todays world.

            Again, this statement isn't backed up with anything.

            Apple

            • Re: (Score:2)

              by drsmithy ( 35869 )

              It was most definitely based on and inspired by UNIX.

              Rrrrright. So exactly which part of a microkernel-esque, pervasively multithreaded, single-user OS with a C++ API and GUI interface do you think was "based on and inspired by UNIX" ?

              Indeed, apart from a certain level of POSIX compliance, what _similarities_ are there between UNIXes and BeOS ?

              OS X supports things like SMP far better than NT does.

              Rubbish. OS X's SMP support is basic, at best. NT has been running on - and being tuned for - multiproc

    • Re: (Score:2)

      by SEMW ( 967629 )
      >Spend some money on improving Wine (it would be really easy for them, compared to anyone else who wants to do it), et voila -- near-perfect backward compatibility

      I don't think so. It is probably possible (though not easy) to get Wine working with any particular application you choose, though it may take more or less time depending on the development team you have. But to have something that works with *every* application? There are many millions of Windows apps out there, a large percentage of wh
    • If you knew much about Windows development, you would know that Microsoft spend a HUGE amount of time making sure they don't break popular programs between versions of Windows, and that is based on upgrading a similar code base. Getting wine up to that level would take years, and might not be possible (there are a few "WONTFIX" bugs in Wine due to programs doing horrible things which involve writing all over memory, which would require root access on linux).
  • tells us that 450,000/500,000 gives us a crash rate of exactly 90% seems about right for a windows system , can't think why they haven't released it earlier.

  • What about compatibility? (Score:3, Interesting)

    by UnknowingFool ( 672806 ) on Monday October 09, 2006 @08:30AM (#16362759)

    Something that I haven't heard much recently is about Vista compatibility. MS has said before that it will be compatible and for most software and hardware, it was true in previous versions. But there were enough exceptions. ME was supposed to be backwards compatible. But many specialized drivers had to be written for it. XP definitely required some driver updates. Since Vista is a architectural change, so one would except some compatibility issues especially when DRM and enhanced security is thrown into the mix.


    Technically would MS classify incompatibility as not a a bug, especially if is does not cause a crash?

  • high crash rate (Score:3, Insightful)

    by 192939495969798999 ( 58312 ) <[info] [at] [devinmoore.com]> on Monday October 09, 2006 @09:05AM (#16363027) Homepage Journal
    more than half a million installed, and 450,000 sent back crash data... so even if we assume it was nearly a million, that's 50% crash rate. I'd guess it was way higher even than that. So, over half of the systems were crashing bad enough for Microsoft to care? Wow! What exactly is the problem? I thought this was supposed to be a newer, better version. Wouldn't we see a 10% crash rate, or even a 25% crash rate at this point if things were really getting any better?

  • High failure rate (Score:2, Insightful)

    by DigitAl56K ( 805623 )
    "More than half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data back to Microsoft"

    So the liklihood of a crash is near 100% ?

  • Big bang testing (Score:3, Interesting)

    by plopez ( 54068 ) on Monday October 09, 2006 @09:29AM (#16363251) Journal
    Sounds like typical end of project 'big bang' testing. All those issues they ignored in development? Let's fix them now and fast! I would hate to be the MS QA person.

    (Yes I am aware I used singular, it was a joke, OK?)
  • Has anyone considered that MS is actually trying to make a decent OS here? Sure, it's fun to poke fun at them for all the crashes and bugs, but isn't that the ENTIRE point of the 'release early, release often' mindset?

    At least they're trying to find their bugs, at least they are running a widespread beta.
  • "... half a million computer users have installed Vista test software, and 450,000 of the systems have sent crash data ..."

    or, "Half a million installs of Vista, and almost half a million have crashed"

  • Haste makes waste. (Score:3, Insightful)

    by dpbsmith ( 263124 ) on Monday October 09, 2006 @10:56AM (#16364239) Homepage
    "Rushing to fix bugs" is like rushing any other meticulous job. It can't be done.

    Bugs are the consequence of rushing the job in the first place. (Taking time, is of course, necessary but not sufficient).

    If Microsoft knows a way to "rush" bug fixes without compromising quality, they would have been able to "rush" their development without creating the bugs in the first place.

Slashdot Top Deals

The Tao is like a glob pattern: used but never used up. It is like the extern void: filled with infinite possibilities.

Close