Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
×
The Internet

The Role of Experts In Wikipedia 266

Hugh Pickens writes "Episteme, a magazine about the social dimensions of knowledge, has a special issue on the epistemology of mass collaboration, with many of the articles focusing on Wikipedia. One of the most interesting articles is by Lawrence M. Sanger on the special role of experts in the age of Wikipedia. Sanger says the main reason that Wikipedia's articles are as good as they are is that they are edited by knowledgeable people to whom deference is paid, although voluntarily, but that some articles suffer precisely because there are so many aggressive people who 'guard' articles and drive off others (PDF), including people more expert than they are. 'Without granting experts any authority to overrule such people, there is no reason to think that Wikipedia'a articles are on a vector toward continual improvement,' writes Sanger. Wikipedia's success cannot be explained by its radical egalitarianism or its rejection of expert involvement, but instead by its freedom, openness, and bottom-up management and there is no doubt that many experts would, if left to their own devices, dismantle the openness that drives the success of Wikipedia. 'But the failure to take seriously the suggestion of any role of experts can only be considered a failure of imagination,' writes Sanger. 'One need only ask what an open, bottom-up system with a role for expert decision-making would be like.' The rest of the articles on the epistemology of mass collaboration are available online, free for now." Sanger was one of the founders of Wikipedia, and of its failed predecessor Nupedia, who left the fold because of differences over the question of the proper role of experts. Sanger forked Wikipedia to found Citizendium, which we have discussed on several occasions. After 2-1/2 years, Citizendium has a few tenths of a percent as many articles as Wikipedia.
This discussion has been archived. No new comments can be posted.

The Role of Experts In Wikipedia

Comments Filter:
  • by Anonymous Coward on Sunday February 15, 2009 @11:09PM (#26868281)
    Citation Needed
    • Re: (Score:3, Informative)

      by Anonymous Coward

      The average is usually below the best and above the worst. People "know" different truths. That wouldn't be much of a problem if only proven facts could be entered into Wikipedia, but Wikipedia actually uses a simple truth-by-discussion approach, weighed by the dedication which people are willing to put behind their point of view. This approach is cemented by the "no original research" rule. A citation is not proof, just a deference to an external evaluation mechanism. It is therefore no surprise that any o

      • Re: (Score:3, Informative)

        by mdwh2 ( 535323 )

        That wouldn't be much of a problem if only proven facts could be entered into Wikipedia, but Wikipedia actually uses a simple truth-by-discussion approach, weighed by the dedication which people are willing to put behind their point of view.

        No, it's about verifiable sources. You shouldn't be discussing what's true, you should be discussing what's supported by references. That's a fundamental Wikipedia policy.

        This approach is cemented by the "no original research" rule.

        Allowing original research would give t

  • by Protonk ( 599901 ) on Sunday February 15, 2009 @11:16PM (#26868343) Homepage
    Larry Sanger, the expert at making an online encyclopedia. We love to talk shit about Wikipedia here on /,; talk about how Knol is going to beat it or how Citizendium is better or how you wouldn't use it as a source (duh). But when push comes to shove, do we have any good competing models of how an online encyclopedia should be made?

    Do we have any good reason to trust Sanger as anything other than a provocateur? What is the meat of the analysis? That open editing and cooperation is what explains wikipedia's success? I'll agree with him there. And that control of articles or processes by internal "experts" is damaging to that open editing and cooperation? I'll take two, please. He's the big problem.

    We don't really know how to make a reasonably reliable, open and comprehensive encyclopedia without some deference to "local fiefdoms". We just don't. People don't contribute for money or fame. They don't have marching orders on which articles to keep free from vandalism or improve to featured status. They control their own production. Where that is the case they will bring themselves to edit on subjects they like and edit those articles in order to bring the distribution of coverage to their liking. We have to allow a little of that because it is those people who keep it from being a nuthouse. Those people spend 20-30 hours on wikipedia a week. They watch recent changes to keep subtle vandalism out. They fight back against civil POV pushers. They are an absolute necessity.

    To they come with drawbacks? Hell yes. There are probably thousands of people who have avoiding wikipedia as editors because their first edits were reverted--even though they might have been productive. I find lots of those reversions and usually don't get a cooperative attitude from editors when I call them on it. Those people make subtle cultural distinctions (I like this and not that). Those people form cliques and cabals. Those people make processes and bureaucracy.

    But I don't have a better way of organizing all of that free labor. Does Larry? Do you?
    • by Protonk ( 599901 )
      Lulz. "He's the big problem" should read "Here's the big problem. I fail at previewing.
    • by Spy Hunter ( 317220 ) on Sunday February 15, 2009 @11:49PM (#26868567) Journal

      Sanger has been saying stuff like this ever since he started Citizendium, but the proof of the pudding is in the eating, and Citizendium sucks.

      I think the problem is evident in his statement quoted in the summary: "Without granting experts any authority to overrule such people, there is no reason to think that Wikipedia'a articles are on a vector toward continual improvement". Well, unless you count the fact that collectively Wikipedia's articles have quite obviously been on a vector toward continual improvement since Wikipedia started. Wikipedia's article quality is not monotonic, but it is increasing. Under what metric is Wikipedia not getting better? Larry, stop speaking in generalities and point us to some actual evidence that Wikipedia articles are not increasing in quality.

      • by Anonymous Coward on Monday February 16, 2009 @12:13AM (#26868769)

        Well, unless you count the fact that collectively Wikipedia's articles have quite obviously been on a vector toward continual improvement since Wikipedia started.

        This is not obvious at all. Let alone "quite obvious". Indeed, it is false.

        • by Protonk ( 599901 ) on Monday February 16, 2009 @12:21AM (#26868817) Homepage
          [Citation needed]
          You can say a lot of things about wikipedia, but if you say that the majority of articles are worse off now than in (say) 2002, you'll be full of shit. You can point to some good articles in the past that have degraded from random edits. Or articles which have been subject to turf wars. But on the whole, there is improvement.
          • by Hal_Porter ( 817932 ) on Monday February 16, 2009 @01:01AM (#26869033)

            An improvement towards what though? Most articles have settled down to reflect the viewpoint of the people that watch them. If you agree with that viewpoint, that's an improvement. If you don't it's not and you give up citing, editing or reading them.

            • by 0123456 ( 636235 ) on Monday February 16, 2009 @02:01AM (#26869357)

              An improvement towards what though? Most articles have settled down to reflect the viewpoint of the people that watch them. If you agree with that viewpoint, that's an improvement. If you don't it's not and you give up citing, editing or reading them.

              And for non-controversial subjects, that generally results in a pretty good article.

              For controversial subjects, if the article is bad it's normally because policies aren't being followed: for example, the 'Apollo Moon Hoax' article (whatever the title may be today as it gets renamed regularly) has been a disaster zone for years, but that's primarily because it's an enormous violation of WP:UNDUE.

              • Re: (Score:3, Interesting)

                by Hal_Porter ( 817932 )

                The trouble is, lots of things can become controversial. Best example is "The Medieval Warm period". As far as I can tell everyone who looked for evidence of this in things like ice core temperature records found it. Unfortunately the fact that the average temperature of the Earth has been warmer in the past is awkward to people who claim that the current warming is unprecedented and an iminent disaster. So the Medieval Warm period article gets attacked by people trying to claim that the warming back then n

              • Re: (Score:3, Informative)

                Comment removed based on user account deletion
          • by chromatic ( 9471 ) on Monday February 16, 2009 @02:19AM (#26869457) Homepage

            [Citation needed]

            Indeed; can you provide citations that Wikipedia's aggregate quality has improved?

      • Re: (Score:3, Insightful)

        by samael ( 12612 ) *

        Well, unless you count the fact that collectively Wikipedia's articles have quite obviously been on a vector toward continual improvement since Wikipedia started.

        Citation Needed.

    • by vitaflo ( 20507 ) on Monday February 16, 2009 @12:06AM (#26868727) Homepage

      Yeah there really isn't a much better way. I ran into this recently on an article on Wikipedia on the Super Nintendo release date. I was a video game reviewer in the early 90's and have a letter signed by Nintendo's PR firm stating I was getting a review system before launch (which was a pretty typical thing for them to do). The letter was dated the same date as when the Wikipedia article said the system was actually released.

      Given this evidence I scanned the letter and posted it to let them know their date was off. Their response was that they couldn't use the letter as proof the date was wrong because they only used published sources of information. Unfortunately the only published sources they had were a handful of websites currently online that had the wrong date written down (no doubt copied from each other).

      At first I was taken aback by this as it was a bit odd that they would turn down physical evidence, but after thinking about it, it was obvious they didn't know me from Adam and can't just take people's word for things at face value, otherwise people could "prove" whatever they wanted. Those kinds of check and balances probably produce entries that aren't always perfect, but it's a lot better than the alternative in my mind.

      • by Darkness404 ( 1287218 ) on Monday February 16, 2009 @12:19AM (#26868813)
        One of the main problems of Wikipedia is it has firm guidelines on what it is and what it is not. There is a ton of information that could be released on Wikipedia but isn't because it isn't 100% verifiable or contains "specialty information". Now, I see where the editors are coming from, but similarly, for the average person interested in something (say a video game) there isn't any one major source of information about each one other than Wikipedia. Sure, you can find loads of reviews, a few walkthroughs, all the gameshark codes you want, and perhaps even a ROM or two of it. But as for real information on the game, that goes beyond that, you have to sort through mountains of Google searches with no real way that you can easily find it. Stuff like that I believe belongs on Wikipedia but keeps getting taken down from editors.

        Sure, I have no objection to vandalism being taken down, but the biggest flaw I see in Wikipedia is a lot of content gets deleted for no reason (face it, storage and bandwidth is dirt cheap).
        • by Beetle B. ( 516615 ) <beetle_bNO@SPAMemail.com> on Monday February 16, 2009 @01:19AM (#26869125)

          One of the main problems of Wikipedia is it has firm guidelines on what it is and what it is not.

          Actually, I wish. It's simply not true. They may have a few core rules (e.g. the one you complain about) that are quite rigid, but overall there is virtually no rule in Wikipedia that is not subject to modification - including by certain senior people at Wikipedia - when circumstances dictate it.

          Don't take my word for it - I got it from the horse's mouth [ted.com]. He says it in so many words.

        • Re: (Score:2, Insightful)

          by crossmr ( 957846 )

          Except wikipedia doesn't believe it belongs on wikipedia, it is why it is taken down. If you want to start a wiki about various games and include all the infinite detail and trivia about it, you can. You can even get it linked in the external links. There is a certain subset of the population of editors who can't seem to tell the difference between an encyclopedia and everything every written, said, thought and just made up about a subject ever. That being said the 10,000 sub articles on pokemon needs to be

          • Re: (Score:2, Funny)

            by Anonymous Coward

            Wait, "wikipedia doesn't believe it belongs on wikipedia"?

            Great, Now I'm going to have to update the Wikipedia page that lists all things that aren't listed in Wikipedia again.

          • Re: (Score:3, Interesting)

            by grumbel ( 592662 )

            Articles are meant to provide all the information the average person would find useful about a subject.

            By that argument most articles about physics and math could be deleted, because they are pretty much completly useless for somebody not working in that field.

            An encyclopedia is a concept that is based around the limitations of paper, Wikipedia isn't printed on paper and therefore should not follow the same restrictions.

          • Re: (Score:3, Insightful)

            by ultranova ( 717540 )

            Except wikipedia doesn't believe it belongs on wikipedia, it is why it is taken down.

            Wikipedia, not being a living being, is utterly unable to believe anything. Some deletionist scum, on the other hand, can only get it up by exercising petty power by deciding what article stays and what goes. It's the closest the ever get to wielding power over life and death (I hope).

            This is why I no longer contribute to Wikipedia and resist the impulse to correct any mistakes I notice: the reward is having my work delete

            • Re: (Score:3, Informative)

              by crossmr ( 957846 )

              Wikipedia, not being a living being, is utterly unable to believe anything.

              If you don't have the intellectual capacity to understand that I was talking about wikipedia as its community of editors then I can't wait for the rest of what you have to say..

              Some deletionist scum, on the other hand, can only get it up by exercising petty power by deciding what article stays and what goes. It's the closest the ever get to wielding power over life and death (I hope). This is why I no longer contribute to Wikipedia a

              • Re: (Score:3, Insightful)

                by ultranova ( 717540 )

                >

                If you don't have the intellectual capacity to understand that I was talking about wikipedia as its community of editors then I can't wait for the rest of what you have to say..

                It is not community who decides to delete a page or reverse an edit, it's an individual editor. The tendency of Wikipedia editors to take some article as their "own" and guard it jealously is infamous, and repeatedly referred in this very discussion.

                We're really getting to it now. Obviously you had some pet page you created tha

                • Re: (Score:3, Interesting)

                  by crossmr ( 957846 )

                  It is not community who decides to delete a page or reverse an edit, it's an individual editor. The tendency of Wikipedia editors to take some article as their "own" and guard it jealously is infamous, and repeatedly referred in this very discussion.

                  And with that you've shown a fundamental misunderstanding of wikipedia. Only in the case of a speedy deletion does 1 person decide to delete something. Even in that case a regular editor first tags it as a concern as meeting a criterion for that and then an admi

        • by DerekLyons ( 302214 ) <fairwater@@@gmail...com> on Monday February 16, 2009 @01:37AM (#26869225) Homepage

          Sure, I have no objection to vandalism being taken down, but the biggest flaw I see in Wikipedia is a lot of content gets deleted for no reason (face it, storage and bandwidth is dirt cheap).

          Which is why they just completed a six million dollar fund raising campaign. With cheap disk space, and cheap bandwidth, and volunteers doing the work... where is the money going?
           
          Setting that aside, the problem isn't the cost of disk space or bandwidth - the problem is the unseen cost of maintenance. Every article on Wikipedia requires some portion of an editors time to maintain accuracy, completeness, coherency, etc..., and to clean up behind vandals. And there are only so many editors at any given time. Too many articles means rot accumulates in the corners and moves inward. Too many articles means too many stubs that remain untouched. Too many articles means an increasing number of articles that say the same thing from different points of views.
           
          And frankly, based on a daily random sampling of articles, Wikipedia seems to be losing the battle.

          • by grumbel ( 592662 )

            And there are only so many editors at any given time.

            Well, yeah, because the rest of them got annoyed over that deletionism and just isn't contributing to Wikipedia any more. The whole problem with deletions is that they happen without a valid logical reason, they happen in a non-democratic manner (i.e. number of people writing the article doesn't help against a single admin wanting to delete it) and they quite often happen on articles that are quite useful.

            I have stumbled across interesting articles that got deleted later on far more often then about article

          • Re: (Score:3, Informative)

            by Anonymous Coward

            With cheap disk space, and cheap bandwidth, and volunteers doing the work... where is the money going?

            $130K - Board expenses, including D&O insurance

            $472K - Executive director expenses. Includes salaries for Executive Director and Deputy Director; some fundraising and travel expenses; consultants and contractors; staff and volunteer development

            $2.7M - Technology. Includes salaries for technical staff, servers, bandwidth and contractor expenses.

            $1.6M - Office & Admin. Includes salaries for finance/admin staff, audit fees, fundraising expenses, office rent, office supplies, bank fees, etc.

            $595K - O

      • Re: (Score:3, Insightful)

        by Jack9 ( 11421 )

        You can "prove" things via publishing that letter in a book. The people with money still successfully control information through Wikipedia. What a waste.

      • Given this evidence I scanned the letter and posted it to let them know their date was off. Their response was that they couldn't use the letter as proof the date was wrong because they only used published sources of information. Unfortunately the only published sources they had were a handful of websites currently online that had the wrong date written down (no doubt copied from each other).

        Not that you necessarily care right now, but you could consider sending the scanned copy of the letter to the publish

      • by DerekLyons ( 302214 ) <fairwater@@@gmail...com> on Monday February 16, 2009 @01:26AM (#26869159) Homepage

        Given this evidence I scanned the letter and posted it to let them know their date was off. Their response was that they couldn't use the letter as proof the date was wrong because they only used published sources of information. Unfortunately the only published sources they had were a handful of websites currently online that had the wrong date written down (no doubt copied from each other).

        Indeed, because I had a related problem. A series of related articles I wished to edit had considerable problems. I worked on the item described in one of the articles while I was in the Navy, I had the unclassified manuals at one elbow, at the other elbow I had a stack of expensive reference books... All were trumped because a handful of websites all referenced the same handful of coffee table books - and disagreed with me.
         
         

        At first I was taken aback by this as it was a bit odd that they would turn down physical evidence, but after thinking about it, it was obvious they didn't know me from Adam and can't just take people's word for things at face value, otherwise people could "prove" whatever they wanted. Those kinds of check and balances probably produce entries that aren't always perfect, but it's a lot better than the alternative in my mind.

        Except there aren't any checks and balances - there is only whether or not the guy you discussing the issue with has more time on his hands and whether or not he can quote an interpretation of policy that supports his position. Your story is one of how the checks and balances fail.

        • I worked on the item described in one of the articles while I was in the Navy, I had the unclassified manuals at one elbow, at the other elbow I had a stack of expensive reference books... All were trumped because a handful of websites all referenced the same handful of coffee table books - and disagreed with me.

          Two things :
          - Unlike the parent which had a letter, but from the point of view of other wikipedia could as well have been forged, you have references in *UN*-classified original manuals written by the Navy themselves. As they are not classified, they can be checked by any other user. It *IS* a reliable source. In addition, as it is written by the original maker of the thing, it could be considered as fist hand source, even better that website conveying second hand information.
          So the situation isn't the same

          • Re:Book references (Score:5, Insightful)

            by DerekLyons ( 302214 ) <fairwater@@@gmail...com> on Monday February 16, 2009 @04:59AM (#26870167) Homepage

            So the situation isn't the same as with the "I have a letter".

            The end result was the same - my edits were rejected because they didn't match the commonly, and easily, available sources.
             
             

            Ok, if the other users don't want to simply edit the date, maybe write a couple of lines saying that the commonly found data on the web and the manuals from the original source seem to disagree, citing your source.

            Well, it was more complex than the date - it was essentially an entire rewrite of half a dozen articles. On each, on the talk page, I noted my sources - and the same person (who had written all the articles and had most of the edits on them) promptly reverted my changes. (This was prior to 'citation needed'.) So I went to the talk page and explained that not only had I worked on this weapon in the Navy, but I had researched them for nearly twenty years and wished to share that research - and was promptly hit with the "no original research" hammer. (Never mind that any comprehensive historical article on the Wikipedia contains the same level of original research.)
             
            Between that and several other examples of asshattery, I realized I didn't have time to fight one fool with too much time on his hands, let alone half a dozen or so. I left Wikipedia and have never been back.

            • Re: (Score:3, Interesting)

              by Teancum ( 67324 )

              This sounds more like a Wikipedia editor who is violating the "WP:OWN" standards in an aggressive manner. It happens far more often than many Wikipedians care to admit, but for some types of articles it can be a real pain to get changes to happen.

              Writing on Wikipedia does take a bit of a thick skin and strongly defending your contributions to a certain extent. When you have a POV pusher or somebody who is upset with changes to "their" article, it can be even worse.

              Knowing your argumentative style here on

        • by Tom ( 822 ) on Monday February 16, 2009 @06:13AM (#26870469) Homepage Journal

          Indeed, because I had a related problem. A series of related articles I wished to edit had considerable problems. I worked on the item described in one of the articles while I was in the Navy, I had the unclassified manuals at one elbow, at the other elbow I had a stack of expensive reference books... All were trumped because a handful of websites all referenced the same handful of coffee table books - and disagreed with me.

          Those stories are probably legion. I've had problems having facts corrected on Wikipedia articles where I was the primary source (one about me in person, one about a project I lead) - and could easily prove so.

          But to Wikipedia, if three newspapers all quote the same source that got, say, my birthdate wrong, that is "more reliable" than me sending in a scanned, signed and confirmed by a notary, copy of my birth certificate. In fact, that would be entirely discarded as a "primary source".

          And that's why Wikipedia usually gets it right on common and readily available articles, and has a 50/50 on articles about obscure topics (other than nerdy geek stuff).

        • by RichiH ( 749257 )

          I know where you are coming from and I agree that the problem exists. Been there, it bit me, too. Hopefully, I never bit anyone else!

          The thing is that, as of right now, we do not have a better system. Hopefully, we will come up with one at some point, but you need to draw the line _somewhere_. And no matter what system we come up with, there will always be situations in which the existing checks and balances fail.

          PS: If those other books are better references, wouldn't it have been possible to let others kn

        • I've been in the same situation. I tried to correct some articles with obvious biases but got stuck in endless edit wars with those article guardians. It is extremely frustrating when you know that you are right, can cite secondary sources citing primary sources but are still reverted. A common tactic of "them" is to first ask for sources, then better sources and then claim that the fact is irrelevant to the article anyway. They are experts at wasting your time and making your editing experience frustrating

        • by Kjella ( 173770 ) on Monday February 16, 2009 @09:51AM (#26871509) Homepage

          Indeed, because I had a related problem. A series of related articles I wished to edit had considerable problems. I worked on the item described in one of the articles while I was in the Navy, I had the unclassified manuals at one elbow, at the other elbow I had a stack of expensive reference books... All were trumped because a handful of websites all referenced the same handful of coffee table books - and disagreed with me.

          The real problem here is that the best sources are not online. On wikipedia an online reference is worth so incredibly much more because it's verifiable - not as truth but as something others can read. Seriously, if you quote page 234 of the operations manual, how many would ever check that? How many even have the book or would hunt it down at a library? Most likely, none of the them know if you're bullshitting them or not.

          I know it's the long way around but you need to find or make credible online sources that reference the offline sources, then make the argument that your online sources are better than the ones currently used. Otherwise you'd see a lot of book spamming on wikipedia, where all the answers are in the spammer's book. Because of the delay from addition to fact checking plus the costs, it'd be an impossible problem. It's fair enough to reference famous books like Porter's book on Porter's theory but in general it would only undermine wikipedia.

      • Re: (Score:2, Interesting)

        by skroops ( 1237422 )
        I agree that Wikipedia editors shouldn't accept your letter as evidence to change the article. But I would think that out of everyone involved with the article, who would seem to have an interest in having the correct information, somebody would take the initiative to get a correction published. As those sources are credible enought to be cited, then they should be credible enough to judge your evidence.

        Someone should send the scan to the websites, along with an explanation about the situation.
      • by BrookHarty ( 9119 ) on Monday February 16, 2009 @01:56AM (#26869331) Journal

        Those kinds of check and balances probably produce entries that aren't always perfect, but it's a lot better than the alternative in my mind.

        This is also why articles are deleted on wikipedia, the editors are biased by their cultural knowledge. If its believed to be correct by their peers, then an alternate and possibly correct view cant be published due to the editors belonging to the same peer group.

        This is really difficult area when it comes to politics, groups/clubs, companies, history, ethnic, gender, minority, etc. You only get one side. Thus the reason it cant be used in colleges, its mass agreement by a very common peer group. And a good portion of the editors are very close in its peer groups.

      • Re: (Score:3, Interesting)

        by Tom ( 822 )

        You should have posted the article to your blog, and then told some gaming magazine to link to it in some "no idea what to write today" newsbit. Then you could've cited that as a secondary source and bingo.

        Laughable, isn't it?

      • Re: (Score:3, Interesting)

        by grumbel ( 592662 )

        A single source telling something different might not be enough to make a final call on a fact, after all the source could just be honestly wrong, it however should be enough to cast some doubt on the data and thus require further research. Especially when it comes to release dates it shouldn't be that hard to verify, there should after all be a plenty of magazines from back then that printed something about it.

        Anyway, the real crux here isn't so much the policy itself, but the reaction to your request on t

    • by Anonymous Coward on Monday February 16, 2009 @12:45AM (#26868947)

      Those with the most time on their hands wins.

      • Mod up. (Score:2, Interesting)

        by Anonymous Coward

        This is absolutely correct.

        It doesn't really matter if the people guarding articles are experts or not, or whether the other people who are trying to edit the article are experts or not. What matters is that somebody with enough free time to outlast the others will keep or gain control of the article and everybody else will eventually lose interest or give up in frustration and the article will became essentially the property of the person with nothing better to do than try to own it.

        So many Wikipedia artic

    • by Yuuki Dasu ( 1416345 ) on Monday February 16, 2009 @12:57AM (#26869011)

      We love to talk shit about Wikipedia here on /,; talk about how Knol is going to beat it or how Citizendium is better or how you wouldn't use it as a source (duh). But when push comes to shove, do we have any good competing models of how an online encyclopedia should be made?

      Well, I don't recall seeing full support here on /. for Knol or Citizendium. There seemed to be people on both sides of the aisle getting their voices heard. Many people attacked Wikipedia for its shortcomings, which you yourself agree exist. So, too, did many people attack the upstarts, with the standard arguments that come out whenever we discuss project forks or restarts here. Finally, there were people who had reasonable criticism for the upstart projects themselves.

      However, Wikipedia itself has been extensively criticized throughout its history; famously, its ability to accumulate knowledge and remove vandalism works "only in reality, not in theory." So on that note, how can we say that we know that those systems are inherently that much worse? Many would argue, as you yourself hint, that Wikipedia's supremacy is eminent in its dominance, and in its success. Yes, Wikipedia has orders of magnitude more content than its competitors. Please consider, though, the possibility that this is simply the result of first mover advantage and network effects.

      Wikipedia came on to the scene to find a fresh niche to fill. Earlier sites existed with similar goals to describe and categorize life (see: everything2 [slashdot.org] and h2g2 [bbc.co.uk]), but Wikipedia had a slightly different defined goal (be "the free encyclopedia") and software to ease the processes of collaboration and presentation. It took off like a shot and quickly established itself before it really had competition.

      Today, Wikipedia has somewhere around 2.8 million articles and a dedicated community. It seems immediately obvious to me that it's impossible today to compete with Wikipedia from square one; any competitor would need to fork the project or have their own equally impressive database. A database that size needs a huge support structure, not only in infrastructure but also in terms of volunteers/workers to police content. It's not possible to get those things all at once. There might be enough people out there who would be interested in helping a different project, but there's no way to get in touch with all of them. New talent either joins up with Wikipedia, or becomes disinterested when they don't fit with that group.

      Wikipedia got where it is because it was good enough at the right time. It does a lot right. It does some things wrong. It's not perfect. But then, I don't have to tell /. about technically inferior products dominating the marketplace due to familiarity...

    • by dargaud ( 518470 )

      There are probably thousands of people who have avoiding wikipedia as editors because their first edits were reverted--even though they might have been productive.

      Yup, besides some spelling corrections, one of my only contributions to wikipedia was to start a missing page (several links to it but no content). After a few days it got deleted: "Not enough context to identify article's subject". And deleted pages don't even have a history...

    • Well, good summary, I very much agree!

      I like to think of myself as one of those careful, involved editors who spends my free time "tending" and improving articles on my major areas of interest and expertise.

      What I've never understood about Sanger's argument is... what is it in Wikipedia's current model that /prevents/ experts from using their expertise?

      What makes an expert?

      I'd guess that an expert knows many reliable sources of information that they can use as references in their edits, to correct inaccurac

  • by rolfwind ( 528248 ) on Sunday February 15, 2009 @11:16PM (#26868347)

    to get where the controversy in the articles/subjects are, so as not to be led astray by any one current revision. I don't get the big deal about doing that extra bit of work.

  • Expert FAIL (Score:5, Insightful)

    by QuantumG ( 50515 ) * <qg@biodome.org> on Sunday February 15, 2009 @11:19PM (#26868363) Homepage Journal

    The purpose of Wikipedia is to approach consensus, not truth. The purpose of experts is to forward their own agenda ;)

    People who think either Wikipedia or experts are interested in the truth are likely to be confused.

    • Re:Expert FAIL (Score:5, Insightful)

      by Protonk ( 599901 ) on Sunday February 15, 2009 @11:29PM (#26868445) Homepage

      The purpose of Wikipedia is to approach consensus, not truth.

      I guess. This kind of critique gets pretty old. The whole point of moving away from "THE TRUTH" was to suggest that no one editing on wikipedia has access to "THE TRUTH". I'm not an expert. You're not an expert. Sure, we probably have our areas of expertise, but they aren't verifiable in a pseudonymous editing environment. In the absence of that verification, we have to trust references, not people. If I say "believe me, this is THE TRUTH", the right response is "Wikipedia isn't interested in THE TRUTH, do you have a source for that."

      It's cute to twist that around, but neither you nor I are Steven Colbert. We won't make it anywhere near as funny. To misread it to think "Oh, wikipedia is only interested in groupthink" is to miss the point. Lots of so called experts come on wikipedia and demand that people listen to them on the basis of their alleged expertise. When people (rightly) refuse to listen to them, those people storm off to /., their blog, or their cat and declare that Wikipedia is only interested in groupthink. Lots of time groupthink does grip wikipedia--just like any other organization. People see comments from editors they know and trust and respond accordingly. New people are often distrusted. These aren't features unique (or even uniquely salient) to wikipedia. They are features found in any community, large or small. Conflating the existence of groupthink with some underlying community desire> to govern through groupthink is inaccurate.

      Really, we don't mind the truth, so long as it has a little blue superscripted number after it.

      • Re:Expert FAIL (Score:4, Insightful)

        by Comatose51 ( 687974 ) on Monday February 16, 2009 @12:00AM (#26868667) Homepage
        I agree and would add another dimension to what you've said. A great deal of knowledge/truth are interconnected so even a non-expert in one area can detect inaccuracies or even outright lies in another when it contradicts what he knows in his area of expertise. Wikipedia has this web of inter-connection in place because of its links and citations. This, I think, pushes Wikipedia or any grand system of knowledge towards "the truth" because reality cannot be inconsistent or self-contradictory so neither can any representation of "the truth". A lie or false statement somewhere can undermine the entire system. We see this happening in science and other fields as the experts in those fields work out the inconsistencies. This also happens to an extend on Wikipedia.
        • Wikipedia has this web of inter-connection in place because of its links and citations. This, I think, pushes Wikipedia or any grand system of knowledge towards "the truth" because reality cannot be inconsistent or self-contradictory so neither can any representation of "the truth".

          Facts + interpretation = "truth"
          Interpretations are subject to inconsistencies and self-contradictions.
          Give us facts and please keep "the truth" to yourselves.

      • by ocularDeathRay ( 760450 ) on Monday February 16, 2009 @12:20AM (#26868815) Journal

        The purpose of Wikipedia is to approach consensus, not truth.

        I guess. This kind of critique gets pretty old. The whole point of moving away from "THE TRUTH" was to suggest that no one editing on wikipedia has access to "THE TRUTH". I'm not an expert. You're not an expert. Sure, we probably have our areas of expertise, but they aren't verifiable in a pseudonymous editing environment. In the absence of that verification, we have to trust references

        the solution has been right under our noses the whole time!... lets just have the guys at Britannica write our consensus for us. Think about it.. then you never have to question the source of our info, cause the REAL experts are taking care of us.

        personally I think all this wikishit is a load of tree hugging hippie crap. I am still using my 1994 CD-ROM version of the World Book encyclopedia because it sounded a lot more official than the names of the people writing wiki. I mean if I need to know the main industries of Sudan, I don't want to hear it from CaPtAinSwampA$$, I want it from a big faceless corporation I can trust.

      • Re: (Score:2, Insightful)

        by S77IM ( 1371931 )

        Lots of so called experts come on wikipedia and demand that people listen to them on the basis of their alleged expertise. When people (rightly) refuse to listen to them, those people storm off to /., their blog, or their cat and declare that Wikipedia is only interested in groupthink.

        If they were really such experts, they should capable of citing adequate evidence to back up their claims.

    • Re:Expert FAIL (Score:4, Insightful)

      by BrookHarty ( 9119 ) on Monday February 16, 2009 @02:01AM (#26869359) Journal

      The purpose of Wikipedia is to approach consensus, not truth.

      And one way is to shut out the people who don't agree.

    • by danaris ( 525051 ) <danaris@NosPaM.mac.com> on Monday February 16, 2009 @10:34AM (#26871931) Homepage

      The purpose of experts is to forward their own agenda

      I'm not sure how tongue-in-cheek you're trying to be here, so if you are, then this is directed at the people who really believe this.

      This is honestly one of the biggest and (to me) scariest and most incomprehensible problems with the American consciousness right now: the belief that not only do they, the average Americans, know better than the experts in their fields, but that those experts are, to a man, interested solely or primarily in putting forward their own "agenda"—which is necessarily something other than "educate people" or "show the truth".

      What reasonable basis is there for believing that everyone who's highly educated is somehow trying to subvert society to some nefarious end??

      Dan Aris

  • by Anonymous Coward on Sunday February 15, 2009 @11:20PM (#26868375)

    "Sanger was one of the founders of Wikipedia, and of its failed predecessor Nupedia"

    There is a reason why he failed; "Without granting experts any authority to overrule such people, there is no reason to think that Wikipedia'a articles are on a vector toward continual improvement," writes Sanger

    And all I have to say is ... so what? Articles at wikipedia frequently link to great off site expert resources. Not only that you have a wealth of choices of where you can get your information from, you're not limited to just wikipedia. If wikipedia was the only encyclopedia on earth his point might make sense. But given the level of alternatives and 'competitors' and number of wikipedia's contributors, there's a point where an article is more then good enough as a starting point.

    The difference between experts and amateurs
    in many instances is not large and for the most part negligible in many area's of knowledge, in other areas not so much.

    Not only that, experts frequently get things wrong, the idea that experts are monolithically better then amateurs and other experts also has serious problems. Given that there has always been contention about certain areas of knowledge, take history for example: How much important stuff is/was and is possibly currently being omitted from history by "experts" for any number of reasons that might bias their testimony?

    Wikipedia works because it has information people want. Experts frequently cull information they deem 'unworthy' of documenting, there's a whole host of articles on wikipedia about culture and entertainment stuff that would not normally be in a regular encyclopedia.

    Truth be told, Wikipedia is an excellent study in the controversial nature of knowledge. Experts are frequently wrong, the history of mankind is one of the constant error in the expert world.

    • I agree, experts are just as likely to disagree, especially on controversial issues.

      As well as that, what constitutes an expert, and has a ranking system been setup, because I haven't got my number yet.

      I would fee qualified to write an article on cluster analysis, which is a form of statistics, but not on other statistical methods. So, in order for such an expert system to be setup, either I would need to say what I'm an expert in (which would result in a situation no different from wikipedia now), or s
    • Wikipedia works because it has information people want. Experts frequently cull information they deem 'unworthy' of documenting, there's a whole host of articles on wikipedia about culture and entertainment stuff that would not normally be in a regular encyclopedia.

      Not really. Its the peer groups idea of what is "worthy".

      Take the mens rights article. Since there are no national mens rights movements. None that are a non-profit or officially accepted to speak as an Acceptable source for documentation. The major sources are infotainment reporters such as Glenn Sacks. The largest group of loosely related sites are Men going their own way (MGOTW), but its article and links on Mens Rights are often deleted due to the "popularity" concerns.

      None of the active editors are fa

    • by Tom ( 822 )

      Not only that, experts frequently get things wrong, the idea that experts are monolithically better then amateurs and other experts also has serious problems. Given that there has always been contention about certain areas of knowledge, take history for example: How much important stuff is/was and is possibly currently being omitted from history by "experts" for any number of reasons that might bias their testimony?

      Sources?

      No, seriously, come on. You're making a bold claim there. I know of a guy who trolls any place he can get to with his "knowledge" that 300 years of medieval history are fake and all the experts are in on the conspiracy.

      At least he provides some evidence for his claim that I can check for myself.

  • by Ex-Linux-Fanboy ( 1311235 ) on Sunday February 15, 2009 @11:29PM (#26868455) Homepage Journal
    It is my feeling that the Wikipedia is getting better. The community has put rules and procedures in place that make the place more pleasant to edit. In the mid-2000s, there were some issues where people could edit their own biography, and people could be obnoxious, flame and stalk other editors.

    Since then, policies and procedures have been put in place. You can no longer get in to edits wars without [[WP:3RR]] [wikipedia.org] stopping you. You can no longer belittle editors who disagree with you without getting blocked for [[WP:NPA]] [wikipedia.org]. You can no longer edit the article about your small open-source project [wikipedia.org] without getting slapped for [[WP:COI]] [wikipedia.org]

    Yes, these policies are not perfect, and yes a lot of articles still have unverified claims, and yes, like any democracy, it sometimes takes time and insanely excessive discussion to get to consensus. But the process on Wiki works and the new policies minimize the problems with articles. Did I mention that it's against Wikipedia policy to control articles on the Wiki, as per [[WP:OWNERSHIP]] [wikipedia.org]
    • But the process on Wiki works and the new policies minimize the problems with articles.

      Does the process stop editors from banning known authorities on a subject as vandals because they dared to correct the editor's pet page? I ask, because I know it's happened at least once, and said authority has never been able to get the ban lifted.

    • by DerekLyons ( 302214 ) <fairwater@@@gmail...com> on Monday February 16, 2009 @05:13AM (#26870229) Homepage

      Since then, policies and procedures have been put in place. You can no longer get in to edits wars without [[WP:3RR]] stopping you. You can no longer belittle editors who disagree with you without getting blocked for [[WP:NPA]]. You can no longer edit the article about your small open-source project without getting slapped for [[WP:COI]]

      Yeah, all those things have been rolled into one meta game - WP:TAA. (Wikipedia:Toss Around Acronyms) The end result the is the same - the win goes to he who has the most time on his hands.
       
       

      Did I mention that it's against Wikipedia policy to control articles on the Wiki, as per [[WP:OWNERSHIP]]

      Yeah, that's part of WP:NRETISRG. (Wikipedia:Not Really Enforceable Though It Sounds Real Good.)

    • by Tom ( 822 )

      Since then, policies and procedures have been put in place. You can no longer get in to edits wars without [[WP:3RR]] stopping you. You can no longer belittle editors who disagree with you without getting blocked for [[WP:NPA]]. You can no longer edit the article about your small open-source project without getting slapped for [[WP:COI]]

      That's the rules.

      And then there's their application.

      I've seen pages nominated for deletion again and again until finally a delete went through. I call that the deletion lottery, you can win it on almost any article that's not frontpage material.

      We've all seen random edits made, reversed, re-made and it was pretty clear that persistance is often more important than rules, knowledge, or even fact.

      And I have first-hand experience of how "well" these rules are sometimes enforced. I've had a COI case where you'r

  • I for one find it annoying to hear about people ragging Wikipedia about its accuracy. Its not meant to be a replacement for actual experience and training. Its a quick crash course. Its meant to be fast, easy, and relatively accurate. Some topics are going to be better, some worse.

    If your looking for information about some opensource project, it probably is pretty good. If you looking for a definition of some obscure abbreviation, your pretty good. If your looking for information about religion, what in the

  • by catbutt ( 469582 ) on Sunday February 15, 2009 @11:42PM (#26868525)
    Wikipedia does pretty well by tapping into wisdom of crowds. But what it really needs is a good karma system to get more quality out of it.

    You may complain about the quality of the comments on slashdot, but compare it to somewhere without any karma system. (this article [sfgate.com] sums up the problem with pure anonymity, and quite humorously so) Slashdot's system is not perfect, but it is a start in the right direction. I wonder how much wikipedia could be improved with a really good system. For instance, people with low karma would have their changes not show up immediately by default, or would be flagged as questionable, or what have you. People who didn't have a history of posting "good" stuff would tend to have few eyeballs ever see their stuff. There is a ton that could be done. It's tough to make it ungameable, but not impossible.
    • Re: (Score:3, Informative)

      I don't know if you have actually given 18 hours a day to Wikipedia. I have (expert enough :p), and I can tell you right now that Wikipedia already has karma [wikipedia.org] system [wikipedia.org].

      Now it is well established in Wikipedia "groupthink" that quality counts more than either. But the crux lies in the question: Who decides the quality?

  • by TheNarrator ( 200498 ) on Sunday February 15, 2009 @11:44PM (#26868535)

    Even leaving out the political issues, Experts are few ,and when well known, consider charging a lot for their work and would probably only devote time to getting published in a scholarly journal rather than some random website.

    It would be like running an open source project where the only people who are allowed to work on it are those people who hold a PHD or are certified to have 10 years experience programming with a major corporation.

    • by Protonk ( 599901 )

      Even leaving out the political issues, Experts are few ,and when well known, consider charging a lot for their work and would probably only devote time to getting published in a scholarly journal rather than some random website.

      It would be like running an open source project where the only people who are allowed to work on it are those people who hold a PHD or are certified to have 10 years experience programming with a major corporation.

      nail, meet head.

    • No. (Score:4, Insightful)

      by S3D ( 745318 ) on Monday February 16, 2009 @02:26AM (#26869493)
      Experts want recognition of their efforts. It should not necessarily be money. It could be some kind of "elite" account, credits, reputation system etc. Of cause no expert would want his contribution be mutilated by opinionated teenager or some crank with agenda. The problem is how to identify experts. Some ID system like OpenID based on the university/corporate site/homepage or like could be useful.
    • by DerekLyons ( 302214 ) <fairwater@@@gmail...com> on Monday February 16, 2009 @05:09AM (#26870201) Homepage

      Even leaving out the political issues, Experts are few ,and when well known, consider charging a lot for their work and would probably only devote time to getting published in a scholarly journal rather than some random website.

      Not true at all - I know many experts (keep in mind that expert and academic are not synonyms) that would love to edit the Wikipedia. But each and every one has ultimately been driven from Wikipedia by various forms of asshattery.
       
       

      It would be like running an open source project where the only people who are allowed to work on it are those people who hold a PHD or are certified to have 10 years experience programming with a major corporation.

      And that's a problem - how? It reduces the pool available for participation, sure. But the project is still open source and thus forkable by any individual who cares to do so.

  • by pongo000 ( 97357 ) on Monday February 16, 2009 @12:23AM (#26868833)

    ...when articles are tagged with the dreaded "primary sources" tag? In case you're not familiar with this tag, it basically states that the integrity of an article is in question because there are not enough cites from secondary sources (no, not a typo) as opposed to primary sources!

    Anyone with an academic background will recognize this acceptance criteria as anathema, as primary sources are usually the only sources that count when it comes to rigorous research. That said, a comment earlier about Wikipedia articles striving for validity through consensus rather than rigorous research now makes it very clear to me what is going on. At the least, Jimmy should be honest and clearly indicate to users that Wikipedia is more a compendium of collective wisdom rather than factual content.

    • by Protonk ( 599901 ) on Monday February 16, 2009 @12:52AM (#26868975) Homepage
      Anyone with an academic background will understand the difference and the reason behind the policy. Substantive original research takes time, expertise and effort. There is a reason we have long training periods for PhDs--we try to ensure those people are capable of conducting research independently. When you can't verify who people are, you have to limit the amount of "original research" they can do.
      It is just a different kind of writing than an original paper. People who make it out to be anti-academic or sneaky miss the boat entirely.
      • That makes no sense whatsoever. The grandparent is discussing the skill of the writer, but the value of the reference.

  • by Anonymous Coward

    First, take a look at this discussion by an expert. [artsjournal.com] (We now pause for the know-nothing kooks to ridicule Kyle Gann, claiming he's not an expert: Gann has of course written scholarly books on the subject of his expertise published by major academic institutions. Finished with the crackpot character assassination yet? Good, let's continue.)

    Now take a look at the Wikipedia article on the Chicago School of Economics. Does it contain any hint that the Chicago School's prescriptions were put into practice in Chi

  • """
    After 2-1/2 years, Citizendium has a few tenths of a percent as many articles as Wikipedia.
    """

    It's not about quantity, it's about quality.

  • What qualifies someone to be an expert?

    You cannot prove that you have degrees on the subject, but many list PHDs even if they never earned any of them. Wikipedia's staff had many members that got exposed as frauds. [wikitruth.info]

    When you cite a reference, they can quickly dismiss it by saying the source is not a reliable source. For example if it was not a liberal web site like the Huffington Post but rather the CATO institute, chances are the citation will be removed as well as the text that explains it. The same way for

  • by Choad Namath ( 907723 ) on Monday February 16, 2009 @02:40AM (#26869571)
    The problem with wikipedia isn't a lack of experts, it's a preponderance of self-appointed "experts." The place is crawling with people whose only qualification is having way too much time on their hands. The whole hierarchy of wikipedia seems to be populated with people like this.
    • They have another "qualification:" the need to show off how smart they are, irregardless of whether they actually know of what they are talking about.

    • I take your point but I think the other main problem is that for every expert there's a passive-aggressive hitler-type who will "guard" articles for no reason other that it makes them feel big. I have been on the receiving end of several of these types, moreso recently, despite now only very occasionally contributing even though there are several topics I can quite rightly claim to be an expert in. A few years ago I contributed heavily to WP and I think I added something of value, but I got very pissed-off
  • (1) There isn't any particular reason to think that credentialed experts are going to want to volunteer their work for free on wikipedia. Close it off to amateurs, and wikipedia will likely shrivel.

    (2) There isn't any evidence that wikipedia is any more unreliable than any other encylopedia -- what studies there have been show them to be roughly comparable.

    (3) The problem with volunteerism isn't the average quality of the volunteers -- the actual problem is subversion. As wikipedia, and things like it,

    • by bbtom ( 581232 )

      "Close it off"? Anyone can edit on Citizendium. The expert Editors just play the role of judging whether or not material in their particular area of expertise is accurate or not.

      There is a simple reason for this: Stephen Hawking knows more about astrophysics than me or most other dorks who hang out on the Interwebs all day long.

  • by Suddenly_Dead ( 656421 ) on Monday February 16, 2009 @02:52AM (#26869635)

    Perhaps associated with its culture of 'experts', or perhaps simply its low population. One needs look no further than the Homeopathy article, which on Wikipedia is strongly rooted in reality, but on Citizendium is largely controlled by one Homeopath editor (who has been banned from Wikipedia for pushing his unsupportable POV), and leans towards promotion and advocacy.

  • I added something that was a) true; b) supported by references; c) well-written; d) not inflammatory or controversial to a certain article. I took my time and did it carefully. Foolish me -- someone who was clearly the self-appointed owner of that article removed it within minutes.

  • How many of the articles on Wikipedia that are missing from Citizendium are cruft like blow-by-blow retellings of individual episodes of children's cartoons, or two-line stubs on insignificant elementary schools, or bot-generated template "articles" on American locations that contain nothing more than a few statistics regurgitated from census data?

    We need to remember that Wikipedia is not an encyclopedia. It is a repository of all the random facts and assertions that anyone was ever able to keep from being

  • Yepp, that's one of the many problems with Wikipedia.

    It does great in summing up what's been called "wisdom of the crowd". I think we used to call that "common sense", but that's less of a buzzword.

    The problem with "wisdom of the crowd" is that Wikipedia 1009 AD would have stated "the earth is flat" as a fact, and linked to many sources for the claim, even though experts had known for well over a thousand years that it isn't so.

    The greatest strength of Wikipedia - that anyone who knows something can come in

  • The original article is just arguing for enforced liberal bias. This guy just wants wikipedia to become the Soviet Union which gave him wet dreams as a boy.
  • by zijus ( 754409 ) on Monday February 16, 2009 @09:52AM (#26871519)

    Hi.

    A long while ago, I contributed to WP quite a bit. I stopped being subject to WikiStress [wikipedia.org]. I learned one thing at that time : when WP things gets on your nerves, just get a break. Then, one realise that an article is no one's little pet. And now one can come back to an article and forget about it easily. Many people who claim to be specialist or to be especially rightful, should take a WikiBreak... and come back later. No, later than that. Now, contributions starts being good.

    I also observed /. threads about WP : They where rather poor, often being aggregation of blatantly incorrect statements. Reason IMHO was that not so many actually contributed to WP, thus ignoring what I now can read in this thread : WP is not a school yard, there are rules, these rules can indeed be gamed, WP is no one pet's toy : being a so called expert does not yield special status, consensus is indeed the driving thing in WP - not The Truth - and so on...

    That's a lot of good comments in one single WP /. thread. Pleasantly surprising. I suppose WP is getting actually known by people. Nice!

    Z

  • by guruevi ( 827432 ) on Monday February 16, 2009 @11:34AM (#26872649)

    Wikipedia is more of an experiment in some type of direct Democracy/Demarchy (where the whole world rules) while Citizendium is an experiment in Meritocracy (where only the smart rule)

    We see the results after a few years. Wikipedia has a much higher level of quality even though there are a lot of bad things about it. Citizendium might be more factually correct on some issues but the problem is that it doesn't have the vast amount of resources Wikipedia gives. Which one is better is a matter of choice. If you give negative points for anything that is not existing, Wikipedia wins. If you don't care that it's not there but what is there is correct then Citizendium is more correct. In the end Wikipedia will be more correct on current culture, things that are evolving constantly and new items while Citizendium will get things that are more scientific better. I think the two should merge some articles and things that are a matter of fact (history, mathematics, chemistry) should be better locked down on Wikipedia. On the other hand our understanding of matters is continually improving even on history but the fact is that even historians keep on rewriting history. Eventually Bush and Hitler will be good (or not-so-bad) guys to certain generations and within a few hundred years we'll have explored the sub-sub-quantum physics as well. It's all evolving and no knowledge is permanent even if they were a matter of "fact", it's all in the eye of the observer.

  • by unity100 ( 970058 ) on Monday February 16, 2009 @01:07PM (#26873947) Homepage Journal

    to which experts can take the contested and faulty articles, and the editions can be decided by input from admins, experts, and people.

    you cant fight something that is solidly proven, and anyone still fighting such an edit will probably be extremists/radicals in wikipedia.

"When the going gets tough, the tough get empirical." -- Jon Carroll

Working...