Journal FortKnox's Journal: Moderation System 38
Have you read Taco's latest journal?? He speaks about some moderation ideas he's brainstorming.
What truely peaked my interest was his idea of using weights. Maybe I'm going WAY overboard here, but here's my thought:
What truely peaked my interest was his idea of using weights. Maybe I'm going WAY overboard here, but here's my thought:
- I like the idea of all moderations affecting the post. If I start at score:2, get 4 upmods in a row, and 1 downmod, I should still be at 5.
- Pending moderation queues. I don't know how feasable this is, but what about making a moderation 'pending' and put into the m2 queue right away, and let people m2 their hearts out. Moderations go from pending->moderated if the m2 works well, or pending->removed if m2 says its bad (and the moderator still loses the point).
- Neural Net intelligent scoring. Based on M2 (for pending AND after moderating), weigh the moderators mod'ing ability. If they are constantly getting approved moderations, the weight will increase the moderation count (it'd mean scoring going into floats instead of ints, and there'd have to be a weight max, of course). Its the AI way of doing things.
I'd even be willing to code out the neural net neuron for the user. What do you guys think? Any other ideas (besides using K5's system)??
I like your ideas. (Score:2)
I wish that there was more group decision-making in the process, just as you say, so that it won't be so easy to mod down. To encourage people to mod up, maybe they should make a rule like "You need more people to agree with you to mod down, and it will cost you 2 mod points instead of 1.". I M2 on a regular basis, & I get the impression that there are people going around, and only modding down. Perhaps they are trying to get rid of the noise, or discourage group think, but still. I don't think that it just a mistake either, because a lot of them seem to be opinion based.
Another suggestion is to be able to mod up or down without affecting a person's karma. There are times, when a message could be moderated differently, but it's nothing to do with the person or opinion.
I definitely like the idea of a queue.
Your idea of +this -that = +5 might be better expressed as "vectored moderation": the positives & negatives shouldn't eliminate each other. The totals should cancel each other out, just like Physics & Algebra.
Keep up the good work.
Re:I like your ideas. (Score:1)
I like sllort's old proposal to ask the moderator the desired score, and then if Current Score Desired Score mod up, otherwise leave it alone. (The converse applies for downmods.)
Re:I like your ideas. (Score:1)
if Current Score < Desired Score, mod up
and for downmods:
if Current Score > Desired Score, mod down
Re:I like your ideas. (Score:1)
I get the impression that the moderator won't know the present scores. Am I correct?
Taco needs balls. (Score:1)
especially when he makes statements like "I know I'm gonna get flames over this." Geez, open it up for some discussion.
Anyway, On the part of moderation, I can see how the friends/foes thing will have a big effect on how overall moderation goes.
Re:Taco needs balls. (Score:2)
log (n) vs linear (Score:3, Interesting)
A couple +1's would boost you up through 2 and 3, but you'd need several more to hit 4, and significantly more to hit 5. The more people that moderated, the more stable a rating would be (in theory) Of course, you's need to implement all moderations affecting the comment, but I think it would be neat...
Re:log (n) vs linear (Score:2)
Re:log (n) vs linear (Score:2)
Actually, what you're describing is similar to how baseball votes for awards like MVP or Cy Young, only in reverse. In baseball, voters for a particular award fill out a ballot, ranking the top three candidates for the award. A first-place vote is worth 5 points, second is worth 3, and third is worth 1. Add up all the points, and Barry Zito beats Pedro Martinez for the AL Cy Young. <grumble />
I like your idea. Maybe make it a function of the current score. So, if you're at +3, it would take 3 upmods to reach +4, or 3 downmods to fall to +2. If you're starting at -1, 0, or +1, it only takes one vote to move in any direction, so a well-spoken AC can get some easy upward momentum, while trolls get smacked down right quick.
It may also address what you described in your other post: the 3-4 gap. This spreads out upmods enough that +5 is really meaningful, but allows other worthy posts to rise to +3 or +4. Upmods from 0 or +1 and posters with the Bonus tend to create a dense pack at +2.
What good is log? (Score:2)
Re:What good is log? (Score:2)
Uhhh (Score:2)
So why not just have the top rating be 50?
the point of log is that it requires exponentially more +1 moderations to get to the next level
Either way, those who have been modded up the most times will show up first. Why bother hiding how many times people have been modded up in some log system? Why not just show how many times the thing has been modded?
Re:Uhhh (Score:2)
Use the same system for karma, and give people lots of mod points. Something like karma/2 per week. Let the karma flow towards neutral. The bigger the absolute value of your karma, the faster it approaches 0. Like one point every 30/abs(karma) days.
And semi-related: new negative moderation reasons. At least give us "incorrect". Maybe "stupid" and "not funny" are a bit too inflammatory. "Crapflood" and "cut-n-paste" would be good as well.
Re:Uhhh (Score:1)
I, too have been disappointed with the moderation reasons... both positive and negative. "Incorrect" would be a good start. "Lame" would work for 'not funny'... "Obvious Karma Whoring" and "Cut-n-Paste" are both too long to go in a drop-down, I'm sure.
My favorite choices for negatives would be:
- Didn't read article (Uninformed?)
- Metoo
- Lame
- Bitching about site being slashdotted. (Whiner?)
- Running gag (ie, hot grits, natalie portman, beowulf cluster)
Maybe some people like the running gag things, but I think it's a waste of space; someone could've voiced their opinion, but instead went for a cheap laugh.Re:Uhhh (Score:2)
Not Funny, on the other hand, is enough self-explanatory that I could picture it in there ! I like your other ideas too (Incorrect, Whiner, Karma Whore, etc.)
Re:Uhhh (Score:2)
Using a log scale isn't hiding the number of people that agree -- it's the forcing factor that makes more people agree. You want the good comments to be seen quickly, but the best comments have to be voted on by a lot of people -- you can't really do that with a scale of 50, or any linear scale.
Re:Uhhh (Score:2)
50 is cumbersome? You don't think people can grok the meaning of 50?
One is easy to figure out, and one isn't. 49 means 49 people have voted this up - and 50 means fifty people have. If you want, I could explain 47 and 23.
If you want confusing, try being a moderator on a logarithmic scale (I used up all my points, but none of the scores changed?)
Using a log scale isn't hiding the number of people that agree
Actually, that's precisely one of the effects this method would have.
-- it's the forcing factor that makes more people agree. You want the good comments to be seen quickly, but the best comments have to be voted on by a lot of people -- you can't really do that with a scale of 50, or any linear scale.
OK, you understand that either method would function exactly the same way, right? Each logarithmic score would just be a way of displaying the true, linear score kept in the database. Any other way of implementing this would function exactly the same, and would be more cumbersome.
Either way, a comment would/could require the same number of votes to be at the maximum rating. Either way, comments that had got more positive votes would show up first. Only the displayed numbers would be different, and the log score's sole benefit would be that it's more confusing.
Re:Uhhh (Score:2)
Because if we limit a strict high-score to 5, the usual filters apply: you may want to read your comments at +3 or above; not necessarely +10 or +20 or +2000. That just wouldn't work because it's impossible to guess how much points would a comment ever have, on the other hand it's easy to define how "good" the comment ultimately becomes if we limit the highest score to +5.
Anyway I prefer the logarithmic system, but that's imho of course.
Btw: the upper limit (perfect 5 score) should only be achieved per story, with the comment achieving the single best metric. All the other comments should be computed relative to it, eg. we should have the next ones down weighted at 4.97, 4.89. 4.45, 4.0, etc. (just throwing numbers in to make my point - not actual computations.. I hope you get the idea :)
Ahhh (Score:2)
That does change things... Would probably work well.
Re:log (n) vs linear (Score:2)
We'd have to limit the scores granularity to 2 decimal places though, for clarity sake :)
Re:log (n) vs linear (Score:1)
Karma: 22.32834 (mostly affected by the bell curve)
piqued (Score:2)
My personal thoughts (Score:2)
So how could
You could then cap out moderator moderation at 3 or 5 and then only allow 5 if enough users agree with the post.
I'd assume you could do likewise with modding down (although that would be more difficult as it is possible a post gets modded down and enough people don't see it).
Some implementation problems:
* uneven user-level moderation (ie some people do it consistently, others don't). I guess then you could make it so users couldn't take away from a mod up, only cap it from getting a 5.
* web-load from gigantic amounts of HTTP-POST/GETs being thrown a
The gain: now meta-moderation or background heuristics are less important as this occurs at runtime. And instead of having some neural net chugging in the background for each user, it could just be a simple function over the amount of bumps a post gets (I also think # of Replies to a post should count as bumps as well. How can a post get 12 responses but not be modded up? Starting dialogue is as important as making good points).
Re:My personal thoughts (Score:2)
I think that's a really good point -- all my posts on /. seem to be schizophrenic like that. They're either modded up to 5, or they stick around at 2 and have tons of response. Strangely enough, it never seems to be both.
Re:My personal thoughts (Score:2)
What if the post is totally inane or completely off-topic, yet one of its responses is interesting and generates a mass of replies? What about flame-wars? This would cause problems with trolls replying to each other, thus causing their posts to be bumped up.
Now if the moderators karma (and moderator record - based on how many fair/poor moderations via M2) was used as a modifier, so the "good" moderators' moderations were worth more than moderations than a bad moderator, things could get interesting...
Re:My personal thoughts (Score:2)
+2 bonus (Score:2)
The biggest problem with the current moderation system is giving editors unlimited moderation points. Slashdot should be a 100% user moderated system instead of being to prone to editorial censorship.
Re:+2 bonus (Score:2)
Hell, I'd be happy with just getting rid of Michael's moderation capabilities. As far as what I have seen, only criticism of him and his postings are what really get abused. I think that editors should only be eligible for mods just like the rest of the users though. Let their accounts get weighted for mod points just like everyone else.
Re:+2 bonus (Score:3, Insightful)
Re:+2 bonus (Score:2)
Amen. :(
Btw, didn't we used to be friends ? ;)
Other ideas... (Score:2)
Your idea on the pending moderation status is a great one. But it would require more people to M2... Maybe have it so the first moderation in any category (Say a comment gets +1, funny, then -1, Offtopic) doesn't count, but the second one counts for one (then each additional is +1) so you'd have a comment with base score of 1, with +2, Funny, -1 Offtopic, and it would be scored at 2. If there were another -1 Offtopic, it would go to 1. It would neutralize the first moderation done, which I think would A) Help with too many +5 comments and B) require more cooperative moderation.
Re:Other ideas... (Score:2)
I think this is a flawed idea, because the labels for the moderation are totally objective and arbitrary. It doesn't make any sense to do this.
Think about how easy it is for something to be informative and interesting, or funny and insightful, or troll and flamebait. And so on. I think it could be argued that a funny and insightful comment should be given MORE value than something that is simply funny or interesting.
Having two moderations the same means one of the following happened: (a) two people who moderated a post happened to choose the same objective label for the post, or (b) the second person to moderate checked the stats of that post to see what the previous mods were and selected a mod that so they agree.
Could someone explain how this would make moderation more effective (and not, perhaps, even less effective)?
What about CF? (Score:3, Interesting)
With a CF system, every user would have the option to moderate any comment he/she wanted. The CF system would take the preference data that it had for each user (based on each user's moderations) and recommend other posts for viewing. Every user would have his/her own personal Slashdot.. Rating of posts would be handled through the intelligence contained in everyone's moderation ideas and weighted based on how similar two people's views had been in the past.
If you just opened an account and didn't have an accumulation of preference data for the system to use to determine how you were likely to moderate new posts, then the system could resort to using the aggregate moderation data to generate a list of popular posts. Also, individual users could choose to view Slashdot this way if they preferred it over the CF recommendations. Note, every time a user 'corrected' a recommendation with either a higher or lower moderation, the system would 'learn' more about that user's preferences.
Also, stories and story submissions could be handled by CF as well. With a well-designed CF system, there would really be no need for editors whatsoever, and Slashdot would branch off in a thousand different directions into a web of overlapping niche communities. The interwovenness and breadth of each community defined by each user's preferences. All and all it would be a far more rich system than what is in place today and would take Slashdot to the next level.
There is a system called MovieLens (just google for movielens and you'll find it) that uses this kind of system for moving ratings. I've found it to be extremely accurate -- far more accurate than any of my closest friends' recommendations.
Re:What about CF? (Score:2)
Anyway I like your idea a lot. But how about ease of implementation... If I were a programmer maintaining Slashcode, I wouldn't know where to start ;)
Re:What about CF? (Score:2)
Ease of implementation: This is a tough one for me to answer, as I have not written any code to do CF. Most of the current work in CF originated at research universities, but now there are some companies such as LikeMinds [macromedia.com] that have a commercial product. LikeMinds is now owned my Macromedia, and is not likely getting much marketing attention (many links on the Macromedia site pertaining to it are dead).
In fact, Likeminds (pre-Macromedia) created a site called www.moviecritic.com (now defunct) that worked extremely well. The Moviecritic project was a research project done by LikeMinds, probably to provide a real-world test for their technology.
There are a lot of resources (white papers, etc) about CF on the web, particularly if you go to MovieLens and look at the GroupLens Research Project at the U of MN. [umn.edu]
As for actually writing the code, I would guess that the hardest parts would be initially creating the CF engine (I don't know of any GPL'ed general purpose CF engines, but hey, that's an idea), and then making it perform adequately. Slashdot gets a bit sluggish sometimes, and adding CF won't make the load on the DB any lighter.
I'd sure like to see it happen, though.
I'm also reminded of Em Emalb's complaint (Score:2)
I think that, once a thread is off the main page it should then take on different moderation rules. Or maybe moderation can only occur if people are posting to the thread at a specific rate?
What is gained from using the neural net? (Score:1)
I understand and like the idea of having moderation ability somehow related to M2. What I don't understand is what you intend to use the neural net for? What is the adaptive part? The function that maps M2 results to mod-points?
If that is what you meant, why is this adaption necessary (or useful)?
Another way to use neural techniques to improve the moderation system is to categorize moderators into different groups (frequent posters, frequent M2:ers etc.) using some kind of unsupervised learning methods.