Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Close but no Cigar for Netflix Recommender System

CmdrTaco posted more than 5 years ago | from the fifty-grand-aint-bad dept.

The Almighty Buck 114

Ponca City, We Love You writes "In October 2006, Netflix, the online movie rental service, announced that it would award $1 million to the first team to improve the accuracy of Netflix's movie recommendations by 10% based on personal preferences. Each contestant was given a set of data from which three million predictions were made about how certain users rated certain movies and Netflix compared that list with the actual ratings and generated a score for each team. More than 27,000 contestants from 161 countries submitted their entries and some got close, but not close enough. Today Netflix announced that it is awarding an annual progress prize of $50,000 to a group of researchers at AT&T Labs, who improved the current recommendation system by 8.43 percent but the $1 million grand prize is still up for grabs and a $50,000 progress prize will be awarded every year until the 10 percent goal is met. As part of the rules of the competition, the team was required to disclose their solution publicly. (pdf)"

cancel ×

114 comments

1.57% (1)

stoolpigeon (454276) | more than 5 years ago | (#21349235)

Is the new margin of improvement for victory then?

Re:1.57% (1)

Billosaur (927319) | more than 5 years ago | (#21349375)

I would think they would not implement the AT&T team's solution given it did not reach the 10% mark, however AT&T has the lead in reaching that mark unless someone comes up with some quantum leap in design./p.

Re:1.57% (4, Funny)

stoolpigeon (454276) | more than 5 years ago | (#21349547)

Right - but the AT&T method is public - so if you can get that the rest of the way to 10%..... I guess step 3 is profit.

Re:1.57% (0)

Anonymous Coward | more than 5 years ago | (#21349921)

"Right - but the AT&T method is public - so if you can get that the rest of the way to 10%"

Except the 10% is now a moving target. You need to improve by 10% over Netflix's current method. If, each year, they implement the smaller improvements, next years contestants have even a larger percent improvement to archive from when it all started.

Re:1.57% (1)

OctoberSky (888619) | more than 5 years ago | (#21349477)

Actually, and my math will suck here. If they implement the new AT&T data, and then ask for 10% it would be much harder then what AT&T themselves did.

If they were at 50% accuracy and AT&T gave them 8.63%, if they implement that they are now at 58.63% accuracy. If they require a 10% increase then a new person will have to bring them up to 68.63% accuracy, much harder then the 60% AT&T was aiming for. Assuming that it becomes harder as you get closer to 100% accuracy.

Re:1.57% (1)

OctoberSky (888619) | more than 5 years ago | (#21349525)

I gave AT&T an extra 0.2% because I love Big Brother. Not because my math sucks.

-Dick Cheney

Re:1.57% (1)

moderatorrater (1095745) | more than 5 years ago | (#21349571)

My guess is that they're giving the award for cutting out 10% of the inaccuracy, ie if they're at 50% and you can get them to 55%. At that point, another 10% would be 4.5% instead of 5%. This is because there's almost no chance of getting to 100% probability, so you're going to the limit of 100% without any chance of getting there.

Not how it works (4, Informative)

illegalcortex (1007791) | more than 5 years ago | (#21349599)

That's not how the contest works. It's based on the RMSE that the original netflix algorithm got at the beginning of the contest. This is fixed and does not change. See the contest site for more details.

diminishing returns (2, Interesting)

caffeinemessiah (918089) | more than 5 years ago | (#21350763)

From my experience with the Netflix Prize, and ML/stat.learning techniques in general, that last 1.57% is going to be the hardest. There is a diminishing returns effect going on here, i.e. the effort required for each successive 1% increase gets progressively larger.

Don't meme me bro (2)

sakdoctor (1087155) | more than 5 years ago | (#21349243)

Any chance of not tagging this story with this meme?

Re:Don't meme me bro (-1, Offtopic)

Anonymous Coward | more than 5 years ago | (#21349421)

dontalgorithmmebro

Re:Don't meme me bro (-1, Offtopic)

dintech (998802) | more than 5 years ago | (#21349437)

dontrecommendmebro

Re:Don't meme me bro (-1, Offtopic)

Anonymous Coward | more than 5 years ago | (#21349489)

been there. done that. will do it again. bro.

Re:Don't meme me bro (0, Redundant)

east coast (590680) | more than 5 years ago | (#21349667)

dontmodmedownbro

Re:Don't meme me bro (1)

MontyApollo (849862) | more than 5 years ago | (#21352661)

dontmodmedownbro

That was funny

Re:Don't meme me bro (-1, Offtopic)

Anonymous Coward | more than 5 years ago | (#21349817)

No.

I'd say... (4, Insightful)

Otter (3800) | more than 5 years ago | (#21349251)

If the people who created Netflix's system are still with the company, I'd say they deserve some retroactive recognition (and bonuses). That's pretty damn good optimization if it's that hard to improve upon, and there seem to have been some really sophisticated people trying to beat them.

Re:I'd say... (4, Funny)

wattrlz (1162603) | more than 5 years ago | (#21349307)

Perhaps they should look at whatever chooses the slashdot page-bottom quotes for inspiration: Mosher's Law of Software Engineering: Don't worry if it doesn't work right. If everything did, you'd be out of a job.

Re:I'd say... (2, Interesting)

Billosaur (927319) | more than 5 years ago | (#21349327)

It's hard to say. On the one hand, it could be that the current system is good enough that improvements are minutely incremental, though 8+% is pretty good if you ask me. On the other, it may be that the system is so fraught with dependencies and/or the relationships are so variable that it's hard to make gigantic leaps in sophistication. Look at Amazon's recommendation system: pretty good overall but still makes some egregious errors. Add the tagging system to the mix and it's possible to lead the recommendations astray. I don't know how Netflix works (I don't have to watch movies on TV, let alone rent them), but I have to think it would be very similar to Amazon.

Re:I'd say... (5, Funny)

Anne_Nonymous (313852) | more than 5 years ago | (#21349579)

Customers who bought the items in your shopping cart also bought:

Empress Charmeuse Silk Sheet Set - Queen - Ivory ~ $399.00
Black Leather Victorian Vintage Shaper Corset Boned Lace Up corset ~ $119.99
Orgazyme Clitoral Stimulation Gel, Topical, 0.8 oz ~ $20.79
Pampers Cruisers, Size 4, Economy Plus Pack, 140 Cruisers ~ $38.99

I don't think Amazon has much room to improve their recommendation technology.

Re:I'd say... (4, Informative)

DerekLyons (302214) | more than 5 years ago | (#21351167)

ook at Amazon's recommendation system: pretty good overall but still makes some egregious errors.

Egregious errors? It's downright useless unless you pretty much buy only one genre of book/music/whatever. Their system is heavily weighted towards whatever you most recently bought - and drops huge slabs of quasi related stuff into your request list at the slightest provocation.
 
I buy (among other things) serious works of culinary history, sociology, etc... Yet my reccomendation list is clogged with food porn (coffee table cookbooks) and the latest crap offerings from whichever TV chef is the current flavor of the moment. It also doesn't recognize the difference between editions - if you buy a hardback, it'll happily reccomend you buy the paperback. If you buy a frequently reprinted SF novel, it'll happily add each new printing/edition to your queue.

Re:I'd say... (2, Interesting)

IceFox (18179) | more than 5 years ago | (#21353277)

Most everyone who tried was able to beat Netflix's existing system. I put together a little framework to help people get up and running faster. A lot of people seemed to be spending time just getting all the data into memory before they got to play with any algorithm ideas. I include a few algorithms including Simon Funk's which should be enough to get you started. http://www.icefox.net/programs/?program=NetflixRecommenderFramework [icefox.net]

AT&T Labs? (1)

NickCatal (865805) | more than 5 years ago | (#21349397)

At the risk of getting marked redundant: I totally agree, and I don't work for Netflix

This is a great contest, considering they have to publicly release the solution.

Although what is AT&T doing working on this problem?

Re:AT&T Labs? (2, Insightful)

Anonymous Coward | more than 5 years ago | (#21349479)

AT&T Labs = bunch of people from former Bell Labs = welfare for AI researchers ;)

Re:AT&T Labs? (1)

eviloverlordx (99809) | more than 5 years ago | (#21349591)

It's too bad they didn't win the full prize, but at least they now have lucrative job opportunities in the booming spam industry. I keep getting spammed with Viagra ads and illegal replica watches, but I really want all of the hardcore pr0n spam.

Re:I'd say... (4, Interesting)

bigbigbison (104532) | more than 5 years ago | (#21349545)

I think the problem is that (and I may be wrong) any new system that researchers come up with isn't allowed to ask the user for more information. This would make if very hard for any system to be acurate if it is based soley on what dvds you rented and how you rated them.
If I liked Die Hard 4, for example, did I like it because of Bruce Willis, the "I'm a Mac" guy, the special effects, the plot, or some other reason that even I don't know?
Personally, I know that I have rated something like 900 movies on the netflix site and nearly all the recommendations are things I've no interest in or they simply say, "Sorry we have no recommendations for you at this time."
I would like to think that if they could ask me why I rated one movie a 4 and another a 1 then they might have more accurate recommendations. Even if they just had a drop down menu with something like, "I liked this movie because of a) the starts, b) the plot, c) the genre and so on" it would make recommendations a lot easier.

Re:I'd say... (2, Interesting)

antifoidulus (807088) | more than 5 years ago | (#21349639)

Thats why they need to get multiple data points to make a recommendation. If you rented a lot of the "I'm a Mac Guy" movies and rated them highly, then there is a bigger chance that that is the reason you liked that movie. If you refused to rent, or rated poorly the movie "Rugrats Gone Wild" then you probably aren't a rabid Bruce Willis fan etc. The entire goal of the project is to find films you like without you having to do a mini-review of every movie you have rented/saw.

Re:I'd say... (3, Informative)

Fnord666 (889225) | more than 5 years ago | (#21349911)

Thats why they need to get multiple data points to make a recommendation. If you rented a lot of the "I'm a Mac Guy" movies and rated them highly, then there is a bigger chance that that is the reason you liked that movie. If you refused to rent, or rated poorly the movie "Rugrats Gone Wild" then you probably aren't a rabid Bruce Willis fan etc. The entire goal of the project is to find films you like without you having to do a mini-review of every movie you have rented/saw.
One of the common approaches to recommender systems is SVD, or Singular Value Decomposition [wikipedia.org] . SVD tries to isolate "features" in the training set that best represent a particular trait of the data and its value, such as the examples above. You may not have any idea what the feature actually represents, but that is fairly common in machine learning. It is an iterative process. Once you have defined one feature as well as you can, you move on to a new one. There are diminishing returns with this approach though, and identifying too many features can overspecialize your system and yield worse results. If your results are not good enough, you can try a different approach. Once you have tried several approaches that are almost good enough, you can try combining the different results to varying degrees to get a hopefully better result. That is what the leaders have done so far.

Re:I'd say... (1)

flymolo (28723) | more than 5 years ago | (#21350121)

I think this problem is solved through cluster analysis. The same way to tell if someone has multiple disjoint interests or a family is sharing an account. The goal is to find a "type" of thing you like and predict other things in that bin.
Your predictions will be more accurate if you don't try to match with other people who liked Die Hard 4 and Finding Nemo, unless they only rent movies that have CGI/special effects.

Re:I'd say... (0)

Anonymous Coward | more than 5 years ago | (#21350657)

Dear lord... you've watched 900 movies? That's nuts

Re:I'd say... (0)

Anonymous Coward | more than 5 years ago | (#21353453)

'not interested' counts as a rating.

Re:I'd say... (1)

bigbigbison (104532) | more than 5 years ago | (#21354353)

I haven't rented 900 movies from netflix but in my entire life I've certainly seen a lot more than 900 movies in my 30+ years.

Re:I'd say... (1)

SpiritOfGrandeur (686449) | more than 5 years ago | (#21350867)

You don't need to explicitly say this to get the data. Movies have a lot of information in them. You can tell which actors are in it, the plot, and the genre. If you make two selections and they both have Bruce Willis in it you can assume that you like Bruce Willis. You do not need to ask these questions of the user.

Re:I'd say... (2, Insightful)

Potatomasher (798018) | more than 5 years ago | (#21351041)

The idea is that with enough data, you could extract the "why" automatically. For example, if you rated all Arnold Scwarzenegger 5, then its probably because you like Arnold. If however you gave a rating of 1 to Kindergarden Cop, as well as "The Game Plan" and a bunch of similar movies, the system could also infer, that as much as you like Arnold, you don't like kids movies starring washed up "action" movie stars.

This is the whole idea behind the field of "machine learning": inferring causes/relationship/structure from raw data.

Is this feasible ? Maybe, maybe not. Is this easy ? Definitly not.
But then again if it was, they wouldn't put a 10$ Million prize tag on it.

Re:I'd say... (1)

fropenn (1116699) | more than 5 years ago | (#21353051)

One additional confounding variable is that my wife and I both rate movies on the same Netflix account. She tends to like romantic comedies, me, not so much. But on Netflix, you would see both romantic comedies and thrillers rated highly, even though for an individual person that connection may not make sense. So you would also need to build into the model a way to identify cases like mine where the ratings are really the results of two people.

Re:I'd say... (1)

coaxial (28297) | more than 5 years ago | (#21353085)

You know what movies people like, and which ones they don't. Compare the movies together, and you can tell that the only reason someone likes these movies is because the have macintosh in them. Anyway, people HATE giving feedback. No one likes filling out a questionnaire. It takes way too long. You suggested a drop down. That only allows a user to pick a single reason. What if they like a movie for multiple reasons? What if they like the stars, but like the plot more? Shouldn't you capture that? Ideally, sure, but people hate giving feedback, so you can't.

Re:I'd say... (2, Informative)

flynt (248848) | more than 5 years ago | (#21349859)

From the Netflix Prize FAQ, they say how they currently do it:

"Straightforward statistical linear models with a lot of data conditioning."

The Netflix programmers shouldn't necessarily get special recognition for using least-squares modeling, but feel free to pass on your praise to Gauss, Legendre, Galton, and Fisher.

What's amazing is how hard it is to improve drastically on these 150-year-old statistical techniques.

Re:I'd say... (1)

Otter (3800) | more than 5 years ago | (#21351051)

"Straightforward statistical linear models with a lot of data conditioning."

There's a whole lot of devil in those details, though.

Re:I'd say... (1)

stranger_to_himself (1132241) | more than 5 years ago | (#21351179)

There's a whole lot of devil in those details, though.

Absolutely. With the size, sparsity, self-censoring and ordinal nature of their dataset, a 'straightforward statistical linear model' would not get very far.

It's a scam. (1)

Poromenos1 (830658) | more than 5 years ago | (#21350875)

Netflix's system is already 90.3% accurate!

Re:I'd say... (1)

krazo (220290) | more than 5 years ago | (#21350921)

It seems to me that it must be hard to optimize because of the 5 star system.

90% of the movies I rate are either 3 or 4 stars. I already pre-filter so I don't watch movies that would get a 2 or 1 often and 5s are hard to fine. Trying to differentiate the emotions generating "meh" and "yeah" is gonna be tough. I don't know if most people rate similarly but I imagine they do.

A 10 star system would add more data points and might be better. But a simple system with multiple axis would be a lot better, I bet. Something as simple as good vs. enjoyable would do a lot to differentiate my personal preferences (action flick might be 2 stars good and 5 enjoyable for an overall 4 where a drama might be 5 stars good vs. 3 stars enjoyable for an overall 4.) The problem with that is you would probably receive less data with a more complex system.

I haven't looked at the data, but I can see why it would be hard to optimize a learning algorithm around a data set with 5 possible outcomes clustered heavily around two and an almost infinite number of discreet features on each item. It might seem like the set of features would allow for a huge diversity of solutions but I really think the limited feedback might be the problem.

Maybe someone who actually worked on this could comment.

Re:I'd say... (1)

|Cozmo| (20603) | more than 5 years ago | (#21352215)

5 stars is certainly not enough for me either. Most movies end up with 4 stars for me, but I'd rather have the option of 6-8 stars, since there is certainly a lot of wiggle room in a 4.

Re:I'd say... (1)

Gr8Apes (679165) | more than 5 years ago | (#21353069)

Actually, most movies should fall into the '3' category, with those you watched but didn't like as 2's, and those you couldn't watch as 1's (even some technical 2's might become 1's). Anything you have no interest in should be tagged as such. 4's are for those you really liked, and 5 for outstanding movies which in most cases you'd watch several times and possibly buy. (Sixth Sense was a 5 but is a one time only movie unless you're interested in dissecting it. There's too many movies in my queue for that;)

This is why... (1)

C10H14N2 (640033) | more than 5 years ago | (#21352605)


When I did use NetFlix, I spent a good amount of time flagging as many movies I did NOT want and would never, ever rent as those I did or would. The result was a pretty consistent selection that reasonably matched my taste.

Aww, I lost... (1)

SailorSpork (1080153) | more than 5 years ago | (#21349297)

I guess my "flip a coin, take a chance" method wasn't worth the $million. Back to the drawing board...

Moving target? (4, Interesting)

ktappe (747125) | more than 5 years ago | (#21349319)

Will Netflix incorporate the near-winners' ideas into their current system? If so, won't future teams be aiming at a moving (improving) target? If not, won't current Netflix customers know that their recommendations could be better if Netflix just incorporated a now publicly-disclosed algorithm into their servers?

Re:Moving target? (2, Interesting)

Ngarrang (1023425) | more than 5 years ago | (#21349389)

I had the same thought. And to what extent is the accuracy of a suggestion system important? Sometimes, throwing in a completely different suggestion might garner you a rental and possibly more rentals because you might like other movies of that type.

Trust (1)

illegalcortex (1007791) | more than 5 years ago | (#21349499)

Accuracy in this contest is defined as the user rating highly the movies that the system would suggest to them. The whole point of it is trust. If you're throwing out lots of suggestions that the user doesn't like just to try to find one they might like, you're destroying their trust in the system. They won't bother even reading the recommendations if they know they're filled with garbage.

Re:Moving target? (1)

JPriest (547211) | more than 5 years ago | (#21352871)

Agreed, I like many movies across a spectrum of genres. Just because I liked Saw does not mean I want to see a bunch of bad horror movies more than a good comedy.

Re:Moving target? (1)

evilviper (135110) | more than 5 years ago | (#21354737)

Sometimes, throwing in a completely different suggestion might garner you a rental and possibly more rentals because you might like other movies of that type.

It's very, very important. If it isn't highly accurate, you're just going to completely ignore what it suggests, and get no benefits from it.

And your analogy is extremely flawed. If it's a movie you would like, then it SHOULD be recommended. That's what the system is there for. The odds that recommending a random movie to someone will inadvertently expose them to something wonderful that they wouldn't have seen otherwise, is about as likely as someone winning the lottery... There's millions of movies and TV shows out there, and most of them are crap.

Re:Moving target? (4, Informative)

illegalcortex (1007791) | more than 5 years ago | (#21349469)

It's not a moving target. It's a very fixed number (RMSE = 0.8563) that the winning algorithm has to come up with. The netflix algorithm never gets re-run on the data for the prize.

Netflix is free to merge any improvements into their actual system in the meantime.

Re:Moving target? (1)

MrBeau (1009661) | more than 5 years ago | (#21349739)

No. The rules state: To qualify for the $1,000,000 Grand Prize, the accuracy of your submitted predictions on the qualifying set must be at least 10% better than the accuracy Cinematch can achieve on the same training data set at the start of the Contest. The official contest site can be found on http://www.netflixprize.com/ [netflixprize.com]

Bad title (4, Funny)

markov_chain (202465) | more than 5 years ago | (#21349433)

The prize was clearly a million dollars, not a cigar! I guess the editors don't even read the summary.

Re:Bad title (2, Informative)

dintech (998802) | more than 5 years ago | (#21349497)

I don't think I'm mistaken in saying that $50,000 buys quite a few cigars too.

Re:Bad title (1)

CopaceticOpus (965603) | more than 5 years ago | (#21349595)

Explain.

Re:Bad title (5, Funny)

shadow349 (1034412) | more than 5 years ago | (#21350159)

Money can be exchanged for goods and services.

Re:Bad title (1)

dintech (998802) | more than 5 years ago | (#21350797)

Explain

Marijuana affects the memory.

Re:Bad title (1)

bcattwoo (737354) | more than 5 years ago | (#21354497)

Woo-hoo!

Re:Bad title (1)

StarfishOne (756076) | more than 5 years ago | (#21349519)

Perhaps the summary was not recommended to them, given their personal preferences.

Re:Bad title (1)

Panitz (1102427) | more than 5 years ago | (#21349523)

They didn't win a million dollars, or a cigar... so really it doesn't matter.

Far better option... (0)

Panitz (1102427) | more than 5 years ago | (#21349473)

IMDB http://www.imdb.com/ [imdb.com] has a recommendation section. Pick a film you like and it gives you a recommendation from their database. Feom experience this works perfectly for me... Plus you can read extensive reviews of films if you like.

Seems netflix are spending a lot of money on something that they seem to have working ok and can be found on other websites anyway.

Huh? (1)

illegalcortex (1007791) | more than 5 years ago | (#21349531)

You do know netflix already HAS a recommendation engine, right? Supposedly a very good one. The whole point of the contest was to significantly improve that engine.

Re:Huh? (1)

Panitz (1102427) | more than 5 years ago | (#21349567)

"Seems netflix are spending a lot of money on something that they seem to have working ok"

Well I think that shows that I know they have one! I was merely saying that there are better ones in my opinion, and it will be very hard for netflix to match them.

Re:Huh? (0)

Anonymous Coward | more than 5 years ago | (#21350927)

If IMDB's system was actually better, there's a good chance they would have entered the contest themselves and won.

Re:Far better option... (0)

Anonymous Coward | more than 5 years ago | (#21351245)

Something tells me you aren't the sharpest knife in the drawer

no breaktrough - just blending (5, Informative)

hashmap (613482) | more than 5 years ago | (#21349529)

Most noteworthy aspect of the winning entry is that their winning method works by combining 107 different types of prediction strategies.

They state that you can get pretty far by blending the 3-4 best strategies, but of course doing so would not have netted them the progress prize

It is kind of sad realization that there actually is no better method. Your best bet is to use brute force and attempt to find some weighting methodology that combines known methods. By the way this is a well known issue in protein structure prediction competitions, for many years now so called meta-servers (predictions work by merely combining other predictions) win all the time. The joke is that we now need meta-meta-servers, combine the results of combiners

Also a clarification on the progress prize: to get it you need to have at least 1% improvement over the previous result. Considering that there is only 1.57% to go there is room for only one more progress prize until it hits the Grand Prize (10% improvement over the original results).

Re:no breaktrough - just blending (1)

MrBeau (1009661) | more than 5 years ago | (#21349873)

Also a clarification on the progress prize: to get it you need to have at least 1% improvement over the previous result. Considering that there is only 1.57% to go there is room for only one more progress prize until it hits the Grand Prize (10% improvement over the original results).
Where did you get that? The rules (http://www.netflixprize.com/ [netflixprize.com] ) state:
To qualify for a year's $50,000 Progress Prize the accuracy of any of your submitted predictions that year must be less than or equal to the accuracy value established by the judges the preceding year.
You just have to be better.

Re:no breaktrough - just blending (1)

hashmap (613482) | more than 5 years ago | (#21351475)

from: http://www.netflixprize.com//community/viewtopic.php?id=799 [netflixprize.com] We have also updated the Prize leaderboard to reflect the award of the 2007 Progress Prize and have established the new accuracy requirement to qualify for the 2008 Progress Prize. Again, in accord with the Rules, the new Prize level reflects a 1% improvement over team KorBell's verified submission, requiring a 9.34% improvement over the original Cinematch accuracy level.

hmmm .... (4, Funny)

Average_Joe_Sixpack (534373) | more than 5 years ago | (#21349543)

if ($director eq "Michael Bay") {
        print "Not recommended";
}

That should improve the system by at least 20%

like trying to win the lottery (1, Insightful)

bzipitidoo (647217) | more than 5 years ago | (#21349597)

I'm skeptical about these sorts of prizes. The X prize, Top Coder, Clay Institute Millennium Prizes-- if those were the only reasons to do something, few would. Seems pretty risky to do a lot of work for what amounts to a lottery ticket. So, who got 2nd place, and how well did they do? 1 group wins a paltry $50K and a little publicity and recognition, maybe even an endorsement or two, and the other 27000 plus get what? Nothing much. It's cool and fun to work on such problems, but people have bills to pay. Nice to have the sort of job where one gets paid to work on stuff like that. Any contestants reading this? Maybe you could enlighten the rest of us on why you bothered competing?

As for Netflix, I wonder how much such an improvement is worth to them? More than $50K, I imagine. Pardon my cynicism, but seems like contests like this are a way to get a lot of ideas and work for very little money.

The thrill of victory (1)

illegalcortex (1007791) | more than 5 years ago | (#21349659)

I can say I played with it because I found it fun. I'm a coder, it's what my brain is interested in. There have been contests for ages simply because human beings like to compete, even if second place gets nothing.

And FYI, netflix doesn't get any "ideas" from anyone but the winner. You only have to submit your code if you win.

Re:like trying to win the lottery (1)

Loke the Dog (1054294) | more than 5 years ago | (#21349927)

Well, I'll tell you something. Most criminals suck at judging risks. They simply tend to forget that robbing a bank is very likely to put them in jail. On the other hand, some people are afraid of speeding even though they're driving on a road that is extremely unlikely to be patrolled by cops. My point is that different people have completely views on risks and I think it's extremely rare that people actually back up their actions with maths and statistics, instead its all about emotions.

I think its good that these competitions are popping up. Those who like risks get a chance to make an honest living instead of opting for gambling or crime. If you don't wanna play, that's fine.

Re:like trying to win the lottery (1)

absoluteflatness (913952) | more than 5 years ago | (#21351573)

Except that the only risk here is wasting your time...

Re:like trying to win the lottery (5, Insightful)

MBCook (132727) | more than 5 years ago | (#21350003)

Two reasons I can think of. One is the challenge. I like to code but I'm not great with coming up with projects to do myself. This kind of thing would be nice for that.

The other is the experience. If you get second in this, no, you won't win the prize. But you can bet that having that on your resume would make getting many jobs much easier. Amazon would like your skills. So would many other retailers.

Also, as a side note, it's not a lottery. There is a three prong legal test in the US to determine if something is a lottery. I think the three parts are you have to pay to get it, everyone has an equal chance of winning, and there is no skill involved. I'm not positive about the second part. This is free to enter and is based quite a bit on skill, so it's not like a lottery.

Don't exaggerate.

This isn't a way to get free work. It's a way to get very smart job candidates to find you. It's a recruiting tool. You don't honestly think that they will take the winning idea, pay the $1m, and then just say "bye" do you? They will offer that person a job if at all reasonable (if it's a team of 500 students, obviously they couldn't).

Re:like trying to win the lottery (1)

marcop (205587) | more than 5 years ago | (#21351923)

The three legal tests are:

1) Prize
2) Chance
3) Consideration - you pay something to enter.

Eliminating item 3 is typically how sweepstakes are made legal.

Re:like trying to win the lottery (1)

IceFox (18179) | more than 5 years ago | (#21353351)

Haha you are thinking way to slow. Most everyone on the top list has already being contacted for jobs, not be Netflix, but all sorts of other companies.

Re:like trying to win the lottery (0)

Anonymous Coward | more than 5 years ago | (#21350061)

Any contestants reading this? Maybe you could enlighten the rest of us on why you bothered competing?

I have been thinking alot about the problem, though I haven't submitted anything yet. My reason was that I thought I would actually have a good chance of winning. I figured there ought to be one best solution, which would be obvious, once you've found it, and the others didn't seem to have found it. Lucky me! It appears now, though, that there perhaps is no obviously best solution (although I still find that a bit wierd). Weighting together a bunch of suboptimal solutions, hardly knowing why, is certainly no fun.

Re:like trying to win the lottery (1)

teslatug (543527) | more than 5 years ago | (#21350727)

That's true, but the prize is just icing to most (all?) of these groups. Many will spend much more than the prize to get it, and everyone involved knows this and still goes through with it. Sometimes it's enough to do it to advance the technology. You can consider it a prize that millions of people will make use of what you produced. Also don't underestimate the fun factor. It's a big drive for what people do. It's a cliche but money isn't everything. Also, research groups could be working on this, while getting paid to do that research by another party.

Re:like trying to win the lottery (4, Insightful)

AdamTrace (255409) | more than 5 years ago | (#21350837)

"Any contestants reading this? Maybe you could enlighten the rest of us on why you bothered competing?"

There are two immediate reasons I can think of why anyone would bother competing:

1) To win money.
2) Because they enjoy the challenge of trying to solve an interesting problem.

I'm just a simple coder, and knew that I didn't have any realistic chance of winning money. But I still found it very satisfying to try to come up with a solution and send it in and see how I did. I don't regret spending hours of my own leisure time on the project.

That said, eventually I gave it up. It was very clear that I'm not smart enough to meet the challenge. I had my fun, and it was time to move on to the next project. In summary, I don't think it's safe to assume that everyone is in it for the money.

"Pardon my cynicism, but seems like contests like this are a way to get a lot of ideas and work for very little money."

I call it "brilliant". Netflix probably put some pricetag on what it would pay to get >10% improvement on their system. That pricetag is probably more than $1 million. That means profit!

Re:like trying to win the lottery (1)

ucblockhead (63650) | more than 5 years ago | (#21351437)

There's a third reason: reputation. Being able to say "won the $FOO Prize" is probably worth lots more in terms of future employment than the actual prize.

Re:like trying to win the lottery (0)

Anonymous Coward | more than 5 years ago | (#21353367)

None of the top teams entered the contest to win the money. The money is nice, don't get me wrong, but it's not the main motivation. The main motivation is the the data set. 2.8 million queries, with relevance judgements. (e.g. "This was good. Give me more." "This was bad. Give me less") It's a great data set for information retrieval researchers, which is exactly what all the top teams are.

We know roughly what the Bellkore team is doing [att.com] . And we know roughly what all the top teams are doing. Simon Funk uses Singular Value Decomposition. [sifter.org] Yi Zhang is using Bayesian hierarchical models and Expectation-Maximization [ucsc.edu] All the top teams know what everyone is doing. It's not a secret. They publish what they're doing.

partial credit? (1)

sootman (158191) | more than 5 years ago | (#21349933)

They should give AT&T $843,000.

Man, that's too easy... (1)

clickety6 (141178) | more than 5 years ago | (#21350189)


some

if age 18 and male then hard porn and south park
if age > 18 and male and lives at home then any sci-fi movie (plus points if it's a sequel)
if age > 18 and female then any movie with Princess in the title

100% match up !

Re:Man, that's too easy... (1)

rossz (67331) | more than 5 years ago | (#21353589)

if age > 40 and female and single, suggest movie showing what pig men are but true love will eventually be found
if age > 40 and male and single, porn

The winning solution... (0, Flamebait)

krazytekn0 (1069802) | more than 5 years ago | (#21350373)

will be able to factor out movies people think that they *should* rate highly from those that they actually like. You all know what I mean, your movie snob friend tells you to watch the latest film festival gay cowboys eating pudding movie and you feel like you have to give it at least 3 stars because otherwise netflix would know you just want to watch stuff get blown up.

You know who you are netflix fakies!

i thought this had already concluded (1)

putch (469506) | more than 5 years ago | (#21350623)

that's what i get for listening to, uh, slashdot: http://slashdot.org/article.pl?sid=06/10/09/1344235 [slashdot.org]

only yourself to blame (1)

illegalcortex (1007791) | more than 5 years ago | (#21352175)

That's what you get for lack of reading comprehension, including familiarizing yourself with what the contest actually is. The contest wasn't to beat Netflix's algorithm. The contest was to beat it by 10%. Nothing in the summary of the original article was incorrect.

Re:only yourself to blame (1)

putch (469506) | more than 5 years ago | (#21353435)

surely i can be castigated for not having rtfa. however, given the fact the original article (http://developers.slashdot.org/article.pl?sid=06/10/02/1359221&tid=97) was seven days prior to the "beaten" story and neither of the writeups in any way mention the 10% threshold I dont feel like my confusion was particular unreasonable. asshole.

My Netflix Experience (1)

HomeLights (1097581) | more than 5 years ago | (#21351199)

Here's the problem I had...I was picking comedy movies - any and all. Netflix recommended Horror and Action flicks. That is 0% correct. Let's see a 10% improvement on 0% accuracy is...0! Sounds right. Netflix NEVER even came close to trying to get it right. It never suggested anything in the genre I was getting my movies from. AND to make it worse, when I would check the "not interested" box by the Horror/Action flicks, they would still show up as recommendations. Come on NetFLix - at least don't show me the ones I told you not to show me.

Yes, please improve it. (1)

Lord Apathy (584315) | more than 5 years ago | (#21351797)

I, for one, have really never found Netflix's recommendations all that useful. It sometimes recommends movies that I've already Netflixed. But to be fair I think they fixed that. It has recommended movies that I already have in my queue but most of the time it will be movies that I have no interest in at all. Then there was the time I turned in a 'G' rated movie, Disney I think, and it recommened ether Saw I or Saw II.

Not really sure where it got that one from. Nothing I had turned in that week had anything remotly to do with the Saw movies. I've even looked at my past rentals and can't find anything I think would tie me to those movies. I'm out of my blood and gore phase years ago.

Maybe it just thought my taste in movies sucked. That has to be it.

Re:Yes, please improve it. (1)

uniquename72 (1169497) | more than 5 years ago | (#21352475)

I get some decent recommendations and occasionally accept one or two, but they could improve their relevance 100% just by not recommending titles I've already gotten from Netflix.

This should be an obvious fix -- where's my million?

Throwing away too much information (1)

jfengel (409917) | more than 5 years ago | (#21351815)

They're looking in the wrong places, and trying to squeeze blood from a stone.

User ratings are a deeply flawed way of getting this information. They're one-dimensional and prone to serious randomizations based on the user's mood; a 5 today might have been a 3 tomorrow. Since most of the movies that a user rates will be between 3 and 5 (it's just not that hard to spot a movie you're going to hate, so why would you rent it in the first place?) that makes the data... well, not valueless, but containing a lot less truth than you'd like.

Netflix has a huge amount of additional data that they're not using:
* What did the user look at?
* What did the user rent?
* How did they order their queue?
* How long did the user keep the film?
* When did the user add additional films that can be considered "related"?
* What did the user mark "not interested" (not included in the data set, IIRC).

If they want better recommendations, it's time to stop looking for the quarter under the lamppost and broaden their horizons. You probably can't anonymize all that data well enough to let the world compete for it, but if their internal developers with all that data can't beat outsiders with less, they need to hire some new researchers.

Sometimes throwing stuff away is good (2, Interesting)

illegalcortex (1007791) | more than 5 years ago | (#21352255)

I think you have to consider that netflix is working off a very large user base with a very large list of titles. In this sense, computation time is going to go way up the more you keep adding all these factors. I'm sure they've had projects internal to netflix to use more data, but found that it just didn't pay off with the increased computation time. It's much better to get good recommendations onto the page instantly than make the user wait 2 seconds for great recommendations. The same is possibly true for doing recommendations ahead of time and having to spend the extra compute time and storage space.

Plus, I think there's always going to be some level of "noise" in the system. People rating things incorrectly (clicked on the wrong number), people changing their minds, etc. And then there's the cases where it makes no logical sense that if I liked movie A, B and C that I should hate movie D. The question is, how good can a recommendation system get when it will always be thrown off by the noise.

So while I agree with you in theory, I think it may not work out to be such a great thing in practice.

Re:Throwing away too much information (1)

northcove (890000) | more than 5 years ago | (#21353101)

You seem to have it figured out (even though you obviously haven't read the competetion rules and what information netflix uses for their current technique. So go give it a shot then, wiseass.

Re:Throwing away too much information (1)

jfengel (409917) | more than 5 years ago | (#21353565)

What makes you think I haven't read the rules? I did read them, and downloaded the data, and decided not to participate because I didn't think that the existing algorithms could be usefully improved upon. The results seem to have borne me out.

Perhaps I should have phrased some of this better: what ELSE did they look at and decide against? What did they rent and change their minds about? (That's the second one and I really should have proof-read that better.)

I said in the original thread announcing the competition that I'd rather incorporate more data than trying to get a small improvement (they're only looking for 10%). The contest shows that they haven't missed something obvious, which is useful, but if they're actually after better results rather than validation that they've done an OK job by themselves they'd be better off using data that they have but are currently not included.

They're not interested in "truth" necessarily (1)

blueZ3 (744446) | more than 5 years ago | (#21353649)

They just want to access the "truthiness" of recommendations :-)

Re:Throwing away too much information (1)

Yogs (592322) | more than 5 years ago | (#21354387)

Let's look at that additional data a little closer.
Some of it might be useful, but a lot of it seems like noise.

* What did the user look at? - Noise, looking is way too weak an endorsement, and if not queued, it's probably not an endorsement at all

* What did the user rent? - Slightly more useful, but because the user wasn't explicit about their feelings (and most users don't rate a lot of movies), it's hard to come up with a convincingly reasonable way to rate them by default.

* How did they order their queue? - Sorry, this is noise, unless you believe newer movies are inferior to older ones. Maybe, maybe if you track difference between availability date and queue date you have something but that breaks badly on the classics.

* How long did the user keep the film? - This is pure noise... depends on hours to burn watching movies more than anything else.

* When did the user add additional films that can be considered "related"? - Somewhat useful, but high degree of randomization around that because surfing habits depend on mood and time available to kill.

* What did the user mark "not interested" (not included in the data set, IIRC) - As before.

A lot of getting to an actual reasonable solution to a problem in a reasonable amount of time involves the willingness to throw away information so you can simplify your analysis. Not saying I'm ready to throw it all away right off the bat, but I'd throw away a lot of it, and more if it didn't look particularly predictive under initial analysis.

Here's an idea... (3, Insightful)

the JoshMeister (742476) | more than 5 years ago | (#21353957)

Why not give users more control over their recommendations? Heck, even a bunch of checkboxes would be useful.

For example, Netflix frequently recommends rated R movies to my family, but we have never rented a single R-rated movie and have no desire to do so. Moreover, every time we get a recommendation for an R-rated movie, we rate it "Not Interested." I've probably marked dozens of R-rated movies "Not Interested," but they continue to be recommended. (Either Netflix is trying to tell me to just give in and rent one already, or they really don't understand my family's movie preferences.)

A simple checkbox for "Do not recommend R-rated movies" would be all Netflix needs to substantially improve its accuracy for my family. I imagine Netflix could add checkboxes for similar criteria as well. In any case, I think a key point is giving more control over recommendations to the users themselves.

Buried gem (2, Interesting)

vrmlguy (120854) | more than 5 years ago | (#21353985)

The most interesting part of the research paper was this: "More specifically, if movie i was rated x days later than movie j, we multiply their similarity by exp(-x/600). The denominator 600 (days) was determined by cross validation, and reflects the fact that after two years, similarity decays by approximately a factor of 3." Apparently Joe Average's tastes in movies slowly evolve over time, and something you liked three years ago may not be that attractive today.

This raises the question, should someone's age affect the denominator? People in or just out of college generally see their tastes evolve quickly, while people in retirement homes might take decades to get tired of something.

I also wonder if this decay factor applies to other fields. Not just books or music, but toothpaste or politicians. In the US, your representative is presumably re-elected before your opinion has time to change much; the president just as you're getting tired of him. It makes me wonder how Senators get re-elected at all.

Theoretical Limit? (1)

z-j-y (1056250) | more than 5 years ago | (#21354535)

How do we know whether it is even possible, theoretically, to improve it by 10%?

1. inherent randomness in each individual's ratings

if you give me a list of movies today to rate them, then the same list a week later, I probably would give inconsistent scores. the more randomness in it, the less predictable it is. hack, Netflix could have deliberately introduced some randomeness in it so that nobody could ever get the prize.

2. sample size

imagine there is a underlying theoretical model that drives us to rate the way we do. that model would have a gazillion number of parameters. even if we have a sample size of a million users each rating a thousand movies, it is unclear that it is big enough.
Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...