Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

'Scrapers' Dig Deep For Data On Web

CmdrTaco posted more than 3 years ago | from the need-a-bandaid dept.

Media 158

srwellman writes "The practice of Web 'scraping' is growing as many firms offer to collect personal, and potentially incriminating, data about users from their social networking profiles and discussions. Many companies even collect online conversations and personal details from social networks, job sites and forums where people might discuss their lives and even potentially sensitive data, such as health issues. These scrapers operate in a legal grey area leaving many users exposed." We ban scrapers like this regularly here simply for not adhering to the rules spelled out in robots.txt.

cancel ×

158 comments

Like Google? (4, Interesting)

bonch (38532) | more than 3 years ago | (#35809300)

Firms offer to harvest online conversations and collect personal details from social-networking sites, résumé sites and online forums where people might discuss their lives.

You mean like Google already does for its advertisers? In fact, one of the related links in the article is a story about Google titled Google Agonizes on Privacy as Ad World Vaults Ahead [wsj.com] , discussing their plans for utilizing their vast archive of valuable user data. The battle for online privacy was lost long ago.

Re:Like Google? (0)

blair1q (305137) | more than 3 years ago | (#35809432)

This is a new form of privacy of which the news has not come to Harvard.

I'm pretty sure information posted for the entire planet to read is not private.

Out on the street, a huckster can size you up in about ten seconds, with 90% accuracy. Online, in text, you're not wearing that tribal-armband tattoo, so it might take a few minutes to figure out you're a joiner with delusions of individuality.

Time to revise my motto: The Internet is not secure, and open forums are not private.

Re:Like Google? (1)

Anonymous Coward | more than 3 years ago | (#35809642)

> I'm pretty sure information posted for the entire planet to read is not private

Well, that's what I think too, but amazingly, about 98% of humanity doesn't seem to agree. It seems to me that they're insane if they expect something posted to the whole world to be private, but there are SO many who think that way, I'm not sure what to make of it.

Re:Like Google? (1)

locofungus (179280) | more than 3 years ago | (#35809914)

The majority of humanity probably think posting something to facebook or whatever is similar to writing "Got totally plastered on holiday" on the back of a postcard and posting it to their local (something that people do)

Sure, it's public but after a few years it will have vanished without trace.

Tim.

Re:Like Google? (2)

hoggoth (414195) | more than 3 years ago | (#35810548)

/ sheepishly pulls sleeve over tribal armband tattoo...

Re:Like Google? (0)

Anonymous Coward | more than 3 years ago | (#35812298)

Speaking from experience: That tattoo means nothing. And anyone good will realize that. I've surprised a lot of people when it became evident that I'm not a sheep to be herded with the rest of the crowd. Now yes, you did say 90% accuracy, but in some of the circles I've traveled 90% ain't good enough. It's the 10% that will literally kill you. Yes, it's been a rough life. But a good one.

Re:Like Google? (4, Insightful)

betterunixthanunix (980855) | more than 3 years ago | (#35809838)

The battle for online privacy was lost long ago.

Only because one side of the battle never bothered to fight. Nobody was forced to go to social networking websites and post their life story, anyone could encrypt their email and IM conversations, and ad blocking software is widely available. Large amounts of the information that these companies are aggregating could have been made far more difficult to obtain if the majority of computer users could have been bothered.

Sadly, the Internet has become more of an adversarial game than a way to unite people.

Re:Like Google? (2)

VolciMaster (821873) | more than 3 years ago | (#35811150)

The battle for online privacy was lost long ago.

Only because one side of the battle never bothered to fight. Nobody was forced to go to social networking websites and post their life story, anyone could encrypt their email and IM conversations, and ad blocking software is widely available. Large amounts of the information that these companies are aggregating could have been made far more difficult to obtain if the majority of computer users could have been bothered. Sadly, the Internet has become more of an adversarial game than a way to unite people.

forced to use social tools? no.

encryption available? yes

understood by anyone in the general public? nope

Nonsense (1)

Anonymous Coward | more than 3 years ago | (#35811152)

Its ridiculous to expect users to anticipate and thwart privacy invasions. These companies could be shut down overnight (or at least rendered illegal) with common-sense legislation. The problem is not users, it is their bought-and-paid-for "representative" government(s) which sell out their constituents to be deceived and abused by sleazy industries.

Re:Like Google? (1)

Americium (1343605) | more than 3 years ago | (#35811180)

The battle for online privacy was lost long ago.

So if I post to a public forum I should expect privacy?

What about CC companies selling data, that was going on before the internet, and seems more intrusive than many of these situations.

Sadly, the Internet has become more of an adversarial game than a way to unite people.

I think all those countries having revolutions in the middle east might disagree with you.

Re:Like Google? (2)

jd (1658) | more than 3 years ago | (#35811408)

There's that and there's the fact that the US (one of the largest consumers of data) has no data privacy laws and has been pressuring places that do (such as the EU) to violate their own laws. The laws don't solve the problem in and of themselves, what they do is make the public more* aware that the problem even exists. (*You can have more than nothing.)

The older ITAR laws and RSA patents didn't help - it effectively criminalized any effort to produce a product, since you'd need to sell the product in the US to be able to generate enough interest.

The problem now is that the legacy protocols are too widely used to be easily replaced and legacy products have so much staying power that a backwards-compatible system would remain effectively insecure for decades.

Screen Scraping's been GOOD to me (& mine) (0)

Anonymous Coward | more than 3 years ago | (#35811942)

I've been doing that for years now, & for a good purpose: To populate a custom HOSTS file with data to block out known bad sites & servers + bad hosts-domains from 7-8 very reputable sources for said material.

I do this as an added layered security approach to things online for myself, my family, & friends... it works for faster & safer online experiences.

(Simply because this file covers ALL webbound apps you have, & it's run @ the TCP/IP stack level in kernel mode (acting merely as a filter for that system to utilze)).

I've just built/co-built/rebuilt a system for that that is better in many ways than its predecessors, in fact...

(3rd one now, 1st was in Borland Delphi Object Pascal, 2nd was Ms-Access SQL (for normalization portion only though), & lately it's Python with REGEX work)...

In fact, that system's running on multiple threads in timings as I type this, and even when I sleep...

(Which takes a burden of 20 minutes work away from me I used to have to do in the a.m. or evenings, before... now? Now I don't HAVE to, anymore, lol, YEA!)

That "all said & aside":

This type of system's NOT THAT TOUGH TO BUILD, not really, because tools like PyThon, Perl, & even std. *NIX shell commands can be "popped together" with some regex work to do so... pretty damned easily too, once you have the "base toolkit" & process in place for it, & really for ANY KIND OF DATA ONLINE!

Too bad folks are using it to potentially & perhaps more than potentially bogus purposes vs. one another... this is human nature I suppose, the beat side (unless the person's a known killer & such, then I'd think it was fair to warn others perhaps... there's always "shades of gray" in any situation, & I don't like "absolutes")

I guess what I am trying to say here though, is this:

Not ALL "Screen-Scraping" going on, is bad...

(My reasons for that, are from what I consider the most insightful portion of your reply below (really well said man)):

---

"Sadly, the Internet has become more of an adversarial game than a way to unite people." - by betterunixthanunix (980855) on Wednesday April 13, @01:16PM (#35809838)

Sometimes, it seems that way, doesn't it? Especially with articles like this one... really, Really, REALLY "well-put" on your part though!

However - Again/Lastly in closing: Not all of the "screen-scraping" stuff online is for "nefarious purposes", sometimes, it's for the general good of others too (per my reason for doing it myself, noted above @ the top of my reply to you).

APK

P.S.=> Didn't mean to "ramble", or go into "too much detail" (because with a "handle/username" like yours, you probably KNOW the detail I am guessing here... the detail was more for those that don't know this stuff) - I really liked how you closed you post though, made me think a bit is all... apk

the darker side of grey (0)

Anonymous Coward | more than 3 years ago | (#35809354)

Is it legal to USE information gathered in this way to discriminate against someone if it was gathered with methods contrary to a site's TOS?

Re:the darker side of grey (2)

Loether (769074) | more than 3 years ago | (#35810072)

I think they are 2 distinct issues that do not combine the way you suggest.

1. If you violate a websites TOS the website can come after you.

2. The info they gain spidering a website is pretty much free for them to use to discriminate against you.

Anything I post on slashdot/FB/any online forum I treat like it is viewable by every future and past employer, insurer, lender, ex girlfriend etc. Anything online will exist forever and if it's not already permanently linked to you, it will be before you die. If that's right or wrong, legal or illegal is really besides the point IMHO.

Re:the darker side of grey (2)

Americium (1343605) | more than 3 years ago | (#35811210)

I don't know how good of a comparison this is.

So if I write a book, can I include TOS that makes it illegal for anyone to use the information within the book? If I write a book about how much my boss sucks, and how I slack off at work, can I include TOS so that nobody is allowed to relay that information to him? Even if I only sell my book to members of a book club, I wouldn't think this changes anything.

If you intentionally post information about yourself on a widely viewable forum, I would expect other people might read it.

Re:the darker side of grey (2)

jd (1658) | more than 3 years ago | (#35811448)

Well, the problem with (1) is that a TOS is an agreement with no signature, no confirmation of acceptance (implicit is unlikely to hold up in court) and no proof that the TOS was even visible by the user (since what is visible to the user is a function of the browser and cannot be established at the server-side).

Re:the darker side of grey (0)

Anonymous Coward | more than 3 years ago | (#35810100)

The TOS violation and the use of the data aren't really related in the legal sense, really the only thing that matters is what you're discriminating on. If you're discriminating based on race, gender, ethnicity, etc. it really doesn't matter where you got the information, it's illegal either way. If you're not discriminating against a protected class (eg you don't hire someone because their facebook account is full of them getting shitfaced), the person you discriminated against has no basis for complaint. The site you scrapped the data from could complain that you violated their TOS, and the user of that site could complain to the site that their data was used in a manner contrary to their TOS, but the end user and the data scraper have no legally binding agreement between them.

They won't get me (2)

Tigger's Pet (130655) | more than 3 years ago | (#35809358)

I'm not on FB, Twitter, MyCloud or whatever else, so there's no data out there about me. If there's nothing to harvest then they can't harvest it - I'd rather be classified as 'boring' or 'not with it' (whatever the fuck 'It' is), than have stuff out there that might come back to bite me in the ass in 10 or 20 years time.

Re:They won't get me (2)

yog (19073) | more than 3 years ago | (#35809482)

Definitely avoid using a real or traceable name in online discussion forums and social sites. Also, avoid embedding your real name into your email address, such as "JohnSurfer@cox.net" or the like.

Unfortunately, my real name is embedded in one of my email addresses, and it's all over the web by now. I guess I can eventually switch to a different address, but the damage is done.

If you have someone's name, you can now obtain their current and past addresses, their age, their schools, possibly where they work, possibly their political party affiliation, and possibly a ton of other information if they have used their real name in online activities. It's not rocket science to do this; the information is just sitting out there waiting to be grabbed.

I suppose if you have nothing to hide and have avoided getting too controversial in your online discussions, or too outrageous in your social network photos and statuses, you're probably safe from major problems. Employers are going to be looking for extreme behavior, not slightly out of the ordinary behavior. If an employer doesn't like some minor thing about you, e.g. a picture of you on Facebook wearing green antennas at a Halloween party, then probably they're not someone you'd want to work for anyway.

Re:They won't get me (2, Funny)

Anonymous Coward | more than 3 years ago | (#35809688)

I suppose if you have nothing to hide and have avoided getting too controversial in your online discussions, or too outrageous in your social network photos and statuses, you're probably safe from major problems.

Yep. That's why my pic on chatroulette is an exact average size penis.

Re:They won't get me (0)

Anonymous Coward | more than 3 years ago | (#35809700)

The downside of that is that the info can be used against one come a political battle, or just a statement.

I will give a good example of that. When I graduated college in December 2008, I didn't have a Myspace/FB/Twitter account, nor did I care about possessing one. However, after repeated interviews where the interviewer asked my account name to friend/follow and I told them that I didn't have one (nor did I care to spill out my guts out for all and sundry on the Internet to see my private stuff), then told me that I was a fossil because I wasn't with the times (and compared to not having a FB ID with not having a telephone or E-mail address), I put up some token accounts with some sanitized stuff on them.

The idea of companies grabbing data is not new. I'm sure it will become more and more common as time goes on, and eventually a "desirability score" will be made of individuals, similar to a credit score as another factor for employers to screen on. Pretty much, the further one is personality wise from Snooki, the less chance of being able to find work.

Re:They won't get me (3, Insightful)

hoggoth (414195) | more than 3 years ago | (#35810616)

Wow, that's pretty inappropriate for an interviewer to require you to open your personal family or friends circle to him. What if my family is discussing my alcoholic father, my pregnant niece, my HIV+ friend, and my habit of killing interviewers and burying them in my backyard?

Re:They won't get me (1)

jshackney (99735) | more than 3 years ago | (#35809962)

Definitely avoid using a real or traceable name in online discussion forums and social sites. Also, avoid embedding your real name into your email address, such as "JohnSurfer@cox.net" or the like.

That's unlikely to help. I'm afraid this fight [randomwalker.info] is already [33bits.org] lost [schneier.com]

Re:They won't get me (3, Insightful)

sakti (16411) | more than 3 years ago | (#35810056)

IMO it's better to have an easy to find public 'you' online for these people to track. You use that for everything 'safe'. You then use multiple anonymous accounts for anything you don't want tracked.

If you have nothing tracking online I think it might start looking more suspicious than not. Plus having nothing might encourage 'them' to dig in and try to relate you to your anonymous account(s).

Re:They won't get me (2)

networkBoy (774728) | more than 3 years ago | (#35810822)

fundamentally that's what I do.
There is a real me on FB. Then there is me here (and this ID is shared across multiple sites) which would not be too hard to link to the real me.
For stuff I really don't want tied to me in re. job interviews, non-gov't background checks etc. I use other identities. For something that I would be afraid of coming out in a relatively thorough discovery && || government background check I simply don't post it on line. At all.
-nB

Re:They won't get me (0)

Anonymous Coward | more than 3 years ago | (#35810740)

"If an employer doesn't like some minor thing about you, e.g. a picture of you on Facebook wearing green antennas at a Halloween party, then probably they're not someone you'd want to work for anyway."

While true, having a job is essential, and it's a privileged few who can always pick and choose their employment so willy-nilly, especially in this economy.

Re:They won't get me (2)

SuricouRaven (1897204) | more than 3 years ago | (#35810916)

There are many applicants for each job, so employers can be picky. If they have a set of candidates who are all qualified and of similar levels of experience, they'll pick the one who is most 'normal' in their personal life, and thus least likely to somehow embarass the company or to just not get on with other employees.

Re:They won't get me (0)

Anonymous Coward | more than 3 years ago | (#35812444)

It also helps to post as Anonymous Coward.

Re:They won't get me (2, Funny)

Anonymous Coward | more than 3 years ago | (#35809938)

That's OK, Phillip Wilkerson of Midland, MI. We still know all about you. Tell Donna and the kids hi for us. Don't forget to pick up dog food on your way home from the tanning salon.

Sincerely,

Google

REGEX + Python or PERL could collect data on you (-1)

Anonymous Coward | more than 3 years ago | (#35810108)

Right here, off of /., & rather easily man... it all depends on how the website's pages are entry/output fields to HTML are formatted... this is where the REGEX work comes into play, and yes, it does work.

I know that tools can be built rather QUICKLY in Python, Perl, or even UNIX shell scripts can do it, fairly "easily" (some grunt work in testing is the worst part, & it's NOT that bad)...

After all, those tools? They're BUILT for text manipulation/conversion is why, & REGEX completes that/is the "linchpin" of it.

E.G./I.E.-> You have a "posts history" that's easily accessible and just a bit more work to walk into every post you do, & then, simply "collect them up" because they're in a "std. format" (your reply posts) which means they have a starting point, & an ending point as boundaries, which is what programs like this largely work on.

(Again, via REGEX (regular expressions) work. PERL or PYTHON are extremely good @ this in fact, & fairly simple to build... also, to a good degree, even *NIX shell scripting is as well @ these kinds of tasks (as UNIX was initially built with string processing in mind @ AT&T/Bell Labs)).

Also - I do it everyday myself, but NOT on individuals!

I go about collecting up data this way for populating a HOSTS file from various producers of that data (this is for a good thing though, system protection from malware online), & it runs continuously off of timers (building a scheduler based model right now as I write this in fact, just for kicks, to see if its superior to a threaded timer version))

I know how it's done, & for years now, so that said?

It's not a big "stretch of the imagination" to adapt something like the process I use, to data on this site (or really, any other, for anyone)... & IF I can do it?

You know that others certainly ARE DOING IT & perhaps NOT for "the right reasons" (which are imo @ least, for the common good as the right reasons).

No, perhaps those doing it already as I do are not necessarily only doing that to YOU only or targetting you, but the capability & potential for it's definitely there (& apparently per this article, also being used that way!)

APK

P.S.=> For "anonymity" though? I suppose the BEST thing you have going here, is the fact you post as a "handle/nickname" here... but that CAN be collected easily enough, here, or any other forums, using the example toolset I noted above that I use myself everyday/minute of the year for years now (but for a good purpose, not spying on others or "collecting potential dirt" on them)... apk

Re:REGEX + Python or PERL could collect data on yo (0)

Anonymous Coward | more than 3 years ago | (#35810874)

this is where the REGEX work comes into play, and yes, it does work.

This is slashdot. We know what regexp is. We don't need to have it capitalized or explained. We also know what perl and python are. Please stop capitalizing things. Also, while you're at it, please stop putting "random" things in "quotation marks."

And I'm sure you're capable of writing a crawler to go to a /. ID page and compile a list of all their posts. This is not a black magic art. Call us when you can do something special.

Speaking for "everyone @ slashdot", are you? (-1)

Anonymous Coward | more than 3 years ago | (#35811080)

Speaking for "everyone", eh? Big mistake:

"This is slashdot. We know what regexp is." - by Anonymous Coward on Wednesday April 13, @02:48PM (#35810874)

Ahem: The poster may not have been aware of it, so don't speak for others who may not be as "well informed" as yourself...

(Damn shame you aren't so well-informed that you don't realize that speaking for others can be a mistake!)

---

"We don't need to have it capitalized or explained. " - by Anonymous Coward on Wednesday April 13, @02:48PM (#35810874)

What is this "we" business? Speaking for others again when you really have NO IDEA whether the person whom I replied to knows about this stuff, or not... poor move.

---

"We also know what perl and python are." - by Anonymous Coward on Wednesday April 13, @02:48PM (#35810874)

Again - What is this "we" business? Speaking for others again are you, when you really have NO IDEA whether the person whom I replied to knows about this stuff, or not... Once more - a poor move.

---

"Please stop capitalizing things. Also, while you're at it, please stop putting "random" things in "quotation marks."" - by Anonymous Coward on Wednesday April 13, @02:48PM (#35810874)

Please stop giving others like myself orders... ok? I am not going to "obey you", so get over it... you're NOT IN CHARGE!

---

"And I'm sure you're capable of writing a crawler to go to a /. ID page and compile a list of all their posts. This is not a black magic art." - by Anonymous Coward on Wednesday April 13, @02:48PM (#35810874)

Nothing to it, like I said in my init. post you replied to... been doing it here for quite a while in fact, albeit for "the good"...

---

"Call us when you can do something special." - by Anonymous Coward on Wednesday April 13, @02:48PM (#35810874)

LMAO - tell YOU what: YOU CALL US, when you've EVER DONE ANYTHING OF NOTE @ ALL, WHATSOEVER, ok?

(I've "been there/done that" many times over the past 2 decades in this art & science of computing... have you? Doubt it!)

APK

Re:REGEX + Python or PERL could collect data on yo (0)

Anonymous Coward | more than 3 years ago | (#35812088)

Please don't respond to Mr One-Note Samba. As you can see, it just encourages him.

Mod him down if you've mod points and you feel so inclined; otherwise just ignore him.

Further trolling by the 'AC troll'? LMAO! apk (0)

Anonymous Coward | more than 3 years ago | (#35812414)

It's funny seeing an ac troll run from replying here http://news.slashdot.org/comments.pl?sid=2082332&cid=35811080 [slashdot.org]

Funny how his "speaking on behalf of /." for everyone seems to have been his undoing... lol!

(After all - he RAN from replying when I asked if HE was "all of /." (which we clearly know, he's not (or was it just you again, trolling as AC?)).

So, go ahead: Hit others with your registered 'LUSER" account effete, & useless 'down-mod points", instead of facing the music in the URL above.

(Thanks for proving a point here: That you're the TRULY "anonymous coward" here (and I stress, coward))

APK

P.S.=> Yes, I am assuming that you're probably just TomHudson doing this ac trolling of myself, as is per his usual, shown here quoted in his own words, no less:

http://slashdot.org/comments.pl?sid=1646272&cid=32150544 [slashdot.org]

Some people are pitiful... & there's no hiding from statements like that one shown in that URL above... apk

Re:They won't get me (0)

Anonymous Coward | more than 3 years ago | (#35810522)

What that means is that when they manage to tie your slashdot ID in with something else they'll put you down as a "mild tin foil hat" personality type.

Re:They won't get me (1)

TaoPhoenix (980487) | more than 3 years ago | (#35810774)

A real pro would be able to do it based on this comment of yours.

http://slashdot.org/comments.pl?sid=2031640&cid=35457796 [slashdot.org]

Re:They won't get me (1)

Tigger's Pet (130655) | more than 3 years ago | (#35812356)

Well done - you can track my previous postings on /. Do you want a prize? I'm now accepted as one of the 6.5 million people in the UK who have their DNA on record because this country stores DNA samples from everyone convicted (and many who are not convicted). Assuming of course that I'm not just posting things to try and make a point and gain Karma points - just like all the people on here who post about "My wife had this happen to her..." - we know that they haven't got a wife or they wouldn't be on here ;=P

Re:They won't get me (0)

Anonymous Coward | more than 3 years ago | (#35810888)

You probably don't like Social Networks. You hold a computer with Linux, you prefer Gnome to KDE and you think "stable, simple, clean, easy to use, easy to configure and gadgets" are important things. You encrypt some data. You think Google is maybe evil et caetera ... Just by reading your /.'s profile.

Re:They won't get me (1)

ceoyoyo (59147) | more than 3 years ago | (#35811266)

In eight years on Slashdot I wonder if you've ever accidentally posted something that might link to you. I can't be bothered to find out, but I'm sure that information might be valuable to someone.

Of course, you probably drag cookies around like everyone else anyway.

they may have already gotten you (1)

bityz (2011656) | more than 3 years ago | (#35811738)

Even though you never post a thing, someone else may post something about you. You may already be tagged in multiple photos on Facebook. You may have loan applications visible on the web. Your information is not entirely under your control - with pervasive digital storage, constant security challenges, and an increasing cultural trend to blurring the line between public and private, there is a growing chance that your information will leak out into the public.

They're coming for you, AC (2)

blair1q (305137) | more than 3 years ago | (#35809368)

That Anonymous Coward guy is going to have a mailbox full of goatse spam.

Re:They're coming for you, AC (1)

dev.null.matt (2020578) | more than 3 years ago | (#35809556)

That Anonymous Coward guy is going to have a mailbox full of goatse spam.

With the kinds of responses he's posted to some of my posts, let me assure you... he already does!

for great good luck! (0)

Anonymous Coward | more than 3 years ago | (#35809698)

Your offerings please Anonymous Coward, keep them coming!

Now lets see (2)

Grindalf (1089511) | more than 3 years ago | (#35809390)

Now what kind of individual stands to gain from the of generating this rumour? Lets see now ...

Bravo (2)

swanzilla (1458281) | more than 3 years ago | (#35809392)

Example 'scrape' FTA:

He used a pseudonym on the message boards, but his PatientsLikeMe profile linked to his blog, which contains his real name.

I don't think we need to dig any deeper to come to the conclusion that this guy is an idiot.

Re:Bravo (4, Funny)

TypoNAM (695420) | more than 3 years ago | (#35809716)

He used a pseudonym on the message boards, but his PatientsLikeMe profile linked to his blog, which contains his real name.

I don't think we need to dig any deeper to come to the conclusion that this guy is an idiot.

Indeed, Joseph Swanson.

Re:Bravo (1)

swanzilla (1458281) | more than 3 years ago | (#35809854)

Indeed, Joseph Swanson.

SEO on a budget. Take notes.

He's an Idiot with Plenty of Company (2)

RobotRunAmok (595286) | more than 3 years ago | (#35810236)

Slashdot is filled to the brim with people who take the time to create an alias and then list their homepage on their profile, which of course, is displayed in a link on the same line as their alias in the post they just made.

I click on those homepages whenever I read something really stupid or ridiculous or inflammatory or completely polar opposite my perspective. Which is to say, I click on them A LOT. I am amazed at how many of these "homepages" are links to commerce sites, or sites advertising some kind of service.

"Why," I inevitably ask myself, "would I ever buy anything from you, you knucklehead, you?"

It's like the guy who walks into a business meeting with a potential new client, someone he's never met before, wearing a big "I Love Obama!" button on his jacket. Or an equally large "Palin/Romney '12" button. Sure, you appreciate their passion -- maybe... if you agree with their POV -- but you immediately question their common sense, maturity, and business acumen.

"We (/.) ban scrapers..." LOL (2, Insightful)

billrp (1530055) | more than 3 years ago | (#35809518)

"We ban scrapers like this regularly here simply for not adhering to the rules spelled out in robots.txt." Hah! robots.txt doesn't stop any decent crawler

Re:"We (/.) ban scrapers..." LOL (1)

Anonymous Coward | more than 3 years ago | (#35809614)

Getting banned sure will though.

Re:"We (/.) ban scrapers..." LOL (0)

Anonymous Coward | more than 3 years ago | (#35810024)

Made me lol.

Re:"We (/.) ban scrapers..." LOL (0)

Anonymous Coward | more than 3 years ago | (#35809618)

Well, no. That's why they get banned. Also, I think your definition of "decent" is a little skewed in this context. The decent thing for any crawler to do is respect robot.txt files and the rules they contain. Of course, they would first have to look for a robot.txt file, which one would think any decent crawler would do.

Re:"We (/.) ban scrapers..." LOL (0)

Anonymous Coward | more than 3 years ago | (#35809682)

"We ban scrapers like this regularly here simply for not adhering to the rules spelled out in robots.txt."

Hah! robots.txt doesn't stop any decent crawler

Yes... not being stopped by robots.txt is the reason they ban them. Which implies that they're using some form of ban that does not rely on robots.txt (and which may or may not be effective).

Re:"We (/.) ban scrapers..." LOL (1)

billrp (1530055) | more than 3 years ago | (#35809736)

I don't think there can be such a "ban" - if humans can browse a website, then crawlers can crawl.

Re:"We (/.) ban scrapers..." LOL (2)

betterunixthanunix (980855) | more than 3 years ago | (#35809784)

However, there are patterns of browsing that are clearly not human. Humans do not make 100 requests in a 10 second timespan, nor do humans traverse every post made by every user.

Yes, it is imperfect and you might ban an occasional human, but this is essentially the situation we have with spam filtering. It is a bit sad that the Internet is becoming so adversarial, but that is what we face.

Re:"We (/.) ban scrapers..." LOL (2, Interesting)

Anonymous Coward | more than 3 years ago | (#35810764)

Humans do not make 100 requests in a 10 second timespan, nor do humans traverse every post made by every user..

That's what I use a Greasemonkey script for, you insensitive clod!

Re:"We (/.) ban scrapers..." LOL (2)

hoggoth (414195) | more than 3 years ago | (#35811040)

A smart discrete scraper will scrape breadth-first, ie: scrape 100 websites alternating the next page from each site in turn, instead of the next page on a single site until that site is finished. Some scraping on active sites like Slashdot or just Google's spidering is never done; It just continues on as new content is created. It would be easy for a scraper to act just like a human on Slashdot, just keep clicking 'refresh' every once in a while. An astro-turf post from GNA would really throw the admins off the trail.

Re:"We (/.) ban scrapers..." LOL (1)

CCarrot (1562079) | more than 3 years ago | (#35811318)

... nor do humans traverse every post made by every user.

...unless they have a fistful of mod points to spend...heck, sometimes I'm just very interested in a story and want to see what everyone has to say about it. True, that doesn't happen often, and I certainly don't read 10 posts a second, but it does happen...

Re:"We (/.) ban scrapers..." LOL (-1)

Anonymous Coward | more than 3 years ago | (#35810762)

You are stupider than your average nigger, aren't you?

Re:"We (/.) ban scrapers..." LOL (1)

Ares (5306) | more than 3 years ago | (#35810800)

iptables -a INPUT -j DROP $Bad_Scraper_IP_Address

Re:"We (/.) ban scrapers..." LOL (2)

Culture20 (968837) | more than 3 years ago | (#35810962)

mod_security [modsecurity.org] is pretty handy at spotting crawler patterns (you have to be a really weird human or a well designed crawler to look like something you're not).

Re:"We (/.) ban scrapers..." LOL (1)

TheSpoom (715771) | more than 3 years ago | (#35809738)

robots.txt isn't meant to have any enforcement capability; by its nature it's just an advisory mechanism telling bots who and what they will and will not accept. If a bot chooses to ignore it (as pretty much all of the types of bots described in this article do), it's up to the site admins to enforce it via IP bans etc.

Re:"We (/.) ban scrapers..." LOL (1)

mgcleveland (2029194) | more than 3 years ago | (#35810398)

I think the point they're making is that crawlers which do not obey the rules spelled out in robots.txt are blocked.

Re:"We (/.) ban scrapers..." LOL (2)

sharkey (16670) | more than 3 years ago | (#35811506)

Actually, it stops ALL "decent" crawlers. It's the ones that behave indecently that ignore robots.txt.

Re:"We (/.) ban scrapers..." LOL (0)

Anonymous Coward | more than 3 years ago | (#35812324)

They ban scrapers precisely because they don't follow robots.txt.

Anyone up for making a few new DNSBLs? (1)

mysidia (191772) | more than 3 years ago | (#35809542)

Known robots, and scrapers

IP addresses that do not honor /robots.txt.

and IP addresses that robotically submit spam on robots.txt disallowed HTML feedback feedback forms

Much web scraping can be automatically detected.

Sites like Facebook/social networking sites are perfect places to trap/detect scrapers, if they would be willing to contribute to a DNSBL

Re:Anyone up for making a few new DNSBLs? (1)

Rizimar (1986164) | more than 3 years ago | (#35810578)

A good place to begin would be to examine the robots.txt of large sites to see what they're blocking. Sometimes they leave helpful comments in the text files as well. The most interesting I've come across so far is Wikipedia's robots.txt file [wikipedia.org] which has comments for every disallow or series of disallows.

Re:Anyone up for making a few new DNSBLs? (1)

mysidia (191772) | more than 3 years ago | (#35811050)

The most interesting I've come across so far is Wikipedia's robots.txt file [wikipedia.org] which has comments for every disallow or series of disallows.

Well.. it bothers the hell out of me that I can't Google VfD/Afd/Page for deletion Articles on Wikipedia, because a few people were annoyed there were VfD articles about their nonnotable vanity page on WP. Wtf are the Wiki people thinking? Sometimes interesting points arise in a discussion, and it would be useful to be able to search those discussions in the future, since they're so massive.....

That's great for the user-agent fields of known bots. Unfortunately, it doesn't contain an IP address banlist. Something tells me they don't bother too much about IPs of bots that don't honor and use generic user agents.

I wonder if anyone's tried listing Firefox/MSIE in robots.txt Disallow entries... does that hurt any bots without impacting human navigation?

Future Politicians (1)

metlin (258108) | more than 3 years ago | (#35809588)

I've always wondered -- how would this work for future politicians from our generation?

All your comments, history etc are probably available in a multitude of places, and anyone with enough motivation can go around digging and find some pretty serious material. Combined with the fact that most people know (or care) little to nothing about privacy, you will have an entire generation of users with a good chunk of their private lives and opinions shared out on the Internet for everyone to see.

And knowing how we all have skeletons in our closets, and how we've all been immature at some point in time or the other in our lives, how many future politicians candidates can claim to be "squeaky clean"?

I mean, I see this primarily as a problem for the right more than the left, given how their voter base expects them to have "conservative values" or some such nonsense.

Re:Future Politicians (1)

dev.null.matt (2020578) | more than 3 years ago | (#35809668)

There's already pretty damning video clips of many US politicians that are widely available. It doesn't seem to have any real impact on their ability to get (re) elected here. Watching the Daily Show for a week, you will come up with numerous examples.

Unless of course you're referring to the effects these sorts of things might have on the political proceedings in smoke filled rooms.

If I can read the page (1)

countertrolling (1585477) | more than 3 years ago | (#35809726)

What's to stop me from 'scraping' the info? What's to stop me from simply downloading the entire site with something like this [httrack.com] ? Slowly if needed to avoid arousing suspicion..

Re:If I can read the page (1)

betterunixthanunix (980855) | more than 3 years ago | (#35809884)

Slowly if needed to avoid arousing suspicion..

How slowly? Could you download all Slashdot comments in a profitable amount of time? You would also have to use a download pattern that is not obviously automated (e.g. sequentially requesting each link on a page).

In short, it is not the easiest thing to do. It is like trying to pass the Turing test (which software is getting pretty good at doing, as it so happens).

Re:If I can read the page (1)

hoggoth (414195) | more than 3 years ago | (#35811066)

Run a separate scraper from different IP addresses for each "category" on Slashdot. Each scraper will read all of the articles in that category and refresh the comments from time to time (random intervals) just like a human would. That would be pretty hard to detect.

Re:If I can read the page (1)

TheRaven64 (641858) | more than 3 years ago | (#35811140)

Depends. Am I allowed to use a botnet? From a previous story, I know that you can buy machines on botnets for about five cents each. For a dollar, I could have 20 machines, all grabbing one Slashdot story per minute (probably slow enough not to be seen as a spider). That's about a million Slashdot stories every four days. Maybe make it a million a week to make sure. Spread it over a big botnet and you can get the entire archive in an hour or so, without it looking like anything other than a few hundred thousand users all looking at archived stories.

OK, I Confess! (1)

ackthpt (218170) | more than 3 years ago | (#35809730)

I did expect the Spanish Inqueisiton!

Hello Scrapper (0)

Anonymous Coward | more than 3 years ago | (#35809760)

Hoping somebody is scraping this message.

Irony (1)

angloquebecer (1821728) | more than 3 years ago | (#35809764)

Soon as I click to read the comments, the ad on the right is for a web scraping solution.

Wait a minute... (1)

Anonymous Coward | more than 3 years ago | (#35809798)

You're telling me that stuff on a public web site is public?

Scraping public data to save money for them and us (2)

garcia (6573) | more than 3 years ago | (#35810034)

Because the public sector has very little time to handle FOIA requests and they sometimes cost more money to complete than I'm willing to pay (usually because they don't do much of their own data work in-house and have to call on a contractor to do it for me), I use their websites to glean the data I want.

Last week I gave a talk about using SAS to do screen scraping and then perform analysis on the data of jail inmate registries [sas.com] and level 3 sex offenders in MN. I have dashboards of the data available on my website and as I mentioned in my presentation it has even been used to help one county avoid what could have been a serious privacy issue. [lazylightning.org]

So while there are any number of pitfalls to screen scraping (not understanding the meaning of the data and trends, being fed incomplete or purposefully incorrect data, or even being banned outright) screen scraping can be great for learning about and reporting on the public sector when they are physically or financially incapable or simply unwilling to do it themselves.

my profile (0)

Anonymous Coward | more than 3 years ago | (#35810356)

I would like to see the profile they have build of me.

a.c.

I worked for a social scraping company... (2)

sdguero (1112795) | more than 3 years ago | (#35810386)

The company was SEM/SEO then they moved to social optimization and scraping. It was a black art, like the SEO stuff, and totally dependent on the provider (in this case facebook and twitter) to not change anything. It's the same basic the problem with SEO and Google; if facebook's (or Google's) API coughs the social media scrapers (or SEM/SEO people) get pneumonia. If Facebook wants to stop it, they can do so fairly easily.

Unfortunately for privacy, a huge part of FB's business model (like Google) is selling that data to the scrapers and the scrapers' clients.

Marketing is a sham (1)

xanthos (73578) | more than 3 years ago | (#35810426)

Face it, the type of people who go into marketing have very little to offer this world. Their whole reason for existence is to hopefully sell something to somebody who might not otherwise buy it. The only redeeming aspect of marketing is that it is a non-violent sinkhole in which to drop money, vs say a war in some God forsaken desert.

Have you ever met a marketing/advertising person who actually liked people?

Re:Marketing is a sham (0)

Anonymous Coward | more than 3 years ago | (#35810692)

As opposed to egomaniacal nerds with no social skills and persecution complexes?

Re:Marketing is a sham (1)

Jeng (926980) | more than 3 years ago | (#35811334)

Marketing Marketing Marketing, where the real money from the movie is made!

I was going to post a response agreeing with you, but the more I think on it, well....

Marketing subsidizes my entertainment choices, considering how much Geico spends on advertising I think basic cable would collapse if Geico stopped advertising.

Marketing also helps the company I'm at. Our marketing consists of our catalog and website with our products and pricing. Without that how would our customers know what to buy from us? Some level of marketing is necessary.

Also, the marketing department where I work is full of some real cool people who do indeed like people.

Stalking? (1)

b4upoo (166390) | more than 3 years ago | (#35810792)

Collecting data about others is somewhat an essential freedom. But my view and the modern view differ as most people do not feel the same way. But if we take the usual view any company collecting data about a specific person could be charged with stalking. We usually think of a pervert stalking a child or pretty girl. But stalking is stalking regardless of whether it is a corporation or a pervert. The motive for the stalking is irrelevant. Considering the current mood huge civil suits might take place and even criminal prosecutions might be applied. This is one demonstration of why hacking and social engineering need to be legal. After all, how will you ever know to what degree others are studying you without being able to penetrate their data? Restricting hacking is a path to tyranny that is quite direct and predictable. The natural balance is to allow all people and groups to completely study each other in great depth.

EULA should stop this behavior (1)

hrieke (126185) | more than 3 years ago | (#35810968)

Add a line in your acceptable use / EULA section stating that you expect the user of the account to be human and that any attempt to scrape the data off of the server is fined at $100,000 per message, plus $10,000 to each message author.

Re:EULA should stop this behavior (0)

Anonymous Coward | more than 3 years ago | (#35811286)

Yeah, because the likelihood of collecting any such fines is high enough to justify the time you spend adding the line to the EULA. Not.

Re:EULA should stop this behavior (1)

hrieke (126185) | more than 3 years ago | (#35811746)

Two minutes of your time to insert the HTML?
A day for your lawyer to write up the text, who is either on a retainer or works directly for your company?
That was hard.

Re:EULA should stop this behavior (1)

Just Some Guy (3352) | more than 3 years ago | (#35811606)

Add a line in your acceptable use / EULA section stating that you expect the user of the account to be human and that any attempt to scrape the data off of the server is fined at $100,000 per message, plus $10,000 to each message author.

And also, you reserve the right to sue the Tooth Fairy for lost unicorns.

There is no "legal gray area" in scraping. By publishing data on a public webserver, you give consent to clients for viewing it. And what does "the user of the account to be human" mean, anyway? Presumably, humans will eventually view the data downloaded by the scraper. Challenge of the day: give me a legally watertight definition of "web browser" that includes user agents like Lynx (which downloads data from a remote server and presents it in a manner almost exactly unlike Firefox), and excludes a scraper (which downloads data from a remote server and presents it in a manner almost exactly unlike Firefox). Bonus points if your definition also accounts for screen readers for the blind, HTML-to-WAP gateways, ad-blocking proxies, and iPhones. Go ahead; we'll be waiting.

Re:EULA should stop this behavior (1)

hrieke (126185) | more than 3 years ago | (#35811864)

Sure- Automated process that stores the results in a database or is otherwise used in a system where the results are aggregated and retrievable for 4th party consumption with a method to tie back to a person.

That wasn't difficult at all. Just because I write something for consumption to the members of a particular web site (assuming that it's NOT out in the public like Slashdot's or any other comment system), I would not expect it to be slurped up and sold by 3rd parties. On a member's only web site, such as talked about in the story, the inclusion of my EULA statement would be a strong deterrent against these scrapers.

Reporting Back... (2)

istartedi (132515) | more than 3 years ago | (#35811090)

The report is back sir, and the results are disturbing. Almost everybody likes sex, and a lot of them are weird. The ones that don't like sex have very strange hobbies. The ones that don't abuse illegal drugs are abusing legal drugs, and almost nobody weighs what they say or looks like their online picture. What should we do?

(boss pauses for a moment) "Don't hire anybody ever again".

We run a "scraper". (1)

Animats (122034) | more than 3 years ago | (#35811202)

Our SiteTruth [sitetruth.com] system does some "scraping". We're looking for the name and address of the company behind the web site, so we can check the business out. We also look for ad links and a few other things, like BBBonline seals, which we check. We use a user agent name of SiteTruth.com site rating system. We don't look very deeply into a site; if after examining the most likely 20 pages, we haven't found out who runs the site, we figure they're not going to tell us. The site is down-rated accordingly.

Our experience is that 0.1% of sites have a "robots.txt" file that tells us to not look at any pages at all. [sitetruth.net] We don't look at those sites, and their SiteTruth rating information says "Blocked". Total exclusion of crawlers is rare. Most sites want some visibility.

One of the more amusing uses of a "robots.txt" file used to be seen on Marchex (the "What you need, when you need it" domainer) pages. The site wasn't blocked from crawling, but the link to the page that told you about Marchex was. That, we suspect, was to keep search engines from noticing that all those domains were really one business. That didn't help Marchex much. Marchex (NASDAQ: MCHX) is still around, stock way down from the peak and reporting a slight loss this quarter.

We do have one exception to obeying the "robots.txt" file. We look at the home page of the site to see if it's a redirect before looking at the "robots.txt" file. Some sites have both a redirect and a "keep out" robots.txt file on the same domain. This is like posting signs that say "Keep Out" and "Please Use Other Door" on the same entrance. That contradiction was apparently a workaround for an old Google crawler bug. Google would index both "example.com" and "www.example.com" separately, then consider them duplicates, which caused some SEO problems.

Actually logging into sites from a crawler is just wrong. I'm amazed that a deep pocket like Nielsen would do that.

If already not following the rules (1)

HikingStick (878216) | more than 3 years ago | (#35811390)

If the scrapers are already not following the rules laid out in the robots.txt file, what's to say they'll honor your ban. They'll find some way around any technical means of blocking them, in time.

Some bad practices in HR that needs to end (2)

yuhong (1378501) | more than 3 years ago | (#35811494)

On this topic, here is some bad practices in HR that needs to end:
1. Hiring based on stereotypes is NOT a good idea. [com.com]
2. The purpose of HR should not be to minimize legal liability.
3. The illusion that celebrities are perfect needs to end.
4. Filtering people based on health problems to minimize health insurance costs is not a good idea.
5. Not hiring people based on debt creates a paradox for those who have to pay it off.
And as a side note, companies with seriously broken HR often have other problems too.

Re:Some bad practices in HR that needs to end (1)

Jiro (131519) | more than 3 years ago | (#35811548)

If you don't try to minimize legal liability, you'll find yourself with more legal liability than you need. And legal liability really hurts.

Re:Some bad practices in HR that needs to end (1)

yuhong (1378501) | more than 3 years ago | (#35811778)

But it should not be the primary purpose of HR.

Re:Some bad practices in HR that needs to end (0)

Anonymous Coward | more than 3 years ago | (#35812136)

it is not, it is only 1/5th of their purpose.

Re:Some bad practices in HR that needs to end (1)

nastyphil (111738) | more than 3 years ago | (#35812294)

If you don't try to minimize legal liability, you'll find yourself with more legal liability than you need. And legal liability really hurts.

Liability only hurts if you have done something actionable.

DNA Scraping? (1)

jasno (124830) | more than 3 years ago | (#35811874)

Would that be legal? Could I setup a company that collected DNA samples without their owners permission(say, by tying the hair clippings from a salon to the CC that paid for the cut)? Could I sell that info to the government?

If no one's done it, someone should, if for no other reason than to scare the shit out of people and hopefully wake them up.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...