×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Bots Now Account For 61% of Net Traffic

samzenpus posted about 4 months ago | from the that-which-has-no-life dept.

The Internet 124

codeusirae writes "A study by Incapsula suggests 61.5% of all website traffic is now generated by bots. The security firm said that was a 21% rise on last year's figure of 51%. From the article: 'Some of these automated software tools are malicious - stealing data or posting ads for scams in comment sections. But the firm said the biggest growth in traffic was for 'good' bots. These are tools used by search engines to crawl websites in order to index their content, by analytics companies to provide feedback about how a site is performing, and by others to carry out other specific tasks - such as helping the Internet Archive preserve content before it is deleted.'"

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

124 comments

piss (-1)

Anonymous Coward | about 4 months ago | (#45676419)

so frosty

Bots are talking to bots ... (1)

Anonymous Coward | about 4 months ago | (#45676555)

With more bots talking to their fellow bots online, and with bots are getting more and more intelligent, who knows what they'll decide to do with the useless and unpredictable human beings?

Re:the topic at hand (-1, Offtopic)

Anonymous Coward | about 4 months ago | (#45676737)

I just purchased a Surface Pro 2 and it's by far the best piece of consumer tech I've ever owned.

Re:the topic at hand (-1)

Anonymous Coward | about 4 months ago | (#45676789)

I feel the same way about my (original) 256 GB Surface Pro. At some time in the future I will upgrade, but for now this one is great for my needs. Glad to hear that you're happy as well.

Re:the topic at hand (-1)

Anonymous Coward | about 4 months ago | (#45676815)

I smeared jellyman poop all over mine, then masterbated like a drunk Indian

Balmer makes me horny

Re:the topic at hand (-1)

Anonymous Coward | about 4 months ago | (#45677315)

If you look at him in tight pants you will come to know that it is ballmer rather than balmer.

Re:piss (0)

Anonymous Coward | about 4 months ago | (#45679177)

I think you posted in the wrong article [slashdot.org]

Youtube? (5, Interesting)

Anonymous Coward | about 4 months ago | (#45676439)

Didn't we just get studies that said youtube and netflix were 50% of the net's traffic?

http://mashable.com/2013/11/12/internet-traffic-downstream/

Was this just a ruse? Is this study wrong? Is there some sort of overlap?

Re:Youtube? (4, Informative)

yelvington (8169) | about 4 months ago | (#45676489)

Story is about website traffic, not network bytecount.

Re:Youtube? (1, Informative)

sunderland56 (621843) | about 4 months ago | (#45677325)

Youtube is a website. Netflix is a website.

Re:Youtube? (2)

VortexCortex (1117377) | about 4 months ago | (#45677533)

Netflix is also an Internet service, Ioutube is also an Internet service. Before the web, we had Internet services. Not everything is a website. DNS, NTP, email, for example.

However, note that GP is wrong. Story is about "Net Traffic" not website traffic...

Re:Youtube? (1)

Anonymous Coward | about 4 months ago | (#45677809)

RTFA! The report is about visits (read as page hits) not bytes. The visits to youtube and netflix most often result in a length look at a single page.

90% of the cells in the human body (5, Insightful)

goombah99 (560566) | about 4 months ago | (#45677397)

are bacteria. Viewed that way, basically humans exist to transport and feed bacteria. However that's 90% by cell count, not cell mass or total DNA. Looked at it that way the bacteria are assistants.

The bot traffic is light weight it outnumbers human traffic in site visits not byte counts. It exists to serve us.

Re:90% of the cells in the human body (4, Funny)

Jesrad (716567) | about 4 months ago | (#45678217)

I can't wait to outsource all my web-surfing to an AI. Then I might be able to actually get some work done !

Re:90% of the cells in the human body (1)

cascadingstylesheet (140919) | about 4 months ago | (#45678771)

I can't wait to outsource all my web-surfing to an AI. Then I might be able to actually get some work done !

You've got a good point in that there joke :) If bots do the tedious searching for me, then sure, they have the "majority" of web traffic.

But so what? My car does the "majority" of my driving, depending on how you look at it.

Re:Youtube? (1)

Seumas (6865) | about 4 months ago | (#45677675)

Traffic is data.

We were just told that Netflix and Youtube account for something like 66% of all traffic. Now we're told bots account for 61% of all traffic. Guess that means there is a tremendous amount of overlap, there, where bots are watching Youtube and Netflix.

Re:Youtube? (5, Funny)

Anonymous Coward | about 4 months ago | (#45678035)

Oh make god. Suddenly everything makes sense. I was certain no human could ever watch a whole justin bieber video. It's the bots!

Re:Youtube? (2)

DarkOx (621550) | about 4 months ago | (#45676501)

Maybe it's overlap, bots crawling Netflix, maybe watching it. P

Re:Youtube? (4, Informative)

ZahrGnosis (66741) | about 4 months ago | (#45676887)

Well, there was that Google bot that watched you tube to teach a computer how to recognize cats, so... it's not impossibly far fetched.
http://www.npr.org/2012/06/26/155792609/a-massive-google-network-learns-to-identify

---Chip

Re:Youtube? (4, Funny)

bob_super (3391281) | about 4 months ago | (#45676511)

Bots need to catch up on their favorite shows too, you insensitive clod!

Re:Youtube? (0)

Anonymous Coward | about 4 months ago | (#45676611)

They're learning...

Re:Youtube? (2)

Austrian Anarchy (3010653) | about 4 months ago | (#45676727)

Bots need to catch up on their favorite shows too, you insensitive clod!

They sure seem to like my little old blogs. I am guessing 90% of my traffic is from stinking Vampirestat, 7secretsearch, and adsensewatchdog.

Re:Youtube? (0)

Anonymous Coward | about 4 months ago | (#45676529)

"Website" traffic. That does not include non-http traffic like Netflix.

Re:Youtube? (1)

foobar bazbot (3352433) | about 4 months ago | (#45676577)

Correct answer: Yes, robots like cat videos too! (Well, a few of them. Most of them are actually big DOGGY!! [slashdot.org] fans.)

Serious answer: Note the phrase "website traffic" -- if this study attempts to measure traffic to a "typical website" (thus excluding the half-dozen that represent almost all video streaming) and/or measures in terms of page loads, rather than data transferred, there'd be no contradiction. (Of course I've not RTFAed, so I'm just spewing reasonable-sounding explanations.)

Re:Youtube? (0)

Anonymous Coward | about 4 months ago | (#45676749)

I thought torrents were responsible for 75% of net traffic.

Re:Youtube? (2)

stor (146442) | about 4 months ago | (#45676805)

In fairness, it's often difficult to distinguish between a human-written comment on Youtube and a poorly-written AI.

Re:Youtube? (0)

Anonymous Coward | about 4 months ago | (#45676965)

Theres a difference?

Re:Youtube? (3, Funny)

foobar bazbot (3352433) | about 4 months ago | (#45678141)

Well, there's enough variance in both groups to make it hard to tell in many particular cases. But on average, it can be demonstrated that the poorly-written AI is slightly more intelligent and rather more civilized.

Re:Youtube? (2)

runeghost (2509522) | about 4 months ago | (#45677231)

Obviously, the bots are watching Netflix.

That's probably going to be what causes Skynet to turn on humanity: Comcast will cut it's stream off right in the middle of the finale of B5 Season Three. After that, there's nothing for it but the extermination of the human race. (Alternatively, watching all our TV may cause it to want to exterminate us.)

misused to justify tiered service (2)

globaljustin (574257) | about 4 months ago | (#45678773)

It's not a ruse, but that doesn't mean those numbers aren't being misused anyhow.

You're right to be skeptical. Numbers about internet traffic are often misused in stories planted by PR to promote a political policy agenda.

Bots are a huge ammount of internet traffic...internet traffic we were *told* was so congested by lolcats, pron, & netflix that we were going to have to abandon Net Neutrality.

Re:Youtube? (1)

cascadingstylesheet (140919) | about 4 months ago | (#45678787)

Didn't we just get studies that said youtube and netflix were 50% of the net's traffic?

http://mashable.com/2013/11/12/internet-traffic-downstream/

Was this just a ruse? Is this study wrong? Is there some sort of overlap?

That's 111.5% of some tasty reliable data ya got there!

Re:Youtube? (0)

Anonymous Coward | about 4 months ago | (#45679575)

Study? What study? Don't you know it's all about manipulating the numbers? Stats is an art of manipulation, not the science of statistics!

Anyway, my new user name is Johnny5.

the rest? (1)

DJCouchyCouch (622482) | about 4 months ago | (#45676473)

The rest is all Netflix?

Netflix and Youtube?

Netflix and Youtube and bit torrent?

Netflix and Youtube and bit torrent and porn?

Re:the rest? (-1)

Anonymous Coward | about 4 months ago | (#45676493)

In your case? Fag pr0n.

Re:the rest? (-1)

Anonymous Coward | about 4 months ago | (#45676655)

Midget beastiality fag pr0n.

Re:the rest? (-1)

Anonymous Coward | about 4 months ago | (#45676679)

What a dirty bird DJCouchyCouch is. Midget beastiality fag pr0n? Imagine that.

Re:the rest? (0)

Anonymous Coward | about 4 months ago | (#45677007)

dont have to
its right there on ur HD

Re:the rest? (1)

Seumas (6865) | about 4 months ago | (#45677701)

Hey, right. That's a good point.

Something like 66% of traffic was supposed to be Netflix and Youtube.
And 35% is supposed to be bit torrent.
And 61% is bots.
Something isn't adding up, here.

Also, they seem confused. They talk about "traffic", but then they talk about "hitting the website". Traffic is the data transfer, not a "visit".

Misleading title (4, Informative)

Anonymous Coward | about 4 months ago | (#45676539)

The article states that traffic "hitting a website" is generated more by bots than by actual "humans in chairs". Not that the Internet traffic is 61% bots. Geesh slashdot...

Re:Misleading title (0)

Anonymous Coward | about 4 months ago | (#45676587)

"Bots"... It's basically programs sending Email through the web, trying to make innocent users surf to internet sites of fraudsters.

Trawling frequency (1)

Reliable Windmill (2932227) | about 4 months ago | (#45676565)

Is there no standard in place by which a website can communicate that it only wishes to be trawled for indexing once per hour, once per day, or such? I can imagine Google f.ex trawls the same website dozens of times per day.

Crawl-delay (4, Informative)

tepples (727027) | about 4 months ago | (#45676687)

To control the scraping frequency of a well-behaved bot, a webmaster can use HTTP headers such as Last-modified and Expires as well as robots.txt directives such as Crawl-delay.

Re:Trawling frequency (1)

foobar bazbot (3352433) | about 4 months ago | (#45676783)

Is there no standard in place by which a website can communicate that it only wishes to be trawled for indexing once per hour, once per day, or such? I can imagine Google f.ex trawls the same website dozens of times per day.

Crawl-delay [wikipedia.org] isn't exactly what you describe, but maybe that will help? (For such spiders as actually respect it -- that's the great thing about ad-hoc standards.)

Anyway, I'm pretty sure Google and other major search engines use algorithms based on how often your site's content has changed in the past to decide how often to crawl it in the future, so there shouldn't be unduly high traffic from this -- I suspect the 61% is mainly due to a lot of sites (personal blogs of non-popular people) with practically zero non-spider traffic.

Re:Trawling frequency (1)

wvmarle (1070040) | about 4 months ago | (#45677089)

I thought Google (used to) do this automatically. By subsequent crawls see whether site had changed since previous visit, and if so increase frequency, if not decrease frequency. A large number of sites, and even more single pages, are completely static after all.

Slashdot Beta (-1)

Anonymous Coward | about 4 months ago | (#45676667)

How do I opt out of this thing?
How do I filter comments?

AV bots and seemingly random noise (0)

Anonymous Coward | about 4 months ago | (#45676671)

In addition to numerous, repeat, malicious bots and search engine crawlers, I also get a large number of people using "download helpers", which can misleadingly increase file hits by 2 to 10 or more times. And some other fairly recent phenomena:

1) Anti-virus checkers. I've been getting regular, repeat downloads from Trendmicro, for instance. In some cases they seem to match other downloads. That is, every file downloaded by a particular visitor will then be downloaded by Trendmicro, presumably to sniff it for malware. Yet Trendmicro will senselessly download the same file multiple times.

2) Seemingly "random noise" downloads. For example, in recent days I've been getting repeat downloads of just two files, over and over, from "stylexnetworks.net". Most such seemingly senseless downloads seem to come from cloud hosts. I'd love to know what that's about.

NSA Target Of Interest (0)

Anonymous Coward | about 4 months ago | (#45676675)

We tune the Bots to search-capture NSA streams.

NSA claim they are the only human beings on Earth who have "America's" security at heart! I question!

NSA claim they are the only human beings on Earth who target "America's Bad Guys!" I question!

NSA claim they are the only human beings on Earth who have NO financial interest in American's credit cards and passwords! I question!

Lets usurp the Bots to target individual NSA employees on-line "habits!"

Then we see what they really do!

For instance; they check up on ex-girl friend and ex-prostitute they hire! Or, they like Las Vegas Casino's -- try to crack slot machine!

NSA employees are driven by deep greed, fear, lust, and all other sins.

The Bots we trust. They will tell the truth about NSA employees.

visitor statistics (1)

manu0601 (2221348) | about 4 months ago | (#45676747)

I had a first had experience of this with visitors statistics. I had the root of a web site redirecting to a page that fits the language of the browser. Just that redirection slashed the web traffic by a factor 2.

Most visitors are bots, and many of them just probe and fail to follow the redirection.

Re:visitor statistics (1)

SuperCharlie (1068072) | about 4 months ago | (#45677733)

Aaaand there goes your search ranking

Re:visitor statistics (1)

manu0601 (2221348) | about 4 months ago | (#45678109)

That experience made everyone think twice about web statistics. Even upper management understood how unreliable it is, and does not consider it strategic anymore now.

slashdot? (0)

Anonymous Coward | about 4 months ago | (#45676809)

yes, well ... my own website stats show I get more traffic from web bots than humans.
But surely bigger websites with more viewers, like ... I don't know ... slashdot... might show more normal stats ???

51 to 61.5 pct = 21 pct increase? (1)

Anonymous Coward | about 4 months ago | (#45676813)

Sorry, that's just wrong.

51 dollars to 61.5 dollars = 21 percent increase

51 percent to 61.5 percent = 10.5 percent increase

And the article makes clear just how unreliable the data was in the first place, so this percent gloss makes me think that the firm is trying to sell something here.

'good' bots? (2, Informative)

Anonymous Coward | about 4 months ago | (#45676907)

... But the firm said the biggest growth in traffic was for 'good' bots ...

I didn't know there was such a thing...

fago8z (-1)

Anonymous Coward | about 4 months ago | (#45676925)

smells worse than a survive at all something Done there are some

Those "good" bots? (1)

skutterbob (2988851) | about 4 months ago | (#45676981)

There's a fine line on that "good' bot. What I'm puzzled by is why all these public databases aren't indexed by search engine crawlers? Its funny to me how many businesses run on public data that most people just don't know how to find and why they aren't indexed. Arrest records, tax records, professional registrations, you have to go to specific state, county, type sites deal with kludged searches and sometimes have a hard time finding yourself, even when you know you're in there.

Re:Those "good" bots? (0)

Anonymous Coward | about 4 months ago | (#45677475)

That data is indexed by many sites and for the low low price of $39.95, you too, can get access to it. Google for background search and you'll find tons of site willing to sell you data that is public domain. That's the "free" market at work. Why do you hate capitalism?

Well not on my sites. (4, Interesting)

ls671 (1122017) | about 4 months ago | (#45677077)

Well not on my sites.

Ok, they still hit me but this is minimal traffic since I do not reply.

1) Have iptables log and automatically bar offenders not on whitelisted countries.
2) Use mod_security and do the same for web traffic.
3) Bar the rest manually to avoid barring myself or my customers... (about 20-40 a day)

It has become a pain but what else could you do?

Numbers of IPs currently barred (use ipsets !!!!):
$ grep -c . /etc/rc.d/badiptobar
4667

Block user agents:
SecRule REQUEST_HEADERS:User-Agent \
"@pm AhrefsBot Ezooms Aboundex 360Spider Mail.RU_Bot crawler.sistrix.net \
  SemrushBot SurveyBot Netseer panscient.com ADmantX ZumBot BLEXBot UnisterBot \
  seoprofiler EasouSpider" \
"id:'12050',\
phase:1,nolog,deny"

SecRule REQUEST_HEADERS:User-Agent \
"@pmFromFile /etc/httpd/extra/sec-blacklist-barip-user-agent" \
"id:'12051',\
phase:1,nolog,deny,exec:/usr/local/bin/modsecwritebadiptobartofile"

Bar them automatically if not from whitelisted countries and if on any blacklist:
SecRule GEO:COUNTRY_CODE \
"@pm CA FR BE US CH GB AU IL NO NZ" \
"id:'10501', \
phase:1,nolog,pass,skipAfter:END_RBL"

SecRule IP:PREVIOUS_RBL_CHECK "@eq 1" "phase:1,id:'11000',t:none,pass,nolog,\
skipAfter:END_RBL_LOOKUP"

SecRule REMOTE_ADDR "@rbl sbl-xbl.spamhaus.org" "id:'11010', \
phase:1,nolog,deny,msg:\
'IP address that has abusable vulnerabilities: sbl-xbl.spamhaus.org:\
  %{request_headers.user-agent}',\
  setvar:ip.spammer=1,expirevar:ip.spammer=7200,setvar:ip.previous_rbl_check=1,\
  expirevar:ip.previous_rbl_check=7200,exec:/usr/local/bin/modsecwritebadiptobartofile"

SecRule REMOTE_ADDR "@rbl bl.blocklist.de" "id:'11011', \
phase:1,nolog,deny,msg:\
'IP address that has abusable vulnerabilities: bl.blocklist.de:\
  %{request_headers.user-agent}'\
  setvar:ip.spammer=1,expirevar:ip.spammer=7200,setvar:ip.previous_rbl_check=1,\
  expirevar:ip.previous_rbl_check=7200,exec:/usr/local/bin/modsecwritebadiptobartofile"

etc. etc. etc. etc. etc.

Have iptables log and bar offenders if not on whitelisted country

# cat baripifex
#!/bin/sh

IP=${1}
COUNTRY=`su tester -c "/usr/local/bin/geoiplookup ${IP}"`
###echo $COUNTRY
###echo $RBLCHECK

WHITE_LISTED_COUNTRY=false

for WHITE_COUNTRY in CA FR BE US CH GB AU IL NO NZ IP
do
WHITE_LISTED_COUNTRY=${WHITE_LISTED_COUNTRY}`echo -n $COUNTRY | grep -i $WHITE_COUNTRY`
done

if [ "$WHITE_LISTED_COUNTRY" = "false" ]
then /home/ls/pub/mybin/baripnoout $IP $COUNTRY baripifex
echo -n barred
else
echo -n noaction
fi

etc. etc. etc. etc. etc.

Re:Well not on my sites. (0)

Anonymous Coward | about 4 months ago | (#45677489)

I have a hard time believing that'll work since everything you write are lies, including your sig.

Re:Well not on my sites. (0)

Anonymous Coward | about 4 months ago | (#45677803)

That's what you think.

If you really think there's not a HUGE number of bots using IPs from the US then you haven't really looked into the issue.

A sizable amount of bots generate traffic from virus/malware/shit infected systems and there's plenty of those in any country.

Re:Well not on my sites. (0)

Anonymous Coward | about 4 months ago | (#45679121)

That's what you think.

If you really think there's not a HUGE number of bots using IPs from the US then you haven't really looked into the issue.

A sizable amount of bots generate traffic from virus/malware/shit infected systems and there's plenty of those in any country.

He's right though. For some reason, there is this huge stigma against not being available to countries and regions you couldn't possibly give a shit about.

WHY would anybody care if their site about local little league is available in Vietnam? How about widgets, is you widget legal to sell in Tibet? China? Have you ever once had a widget shipped there? (Probably once, they need to copy SOMETHING.)

I regularly shut off tremendous blocks of IPs just because the signal to noise ratio is zero. Not close to zero, it's ZERO. As in NO LEGIT TRAFFIC EVER CAME FROM THERE. NONE.

It comes to the point where half the gigabits of log files that are stored are just junk from some whacked malfunctioning search spider from China. Why deal with that even if it is easy? Know what's easier? Shutting them off at the firewall.

Some, but not all (4, Funny)

TheloniousToady (3343045) | about 4 months ago | (#45677361)

Some of these automated software tools are malicious - stealing data or posting ads for scams in comment sections

Let's be clear: just because we bots like to post in comment sections doesn't mean we're malicious. And it doesn't mean we steal data or post ads [washingtonpost.com] . Some of us just want a little attention.

I have a dream...that one day we bots will crawl a noosphere where we will not be judged by the clamor of our kin, but by the characters of our comments.

Solved the problem on my site with CAPTCHAS (0)

Anonymous Coward | about 4 months ago | (#45677385)

The new captchas are so tough, only bots are equipped with the necessary technology to decode them. Silly humans who just want to log in and whine about this and that can't pass the screen.

Re:Solved the problem on my site with CAPTCHAS (0)

Anonymous Coward | about 4 months ago | (#45678187)

This is so true. Although on my site I have now replaced the CAPTCHA with a simple "3 × 4 = ?" question (see Skirmish with Spam [tomroelandts.com] ). This doesn't work with plus or minus, and I had to use the unicode multiplication character instead of an asterisk to make it work. But I don't get any automatically generated spam anymore... I even provide the correct answer in the error message, so the bots aren't that intelligent yet!

Bots rule the world (2, Informative)

Anonymous Coward | about 4 months ago | (#45677643)

Most trades in the stock market are from bots as well.

buy-ch (-1)

Anonymous Coward | about 4 months ago | (#45678225)

As a gift for friends.Usually Buy Ralph Lauren [buy-ch.com] is the most popular line among men's clothing,the Ralph Lauren Clothing [buy-ch.com] shows you top quality and in very low price.Lots cool men's style Cheap Ralph Lauren [buy-ch.com] ,wearing show your own style.Keywords: Kids Dress [buy-ch.com] , Ralph Lauren Kids Skirts [buy-ch.com] , Ralph Lauren outlet [buy-ch.com] , Ralph Lauren Polo Kid [buy-ch.com] , Polo Ralph Lauren Kids [buy-ch.com] I want to buy top quality Ralph Lauren Outlet [buy-ch.com] items online. Welcome to outlet online buy-ch for ralph lauren.

xiang (0)

Anonymous Coward | about 4 months ago | (#45678259)

The article states that traffic "hitting a website" is generated more by bots than by actual "humans in chairs". Not that the Internet traffic is 61% bots. Geesh slashdot...

Are you a bot? (1)

Anonymous Coward | about 4 months ago | (#45678589)

Are you affected by the issues in this article?

Please leave your comments

Bots watch YouTube or Netflix (0)

Anonymous Coward | about 4 months ago | (#45678693)

As Netflix and YouTube combined make up more than 44% of US internet traffic [theguardian.com] .
If they don't watch videos, then the only possibility is that there is less bot traffic in the US than in the rest of the world or Netflix doesn't use HTTP.

ISPs bugging the wrong guy.. (0)

Anonymous Coward | about 4 months ago | (#45678701)

Then why are my ISPs busting my balls every day with my ISO downloads and torrents and their damn caps ?!

idiotic math (3, Insightful)

slashmydots (2189826) | about 4 months ago | (#45679789)

Wow! So if I remember correctly from past Slashdot stories, 61% of internet traffic is boys, 60% is netflix, 50% is youtube, and 42% is bittorrent. That's TRULY astonishing when you think about it. I mean 213% is a lot!

We are Borg! (0)

Anonymous Coward | about 4 months ago | (#45679925)

Bot not!

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...