Journal WhiteDragon's Journal: slashdot's robots.txt 3
Why is slashdot's robots.txt file allowing msn's and yahoo's indexing bots to crawl comments, user pages, and user journals, but google is prohibited? The stranger thing is that google's ad sense bot is allowed to do so.
Maybe at Google's request? (Score:1)
Re:Maybe at Google's request? (Score:2)
Re:Maybe at Google's request? (Score:2)