Disqus - Latest Comments for WillMacc

Re: Scraping Starts from the Very First Post

WillMacc — Tue, 22 May 2007 14:30:53 -0000

The moving of the blog and all that came after really wasn't to see who was scraping; but it did provide somewhat confirmation that probably a lot of those still visiting the site and the site feed was more than likely bots that are still habitually visiting the site/feed daily.
I understand that people don't regularly check RSS readers - as in Google Feed Fetcher - but I would dare to guess that's only a small percentage of the visits.
Moving the blog also provided me a chance to see who/what visits the site; as where with my WordPress.com blog, I could only see hits and not the actual visit information.
Since the move; probably 10 crawlers have been shotdown from scraping the content of the blog, BUT, scraping a blog like mine isn't that big of a deal since it's the information on the blog that's important. So, if my content gets scrapped and ends up up another blog - fine; the information is still valid and people still get to see who's doing what with what and whom.. :)

Thanks,
WillMacc

Re: Scraping Starts from the Very First Post

WillMacc — Tue, 22 May 2007 14:41:33 -0000

Also... :)
A lot of "pseudo" feeders are attached and monitor other ping services.
I've seen countless visits from known crawlers with "bad intentions" hit the site as soon as a ping is sent out.
If you have a blog hosted on your own domain, you can issue a ping (to only one service - say; pingomatic) and then sit back and watch who starts hitting the site.
You'll see quickly a boat load of crawlers come and a lot of them will not appear as crawlers, but as regular user-agents. If you follow the trends of the crawlers/visitors after a ping, you'll probably start noticing some visitors will not pull any graphics on the blog; or only pull one hit as where most visitors will have line upon line of various content, items, and graphics that's embedded into the blog themes and within the articles.
Those that do that are Usually bots and not legit users, but having said that, you'll have to be careful and pick out the rss readers from the bots and crawlers.

Thanks,
WillMacc