Disqus - Latest Comments for thesearchmonitor

Re: The Search Monitor

thesearchmonitor — Fri, 04 Jul 2008 11:34:29 -0000

Hello Dominic,

Google and others do have limits, we are careful not to impose on those. We are also careful to be sure that the SE's know that we are a crawler so that there is no negative impact on impressions and CTR's.

We look forward to working with you.

Re: The Search Monitor

Dominic — Fri, 04 Jul 2008 11:30:40 -0000

Sounds good, I will give it a go.

Do you find that the SEs throttle your crawling capacity? I know that Google has some limits in terms of frequency.

Re: The Search Monitor

thesearchmonitor — Thu, 03 Jul 2008 14:57:40 -0000

Hi Dominic,

We are crawling the search providers using certain urls that provide information on the sponsored bidders. Crawling occurs many times per day to cover time zones and day part behaviors.

The crawler also crawls blogs, press, optimized sites, and landing pages.