Botmobi is a web crawler indexing mobile content for the find.mobi search engine. You can identify it in you server logs by looking for the UA string with "botmobi". We visit all types of web site, but index only those with mobile content.
Our crawler is a little non-standard in the way that it works: because we are searching for mobile web content, we don't just crawl the public areas of a website - we also speculatively try a few different combinations of some common mobile web patterns in an attempt to find the mobile version of a site. So, as an example, if we are crawling domain.com we would also attempt a GET of the following URLs:
We try a total of 12 different combinations for each domain. The net effect of this is that for each domain we crawl, there is a possibility that we can generate up to 12 separate 404 errors. To some this may look like a an attempt to "hack" their site, but it isn't. Furthermore, causing these 404 errors has a neglible effect on a web server (many single pages that we come across on sites cause more 404 errors than this).
The crawler is fair and shouldn't hit your site too much. It also supports robots.txt rules. You can use these rules to slow down the crawl (Crawl-delay: ) or to deny the acces to some pages (Disallow: ).
So, in summary, our crawling activity sometimes looks a little unusual for people that keep a close eye on their logs, but our activity is for the site owner's benefit and our impact is negligible. If you would like to enquire further, please get in touch with us by emailing tech@mtld.mobi