If you're reading this, chances are you've seen the Knight robot/crawler visiting your site while looking through your server logs. Our software obeys robots.txt files and robot META tags in HTML. These are the standard mechanisms for webmasters to tell web robots which portions of a site a robot is welcome to access.
Knight is the robot/web-crawler that is deployed by Ziva Software Pvt Ltd to gather content for its mobile search engine Zook.
Knight is based on the open source web-search software Nutch.
If your are a sysadmin or a webmaster, you can control how Knight behaves on your website in one of the following two ways:
Our software obeys the robots.txt exclusion standard, described at http://www.robotstxt.org/wc/exclusion.html#robotstxt. To ban our crawler from your site, place the following in your robots.txt file:
User-agent: Knight
Disallow: /
If you do not have permission to edit the /robots.txt file on your server, you can still tell robots not to index your pages or follow your links. The standard mechanism for this is the robots META tag, as described at http://www.robotstxt.org/wc/meta-user.html.
If your site has problems or questions about the Knight crawler, please send an email to knight@zook.in.