Bot

We responsibly crawl your website while looking for the jobs. During crawl we try not to degrade the performance of your website for other users. Below are the details about our crawling policies:

• User-Agent string for our bot is- harsilbot/1.1 (+https://www.harsil.com/bot)
• Our bot respect robots.txt on your website and understands following fields
  • user-agent
  • allow
  • disallow
• The minimum duration between crawls for a website is 24 hours.
• Crawling algorithms are optimized to primarily look for job pages.
• Our bots avoid aggressive crawling by limiting the parallel requests.