Indeed Crawlers

IndeedJobBot

IndeedJobBot is the name of a bot that Indeed uses when crawling for job and job-related information.

You can identify the IndeedJobBot by looking at the HTTP user-agent request header in the request, if it is an Indeed bot you will see User-Agent: IndeedJobBot.

How IndeedJobBot accesses your site

IndeedJobBot is interested in job related information, this includes titles, descriptions, location, compensation, benefits as well as other data such as job requirements. It may also want to crawl pages which include company locations such as stores, branches or other location identifying information but only as this pertains to jobs.

IndeedJobBot does not intend to crawl the entire site or follow any and all links present. The only links that IndeedJobBot will crawl are directly related to job lists, job details and/or location lists and location details pages.

Blocking IndeedJobBot from visiting your site

If you want to prevent IndeedJobBot from crawling content on your site you can do one or more of the following methods:

Use a robots.txt file
- Example robots.txt for only allowing access to specific pages (assuming your jobs are present under the /jobs/ path and you want Indeed to only crawl those pages)
```
User-agent: IndeedJobBot
Allow: /jobs/
Disallow: /
```
- Example robots.txt for disallowing any access
```
User-agent: IndeedJobBot
Disallow: /
```

Block these IPs:

18.219.101.101
3.128.63.195 
3.21.142.42

Verifying IndeedJobBot

Before you decide to block IndeedJobBot, be aware that the HTTP user-agent request header used by IndeedJobBot can be spoofed by other crawlers. It's important to verify that a problematic request actually comes from Indeed. The best way to verify that a request actually comes from IndeedJobBot is to match the source IP against the one of these crawler IPs: ips-v4