BuildSiteReader
BuildSiteReader is how Build reads a company’s own public website to fill in that company’s public card on Build: what it builds, its trades and services, the places it works and its certifications. It identifies itself as:
BuildSiteReader (+https://www.builderslive.org/bot)
What it reads
- robots.txt first. If it says no, nothing else is read.
- The homepage, and up to three more pages on the same site whose links say services, about, projects or capabilities.
- Web pages only: no files, no forms, no logins, and no scripts are run. At most 1 MB a page and 10 seconds a request, one page at a time.
What Build does with it
Each fact on the company’s card cites the page it came from. The company can fix or forget any fact, and a forgotten fact is never read in again. What a company says about itself on Build outranks what was read from its site.
A site is read again at most once every 90 days. A site that turns BuildSiteReader away in robots.txt is left alone for 90 days before robots.txt is checked again.
robots.txt
BuildSiteReader follows the group in robots.txt that names it, or the group for every crawler (User-agent: *) when none does, and honors a crawl delay of up to 5 seconds. To keep it off your whole site, add:
User-agent: BuildSiteReader Disallow: /
To keep it off part of your site, disallow just that path, for example:
User-agent: BuildSiteReader Disallow: /private/
Questions, or a fact to take down
Email support@builderslive.org with your company’s website and what you’d like changed. You can also write to Build.