Our bots
SiteBrite rebuilds websites for small local businesses. To do that, a few of our programs read web pages. If one of them visited your site, this page says which one it was, what it read and how to keep it away.
How they behave
- Each one names itself in the User-Agent header and links to this page.
- They follow your robots.txt, both the rules for all bots (
User-agent: *) and the rules that name them. The one exception is the website checker below. - They read public pages only. They never sign in, fill in forms or try to get past a security check.
- They read a limited number of pages, slowly, and never train an AI model on what they read.
The bots
SiteBriteImporter
Reads a website’s pages, photos and public data so we can rebuild it as a new site.
When: When the business asks us to convert its website, or when we make a free preview for a business we’d like to show it to. For a preview the business didn’t ask for, it also stays away from any site whose robots.txt turns away AI crawlers or says ai-input=no.
- User-Agent
Mozilla/5.0 (compatible; SiteBriteImporter/1.0; +https://sitebrite.app/bots)- robots.txt name
SiteBriteImporter(it also obeys rules for its old name,sitestudioimporter)
SiteBriteProspector
Looks at a local business’s home page, its contact page and its sitemap, to learn what the site is built with and how to get in touch.
When: While we look for local businesses whose websites we could improve. It reads a few pages, slowly, and passes over any site whose robots.txt turns it or AI crawlers away.
- User-Agent
Mozilla/5.0 (compatible; SiteBriteProspector/1.0; +https://sitebrite.app/bots)- robots.txt name
SiteBriteProspector(it also obeys rules for its old name,sitestudioprospector)
SiteBriteChat
Reads up to 8 pages of a business’s website so its Ask us chat can answer visitors’ questions.
When: Only when the business names its own website for its chat, and again once a month to stay up to date.
- User-Agent
Mozilla/5.0 (compatible; SiteBriteChat/1.0; +https://sitebrite.app/bots)- robots.txt name
SiteBriteChat
SiteBriteTools
Our free website tools: the website checker reads one page, and the redirect map lists a site’s pages.
When: When someone types the address into one of the tools. The redirect map obeys robots.txt. The checker reads only the one page it was asked about, the way Google’s PageSpeed Insights does.
- User-Agent
Mozilla/5.0 (compatible; SiteBriteTools/1.0; +https://sitebrite.app/bots)- robots.txt name
SiteBriteTools
Keeping them away
To keep all of them off your site, add this to your robots.txt:
User-agent: SiteBriteImporter
User-agent: SiteBriteProspector
User-agent: SiteBriteChat
User-agent: SiteBriteTools
Disallow: / To keep one away, use only its line. Names aren’t case-sensitive. A rule for part of your site (Disallow: /private/) keeps them out of just that part.
We also leave alone a site that turns away AI crawlers (ClaudeBot, GPTBot and others) or says Content-Signal: ai-input=no: we won’t make a preview of it unless its owner asks us for one.
Questions, or want us to forget your site? Email help@sitebrite.app. We’ll make sure nothing of ours reads it again.
The names our bots obey in robots.txt: sitebriteimporter, sitestudioimporter, sitebriteprospector, sitestudioprospector, sitebritechat, sitebritetools.