Brightspot CMS User Guide

Customizing a site's robots.txt

A robots.txt file provides crawling guidance to search engines, such as the following:

  • Which user agents are allowed to crawl or not crawl
  • Which URLs to crawl or not crawl
  • Locations of sitemaps
  • Limiting the frequency of crawling

Properly configuring your site's robots.txt enhances search-engine optimization.

Brightspot provides a default robots.txt with the following directives:

User-agent: *
Crawl-delay: 10

Brightspot creates the file robots.txt at the location in the field Main > Default Site URL. For example, if your site's default URL is, Brightspot creates the robots.txt file at

To customize a site's robots.txt:

  1. Click menu > Admin > Sites & Settings.
  2. In the Sites widget, select the site for which you are configuring robots.txt, or select Global to configure robots.txt for all sites.
  3. Click search, located to the left of more_horiz, and type robots.txt.
  4. In the robots.txt field, enter directives for the search engine. See the search engine's documentation for the list of honored directives.
    Example robots. Example robots.
    Example robots.txt configuration
  5. Click Save.

See also:

Previous Topic
Configuring sitemap generation
Next Topic
Brightspot Analytics
Was this topic helpful?
Thanks for your feedback.
Our robust, flexible Design System provides hundreds of pre-built components you can use to build the presentation layer of your dreams.

Asset types
Module types
Page types
Brightspot is packaged with content types that get you up and running in a matter of days, including assets, modules and landing pages.

Content types
Landing pages
Everything you need to know when creating, managing, and administering content within Brightspot CMS.

Admin configurations
A guide for installing, supporting, extending, modifying and administering code on the Brightspot platform.

Field types
Content modeling
Rich-text elements
A guide to configuring Brightspot's library of integrations, including pre-built options and developer-configured extensions.

Google Analytics
Apple News