serpDecide which requests you want to control
Start with a route inventory, not a copied robots.txt template. Separate public articles, account pages, internal search results, preview routes, and static assets. For each group, decide whether it should be accessible to search crawlers and whether users should find it through search. Those decisions are related but not identical. Bing's documentation distinguishes crawler instructions from other indexing controls. A sensible robots policy describes actual route behavior in your application. A broad rule borrowed from a different CMS can block your best content or fail to cover the paths you intended to exclude.
Read the effective Bingbot group
Review user-agent groups carefully. Bing's guide explains that a specific Bingbot section changes how generic directives apply, so do not assume restrictions from another section are automatically combined in the way you expect. Keep the file simple enough for a teammate to reason about. Test representative allowed and disallowed URLs against the intended rules. Include paths with similar prefixes to catch overly broad matching. A rule meant for an internal /search/ endpoint should not accidentally cover a useful /search-engine-guide/ article because someone abbreviated the path without considering how it matches.
Do not hide the instruction you need read
If the goal is removing a public page from search through a page-level directive, consider whether a crawler can retrieve that directive. Blocking access and expecting the crawler to read the blocked document creates a logical conflict. Likewise, robots.txt is not an authentication system for private customer content. Protect private resources with real access controls. Keep the operational intent explicit in your ticket: reduce crawling of this route family, exclude this page from search, or restrict access to this data. Each goal has a different implementation and a different acceptance test.
A hypothetical preview leak
Suppose a fictional blog exposes /preview/article-name without authentication and then adds a robots rule as its only protection. That may communicate a crawl preference, but it does not prevent a person with the URL from viewing the draft. The proper access decision is to require authorization for previews. Separately, the public article route needs a deliberate indexing policy when published. This example demonstrates the difference between confidentiality and crawler behavior. It does not imply that every publicly reachable preview will be indexed or that a robots file itself reveals sensitive content.
Check the deployed file after every routing change
Request /robots.txt on the production origin and inspect its body. A framework fallback can return HTML instead of the intended file, while an environment variable can accidentally deploy a staging block to production. Add a simple release check for the homepage, a current article, and robots.txt together. If your sitemap is referenced there, confirm the address still resolves. Store a short explanation beside each non-obvious rule in version control. When the site's route structure changes, review those explanations rather than carrying historical exclusions forward indefinitely. The best file is understandable, intentional, and aligned with the live application.
Write crawler rules from your actual route inventory, and keep access control, crawling, and indexing decisions distinct.
Sources & further reading
Official documentation informs this guide. Examples are illustrative, and product interfaces or limits can change.
From search signals to a useful first draft.
SERP.so is in development. Join the waitlist for a workspace that connects research, content review, and publishing.
Join early access ↗