Searchlab Robots.txt and Sitemaps: Telling Google Exactly What to Index
Your robots.txt and sitemap.xml are the two files Google looks for first when it visits your site. Get them right and you control what gets indexed. Get them wrong, and you can accidentally block your entire site from search.
What robots.txt actually does
It's a plain-text file at yourdomain.com/robots.txt that tells crawlers which pages to skip. Common uses: blocking admin pages, staging URLs, duplicate filtering pages, internal search results.
It does NOT prevent indexing. If another site links to a blocked URL, Google can still index it (just without crawling the content). To truly remove a page, use a noindex meta tag, not robots.txt.
A robots.txt that won't bite you
User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /*?session= Sitemap: https://yoursite.com/sitemap.xml
Sitemap rules that matter
- Only include URLs you want indexed. Don't list 404s, redirects, or noindex pages.
- Use
<lastmod>dates. Helps Google know when to recrawl. - Split big sites. If you have more than 50,000 URLs, use a sitemap index that points to multiple sub-sitemaps.
- Submit to Search Console. Just having the file isn't enough, tell Google where it is.
The classic disaster: Developers push a staging robots.txt to production:
User-agent: * / Disallow: /. Your entire site disappears from Google overnight. Always inspect your robots.txt before deploys.Frequently asked
How fast does Google read my robots.txt?
Usually within minutes of the next crawl. But changes can take 24 hours to fully propagate across all of Google's systems.
Should I have a sitemap if I only have 10 pages?
Yes. Even small sites benefit, sitemaps give you a definitive list Google works from, instead of relying on internal linking alone.
What about XML sitemap vs HTML sitemap?
XML is for search engines. HTML (a page that lists your URLs in human-readable form) is for users. Both can be useful, but XML is far more important for SEO.