robots.txt and sitemap.xml Explained for Site Owners

Updated 2026-10-28 · 5 min read · By Kristina, digital artist and web developer

The robots.txt file tells search engine crawlers which parts of your site they may visit, and the sitemap.xml file lists the pages you want them to find. Both are small text files placed at the root of your website. If they are missing or wrong, search engines may miss your pages or avoid your whole site.

What robots.txt does

When a crawler arrives at your site, it first looks for a file at yourdomain.com/robots.txt. This file contains simple rules: which crawlers the rules apply to and which paths they should not visit. It is a request, not a lock. Well-behaved crawlers follow it, but it does not protect private information. Anything that must stay private should be protected by a login, not by this file.

A very simple robots.txt looks like this:

The first line says the rules apply to all crawlers. The empty "Disallow" line means nothing is blocked. The last line tells crawlers where the sitemap is.

The mistake that hides whole sites

A single line can block everything:

This is sometimes used while a site is under construction. If it stays in place after launch, search engines will not crawl your pages. If your site is missing from search, this is the first thing to check. See why your website may not be on Google.

What sitemap.xml does

A sitemap is a list of the addresses of your pages, in a format that machines read. It does not make your pages rank better, but it helps search engines discover them, especially on new sites or sites with many pages. It can also carry the date when each page was last changed.

A minimal entry looks like this:

Most builders and content systems create the sitemap automatically. If you build your own site, many tools can generate it for you.

What belongs in the sitemap

• Pages you want people to find: services, articles, portfolio pages.
• Only the final addresses, not pages that redirect elsewhere.
• Only pages that work. A sitemap full of broken links is less useful and may produce errors in the reports.

Leave out thank-you pages, test pages and pages with no value for search.

How to check your files

1. Type your domain followed by /robots.txt in the browser. You should see a text file.
2. Type your domain followed by /sitemap.xml. You should see a list of addresses, or a list of other sitemaps.
3. Read the robots file for any line that blocks all pages.
4. Submit the sitemap in the free search tools for site owners and look at the report. It will tell you how many pages were found and whether there are errors.

When to update them

If your system creates the sitemap automatically, it updates itself. If you maintain it manually, update it when you add or remove pages. Check the robots file after changing your host, theme or platform, because these changes sometimes replace it with a default version.

Other helpful signals

These files are only a part of the picture. Search engines also rely on links between your pages, so make sure every page is reachable from another. Use clear titles and descriptions, and keep one main version of every address. The overall approach is described in what SEO is, and a pre-launch routine in the website launch checklist.

A note on AI crawlers

Some companies that build AI assistants and search tools use their own crawlers, which you can allow or block in the robots file by naming them. Whether to allow them is your choice. Blocking them may mean your content is not used by those tools, and allowing them may mean it is, but check each tool's documentation for what it actually does. Check the documentation of each tool for the exact crawler names.

Common questions when you set the files up

Many site owners ask whether they should block certain folders, such as pages for administration or search results inside the site. It is reasonable to block areas that have no value for search, such as login pages, as long as you do not block pages that you want people to find. Be careful with blocking style and script files, because search engines use them to understand how a page looks. When in doubt, block less.

Another frequent question is the difference between blocking crawling and blocking indexing. A robots rule asks a crawler not to visit a page. A "noindex" instruction on the page itself asks search engines not to show it. If you want a page kept out of search results, the second is the usual tool, and the page must not be blocked in robots.txt, or crawlers will never see the instruction. A page blocked only by robots.txt can still be listed if other sites link to it.

A short routine

• After launch, open both files in your browser and read them.
• Submit the sitemap in the search tools once.
• Look at the report after some days and fix any errors it lists.
• Check again after any change of platform, theme or host.

FAQ

Do I need a robots.txt file?

It is not required. Without it, crawlers assume they can visit everything. It is still useful for pointing to your sitemap.

Does a sitemap improve my ranking?

Not directly. It helps search engines find your pages, which is the first step.

Can I use robots.txt to hide private pages?

No. The file is public and only asks crawlers to stay away. Use a login to protect private content.

Where do I put the files?

At the root of the site, so that they open at your domain followed by the file name.

How do I know if the sitemap works?

Submit it in the search tools and read the report. It shows the pages found and any errors.