Back to Hub
SEO Indexing Toolkit

Robots.txt & XML Sitemap Generator

Generate clean, search engine-compliant robots.txt directives and structured XML sitemaps to accelerate Google indexation and control crawler access.

Crawler Directives
robots.txt Preview UTF-8 Text 


            

                
                
            

        

    


    
    

    
    

      
The Ultimate Developer Guide to Robots.txt Directives & XML Sitemaps

      
      

        In modern technical search engine optimization (SEO) and web administration, controlling how search bots crawl and index your site is vital for indexing efficiency and crawl budget allocation. Robots.txt acts as the gatekeeper for web crawlers (such as Googlebot, Bingbot, and AI bots), while sitemap.xml provides a structured roadmap of canonical URLs.
      

      

        Our Robots.txt & XML Sitemap Generator operates 100% client-side inside your browser. No website URLs, private directories, or staging routes are ever uploaded to remote servers.
      


      
1. Robots.txt Common Directives & Bot Tokens Reference

      

        
          

            Directive / Crawler
            Syntax Example
            Action / Purpose
            Supported By
          

        
        
          

            User-agent: *
            User-agent: *
            Applies following rules to all web spiders
            All standard search crawlers
          

          

            Disallow Directory
            Disallow: /admin/
            Blocks crawling of the specified path and sub-paths
            Google, Bing, Yahoo, Yandex, Baidu
          

          

            Allow Subpath
            Allow: /admin/public/
            Explicitly whitelists a sub-URL inside a disallowed parent
            Googlebot, Bingbot
          

          

            Block AI Bots
            User-agent: GPTBot
Disallow: /		
            Prevents AI training bots from scraping website content
            OpenAI (GPTBot), Anthropic (ClaudeBot)
          

          

            Sitemap Directive
            Sitemap: https://site.com/sitemap.xml
            Directs crawlers to the canonical XML sitemap location
            All modern search engines
          

        
      


      
2. XML Sitemap Schema Tags Explained

      

      
3. Technical SEO Best Practices

      

        
Crawl Optimization Checklist:

        
1. Never disallow CSS, JavaScript, or theme assets in robots.txt—Googlebot needs them to render pages for mobile indexing.

        
2. Keep sitemap file sizes under 50MB and fewer than 50,000 URLs per individual sitemap file (use a sitemap index for larger sites).

        
3. Submit your sitemap directly to Google Search Console and Bing Webmaster Tools for rapid URL discovery.

      

    


    
    

      
Frequently Asked Questions about Robots.txt & Sitemap Generator

      

        

          What is the primary difference between robots.txt and sitemap.xml?
          

            robots.txt tells search engine bots which directories or URLs they are forbidden or allowed to crawl. An XML sitemap provides a structured list of canonical URLs that you actively encourage search engines to discover and index.
          

        

        

          Where should robots.txt and sitemap.xml be hosted on a website?
          

            Both files should reside at the root of your domain: https://example.com/robots.txt and https://example.com/sitemap.xml so automated web crawlers can find them instantly.
          

        

        

          How do I block AI scrapers like GPTBot and CCBot in robots.txt?
          

            Specify the user-agent and add a disallow directive: User-agent: GPTBot followed by Disallow: / to prevent OpenAI crawlers from indexing your site content.
          

        

        

          Does disallowing a page in robots.txt guarantee it will not appear in Google search results?
          

            No. robots.txt blocks crawling, not indexing. If other websites link to that disallowed URL, Google may still index the URL without crawling the page content. To completely prevent indexing, use a noindex meta tag.
          

        

        

          Is my site URL data private when using this generator?
          

            Yes, 100%. The tool executes completely client-side in your local browser sandbox without transmitting your domain or URL lists to external servers.