Free tool

Robots.txt Generator

Build a valid robots.txt with per-crawler rules, crawl-delay and sitemap directives. Copy or download, free and in-browser.

Rule group 1

robots.txt

User-agent: *
Disallow:

Place this file at the root of your domain: /robots.txt

Note: robots.txt controls crawling, not indexing. To keep a page out of Google, use a noindex meta tag, not just a Disallow rule.
Getting crawl and indexing right is foundational technical SEO. Our SEO services cover it end to end.

How robots.txt works

A robots.txt file tells crawlers where they can and can't go. Each rule group names a User-agent (use * for all) and lists Disallow and Allow paths. You can target specific bots differently, for example, blocking an aggressive scraper while letting Googlebot roam freely.

The single most important thing to remember: robots.txt governs crawling, not indexing. If you truly need a page kept out of search results, use a noindex directive rather than a Disallow. For a crawl and indexing strategy done properly across a whole site, see our SEO services.

Frequently asked questions

robots.txt is a plain-text file at the root of your domain (yoursite.com/robots.txt) that tells search-engine crawlers which parts of your site they may or may not crawl. It's the first file most crawlers request.

Other free tools

A growing kit of free utilities for builders. No sign-up required.

Smart Contract Vulnerability Scanner

Paste your Solidity code and get an instant heuristic scan for the OWASP Smart Contract Top 10 vulnerabilities.

Smart Contract Audit Cost Estimator

Estimate what a professional smart contract audit will cost based on your codebase size and complexity.

Gwei / ETH / USD Gas Fee Converter

Convert between Gwei, ETH and USD to understand transaction and deployment gas costs.

Keccak-256 Hash Generator

Generate the Keccak-256 (Ethereum SHA-3) hash of any string or function signature.

Unix Timestamp Converter

Convert between Unix timestamps and human-readable dates, handy for block times and vesting schedules.

Wallet Address Validator

Check whether an Ethereum address is well-formed and EIP-55 checksum valid.

Meta Tag Generator

Generate SEO title, description, Open Graph and Twitter tags with a live Google search-result preview.

Schema Markup (JSON-LD) Generator

Generate FAQ, Article and Breadcrumb JSON-LD structured data to paste into your pages.

Ready to turn on-page SEO into traffic?

Crawl control is foundational. We handle technical SEO, content and structured data end to end.

Explore our SEO servicesBook a free consultation
Technical SEOContent that ranksStructured dataMeasurable results