SEO Forge lets you edit your WordPress site’s robots.txt directly from the admin — without uploading a physical robots.txt file via FTP. The plugin uses WordPress’s built-in robots.txt filter to serve your custom rules dynamically, which is cleaner, safer, and easier to manage than maintaining a file on disk.
The robots.txt file sits at the root of your site (example.com/robots.txt) and tells search engine crawlers which parts of your site they’re allowed to crawl. It’s one of the most important and most commonly misunderstood pieces of SEO configuration.
A correct robots.txt blocks crawlers from wasting your crawl budget on pages that shouldn’t be indexed (admin pages, search result pages, feed URLs, parameter-heavy URLs). A wrong one accidentally blocks pages you do want indexed — which is how entire sites drop out of Google overnight.
Edit robots.txt from within the WordPress admin. No file manager, no wp-content write permissions, no accidentally creating a file in the wrong location.
A physical robots.txt file can be overwritten or deleted during migrations, hosting changes, or core updates. A virtual one generated by SEO Forge is tied to your plugin settings and persists through all of that.
SEO Forge automatically appends the current sitemap URL (Sitemap: https://example.com/sitemap_index.xml) to your robots.txt. If you change the sitemap structure later, robots.txt updates automatically. A physical file would need manual editing.
The admin editor shows the current robots.txt content alongside syntax validation and rule suggestions. Much easier to review than a raw file on the server.
The SEO Forge → Robots.txt admin page shows:
/wp-content/uploads/ which hides your images)The defaults SEO Forge loads are:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Sitemap: https://example.com/sitemap_index.xmlThis blocks the admin area from crawling (saving crawl budget), allows AJAX calls (needed for some plugins), blocks WordPress core includes (never useful to crawlers), and points to the sitemap.
A classic mistake: Disallow: /wp-content/uploads/. This blocks Google from crawling your images, which kills Image Search traffic. SEO Forge’s validator warns you.
Disallow: / blocks everything. SEO Forge’s validator flags this as a critical warning.
If SEO Forge’s XML sitemap is active but robots.txt doesn’t reference it, the validator suggests adding the Sitemap: line (or does it for you automatically).
Google needs to render pages fully to understand them. Blocking /wp-content/themes/ or /wp-includes/js/ breaks rendering. SEO Forge’s validator warns on these patterns.
Robots.txt is case-sensitive on most servers. Disallow: /Admin/ does not block /admin/. SEO Forge’s validator catches case mismatches.
Block or allow specific crawlers by name:
User-agent: Googlebot
Disallow: /example-page/
User-agent: Bingbot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /Popular crawlers to consider blocking (depending on your preference):
SEO Forge provides a “AI Training Crawlers” toggle in the admin that adds recommended blocks for this category of bot with one click.
After saving changes, test with:
curl https://example.com/robots.txtconfirm the file is being served correctlyGet SEO Forge — from $39/year →
The Robots.txt Editor is included in every version of SEO Forge, including the free one.