19. Robots.txt | SEO Forge - Rank Higher with AI-Powered SEO

SEO Forge provides a virtual robots.txt via the WordPress robots_txt filter. FREE feature.

How It Works

Hooks into the robots_txt WordPress filter at priority 10
Uses custom content from seoforge_robots_txt option if set; otherwise uses WordPress defaults
Automatically appends Sitemap: directive pointing to your sitemap index

Customization

// Set custom robots.txt content
update_option( 'seoforge_robots_txt', "User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /" );

// Remove custom robots.txt (revert to default + sitemap)
delete_option( 'seoforge_robots_txt' );

Extending via Filter

// Block AI training crawlers
add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "nUser-agent: GPTBotnDisallow: /n";
    $output .= "nUser-agent: ChatGPT-UsernDisallow: /n";
    $output .= "nUser-agent: Google-ExtendednDisallow: /n";
    return $output;
}, 20, 2 );

// Block everything on staging
add_filter( 'robots_txt', function ( $output, $public ) {
    if ( defined( 'WP_ENVIRONMENT_TYPE' ) && WP_ENVIRONMENT_TYPE === 'staging' ) {
        return "User-agent: *nDisallow: /n";
    }
    return $output;
}, 20, 2 );

—