Robots.txt is a text file in the root directory of your website that tells search engines which parts they are allowed to crawl. A properly configured robots.txt prevents Google from wasting time on unimportant pages such as the admin area or search results, and ensures your actual content gets priority. Below you’ll find how to create the file yourself, test it, and adjust it.
What robots.txt does and doesn’t do
Robots.txt gives crawlers guidelines about which folders and pages they may visit. It’s especially useful for directing crawl budget to important content and keeping search engines away from admin folders, filter pages, or internal search results. What the file does not do: truly hide content. A page you block with Disallow can still sometimes appear in search results if it’s linked to elsewhere, just without a description. For pages that absolutely must not be indexed, use a noindex meta tag instead of (or alongside) robots.txt.
Creating robots.txt in WordPress
Before you begin, first check that WordPress itself isn’t blocking indexing. Then you create the file manually for full control.
Check visibility setting
Go to Settings → Reading in the WordPress dashboard and make sure ‘Discourage search engines from indexing this site’ is disabled.
Open file manager
Log in to your hosting account and open file manager or an SFTP connection. Navigate to the public_html folder, the root directory of your website.
Create robots.txt
Create a new text file with the exact name robots.txt, without any additional extension, and place it in the root directory.
Add rules and save
Paste the desired configuration (see examples below) into the file and save it.
Check the result
Open yoursite.com/robots.txt in your browser to check whether the file is correctly visible.
Practical examples
A simple, safe base configuration for most WordPress sites:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?s=
Disallow: /trackback/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Sitemap: https://yoursite.com/sitemap.xml
Explanation of the main rules:
Disallow: /wp-admin/ blocks the admin panel, but Allow: /wp-admin/admin-ajax.php keeps a file accessible that many plugins and themes need to function correctly. Disallow: /?s= prevents internal search results from being indexed as duplicates. Allow: /wp-content/uploads/ ensures images remain findable in Google Images. The Sitemap rule points crawlers to your XML sitemap so they find new content faster. Avoid blocking /wp-content/plugins/ or entire theme folders, since some CSS and JS files within them are needed to correctly render pages for Google.
Testing and checking
After each change, check that the file works as intended. Open your robots.txt URL directly in the browser and review the contents. Also use Google Search Console to see whether important pages are actually being crawled and whether no unintended blocks occur. Regularly check, especially after installing new plugins, changing permalinks, or adjusting your sitemap, whether the rules still hold up. For questions about server settings or Plesk, you can always visit the knowledge base of Tandata.
Frequently asked questions
Can I fully hide content from Google with robots.txt?
No, robots.txt blocks crawlers but does not fully hide content. For truly sensitive information, use HTTP authentication or a noindex meta tag.
Does WordPress already have a default robots.txt?
If no custom file is present, WordPress automatically generates a simple version. A manual file, however, gives you full control.
Do I need to update robots.txt often?
Not continuously, but do check it after major changes to your site, such as new plugins, a different sitemap, or structural changes.
Does robots.txt also block my email or other services?
No, robots.txt only affects web crawlers of your website. Settings for business email are completely separate from this.
Does robots.txt work the same with a new domain?
Yes, the principle remains the same. When registering domain names or when moving via switching to Tandata, you simply create a new robots.txt in the root directory.
Conclusion
A correctly configured robots.txt helps search engines crawl efficiently and protects non-public parts of your WordPress site from unnecessary indexing. Start with a simple, safe base configuration, check the result in the browser and Search Console, and adjust the rules as your site changes.