Robots.txt Configuration

Guide search engine crawlers towards the pages that matter and away from low-value URLs, without accidentally blocking your important content.

Review My Robots.txt

A Small File With Big Consequences

Every website can have a robots.txt file at the root of its domain, for example yourdomain.com/robots.txt. It is a plain text file containing instructions for automated crawlers, telling them which parts of the website they may or may not request. Search engines such as Google and Bing check this file before crawling, and most reputable crawlers respect it. Used well, robots.txt helps search engines spend their crawling effort on your useful pages rather than on endless filter combinations, internal search results, admin areas or other low-value URLs.

Because the file is so powerful and so simple, small mistakes can have serious effects. A single line such as Disallow: / can block an entire website from being crawled. This happens surprisingly often when a development site’s settings are carried over to the live site at launch. Other common errors include blocking the CSS and JavaScript files that search engines need to render pages properly, blocking pages that are meant to rank, or relying on robots.txt to keep private information hidden, which it cannot do. Equally, a robots.txt that blocks nothing at all may let crawlers waste time on thousands of unhelpful URLs on a large site.

ViV Softech Solution is a website designing and digital marketing agency based in Delhi. We audit, write and test robots.txt files for business websites, ecommerce stores and portals across India. This page explains how robots.txt works, what it can and cannot do, the difference between blocking crawling and preventing indexing, sensible rules for WordPress and ecommerce sites, the errors we most often find, how to handle AI and other crawlers, how we test changes and what affects cost. It is part of our technical SEO services.

What a Good Robots.txt Achieves

Important Pages Open

Service, product, category and content pages remain fully crawlable by search engines.

Low-Value URLs Managed

Internal search, endless filters and system paths are kept out of crawlers’ way where appropriate.

Rendering Resources Allowed

CSS, JavaScript and images needed to display pages are not blocked.

Sitemap Referenced

The location of your XML sitemap is listed so crawlers can find it easily.

Valid and Tested

Rules use correct syntax and are tested against real URLs before going live.

Simple and Documented

Rules are kept short and clear, with comments explaining why each one exists.

How Robots.txt Works

User-agent

Each group of rules starts by naming which crawler it applies to. An asterisk means all crawlers, while names such as Googlebot or Bingbot target specific ones. A crawler follows the most specific group that matches it.

Disallow and Allow

Disallow lists paths the crawler should not request; Allow can make exceptions within a disallowed section. For example, WordPress sites commonly disallow the admin area but allow the admin-ajax file some themes and plugins need.

Pattern matching

Major search engines support the asterisk as a wildcard and the dollar sign to mark the end of a URL, which allows rules such as blocking URLs containing a particular parameter. Patterns need care, because an overly broad rule can block far more than intended.

Sitemap

A Sitemap line gives the full URL of your XML sitemap or sitemap index. It can appear anywhere in the file. See XML sitemap setup.

Robots.txt applies per host and protocol, so subdomains such as shop.yourdomain.com need their own file. Google treats the file as case-sensitive for paths, and it caches the file for a period, so changes are not always picked up instantly.

Blocking Crawling Is Not the Same as Preventing Indexing

This is the most misunderstood point about robots.txt. Disallowing a URL tells search engines not to crawl it, meaning not to fetch its content. It does not reliably keep the URL out of search results. If other pages link to a blocked URL, a search engine may still index the address and show it, usually with little or no description, because it knows the page exists but cannot read it.

To keep a page out of search results, use a noindex meta tag or HTTP header, and leave the page crawlable so search engines can see the noindex instruction. Blocking it in robots.txt would hide the noindex from them.
To protect private content, use password protection or proper access controls. Robots.txt is publicly visible to anyone, so listing secret paths in it can actually draw attention to them.
To remove something urgently from Google, Search Console offers a temporary removal tool, used together with a permanent fix such as noindex or deletion.
To manage duplicate URLs, canonical tags and redirects are often better than blocking, because blocked pages cannot pass their signals. See canonicalization.

Sensible Rules for WordPress and Ecommerce Sites

WordPress websites

WordPress generates a simple virtual robots.txt by default, which SEO plugins can modify. For most business sites, a minimal file is best: disallow the admin area while allowing the admin-ajax file, and reference the sitemap. Older advice to block folders such as wp-includes or wp-content can stop search engines loading the CSS, JavaScript and images they need to render pages, so we generally avoid it. Internal search result URLs are better handled with noindex, though some large sites also disallow them.

Ecommerce stores

Stores can generate huge numbers of URLs through filters, sorting, comparison and wishlist functions. Carefully written rules can stop crawlers wasting effort on sort orders and certain filter parameters, while allowing valuable category and filtered pages you want indexed. Cart, checkout and account pages are usually disallowed or noindexed. Because a wrong pattern can hide products, we test every rule against a full list of real store URLs before making it live.

Common Robots.txt Errors We Find

Disallow: / left in place after launch, blocking the entire site.
Blocked CSS, JavaScript or image folders, preventing proper page rendering.
Important sections blocked by a broad pattern written for something else.
Pages blocked and noindexed at the same time, so the noindex is never seen.
Missing or outdated sitemap line, pointing to an old or wrong sitemap.
Robots.txt returning a server error, which can cause search engines to pause crawling of the whole site.
Different rules on www and non-www or http and https versions.
Sensitive paths listed publicly in the belief that this hides them.
Syntax mistakes such as missing colons, wrong case in paths or rules placed outside a user-agent group.
Crawl-delay directives that Google ignores, used instead of fixing real server capacity issues.

AI Crawlers and Other Bots

Besides search engines, many other crawlers visit websites, including SEO tools, price comparison services, archive services and crawlers used to gather content for AI systems. Several AI companies publish the names of their crawlers and state that they respect robots.txt, so businesses can choose to allow or block them individually. This is a business decision rather than a purely technical one: blocking some crawlers may reduce the chance of your content being used in certain AI tools, while allowing them may help your business appear in AI-generated answers. The landscape changes frequently, and crawler names and policies are updated over time. We explain the current options, implement your choice, and note that robots.txt is voluntary, so badly behaved bots that ignore it need to be handled at the server or firewall level instead. Blocking Googlebot or Bingbot themselves would, of course, affect your search visibility.

Staging, Development and Multiple Environments

Development and staging copies of a website should not appear in search results. Many teams rely on a robots.txt disallow for this, but as explained above, that does not reliably prevent indexing, and it is easily copied to the live site by mistake during launch. A safer approach is password protection on staging environments, optionally combined with noindex. On launch day, we check that the live robots.txt allows crawling, that WordPress’s discourage search engines setting is switched off, and that no noindex tags have been carried over. These checks form part of our launch and website migration checklists, because blocking a new site by accident can delay its visibility for weeks.

How We Test Changes

Before any change goes live, we test the proposed rules against a list of your real URLs, including important pages that must stay open and low-value URLs meant to be blocked, using robots.txt testing tools and parsers that follow search engine behaviour. After publishing, we check the robots.txt report in Google Search Console, which shows the version Google fetched and any errors, and use the URL inspection tool to confirm key pages are crawlable. We then monitor crawl statistics and the page indexing report over the following weeks for unexpected drops. We keep a copy of the previous file so any change can be reversed immediately if needed.

Our Process

1. Audit: read the current file on every host and protocol and check Search Console reports. 2. Crawl: identify which URL patterns exist and which waste crawl effort. 3. Plan: agree which areas to allow, disallow, noindex or canonicalise. 4. Write: a minimal, commented robots.txt. 5. Test: against real URLs before publishing. 6. Publish: through your SEO plugin, server or platform settings. 7. Monitor: crawl stats and indexing for several weeks.

What Affects Cost

Website size and URL complexity.
Ecommerce filters and parameters to analyse.
Number of hosts and subdomains.
Related fixes such as noindex, canonical or sitemap changes.
Monitoring period after changes.

Frequently Asked Questions

Not reliably. Use noindex on a crawlable page to keep it out of results, or password protection for private content.

It is not mandatory, but a simple file that references your sitemap and blocks only clearly unnecessary areas is good practice.

Generally no, because search engines need CSS, JavaScript and images from these folders to render your pages.

No. The file is public and voluntary. Use proper authentication for anything private.

It is a business decision. We explain the current options and trade-offs and implement your choice.

Search engines cache robots.txt, so changes usually take effect within a day or so, though it can vary.

It is one of the first things to check, along with noindex tags and WordPress visibility settings. Contact us if you need urgent help.

Related Services

Make Sure Crawlers Can Reach What Matters

Share your website. We will review your robots.txt and propose safe improvements with a quote.

Call Now: +91 8808577444

Send an Enquiry