Real conversation. Actual client. Last year.
Me: “Your entire website is blocked from Google.”
Client: “That’s impossible. I can see it fine in Chrome.”
Me: “Yes. You can see it. Google cannot.”
Client: “But our developer said it was fine.”
Me: “Your developer also left Disallow: / in your robots.txt for eleven weeks.”
(Silence.)
(More silence.)
“What’s a robots.txt?”
Right. Let’s fix that.
Table of Contents
What Is Crawlability and Why Does It Matter?
Before Google can rank your page, it has to read it. Before it can read it, it has to find it. Before it can find it, it has to be allowed in.
That entire process — finding, visiting, and reading your pages — is called crawling. Google deploys automated bots (called Googlebot) that continuously travel the web, following links from page to page, collecting content, and feeding it back to Google’s index.
The index is the database. Ranking comes from the index. No crawl means no index. No index means no ranking. No ranking means no traffic.
It really is that linear.
Crawlability is simply the answer to the question: can Google actually access my pages without anything getting in the way?
Sounds basic. You’d be surprised how many things get in the way.
What Is Robots.txt?
Robots.txt is a plain text file that lives at the root of your website — accessible at yourwebsite.com/robots.txt.
It’s one of the first things Googlebot checks when it visits your site. Think of it as the front door policy. The sign that tells visitors — specifically automated crawlers — where they’re allowed to go and where they’re not.
The file uses a simple syntax to either allow or disallow crawlers from accessing specific parts of your site. You can write rules for all bots or target specific crawlers by name.
It does one job: tells crawlers what to crawl and what to skip.
When working correctly, it’s invisible. When broken, it can silently kill your entire online presence — and the terrifying part is your website will still look completely normal to anyone visiting it in a browser.
How Robots.txt Works — The Syntax Explained
The file uses a few basic directives:
User-agent — specifies which crawler the rule applies to. * means all crawlers.
Disallow — tells the crawler not to visit this path.
Allow — explicitly permits access to a path, even if a broader Disallow rule would otherwise block it.
Sitemap — points crawlers to your XML sitemap location.
A clean, standard robots.txt for most websites looks like this:
textUser-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.com/sitemap.xml
Translation: “All crawlers are welcome everywhere, except our admin dashboard. The sitemap is here if you need it.”
Simple. Clean. Non-catastrophic.
Now here’s the version that ends careers:
textUser-agent: *
Disallow: /
Translation: “All crawlers, please leave immediately. Don’t look at anything. Goodbye.”
One character — the / — blocks your entire website. Every page. Every post. Every product listing. Every piece of content you’ve ever published. All of it, invisible to Google.
And the truly painful part? Your website loads perfectly in every browser. Your users can see it just fine. You can see it just fine. The only visitor who’s been politely told to leave is the one whose opinion determines whether you appear in search results.
What You Should (and Absolutely Should Not) Block
Things you typically SHOULD block:
/wp-admin/— your CMS admin panel/cart/,/checkout/— for e-commerce sites; these have no search value/search?— internal search result pages that create thousands of low-value indexed URLs/tag/,/author/— depending on your WordPress setup, these may be thin archive pages- Staging subdirectories or staging subdomains (via a separate staging robots.txt, ideally)
Things you should NEVER block:
- CSS and JavaScript files — Google needs these to render your pages correctly; block them and Googlebot sees broken pages that look nothing like what users see
- Your main content pages, category pages, or product pages — obvious, but
Disallow: /products/is a thing I have seen - Your sitemap — blocking your own sitemap from crawlers is a special level of self-sabotage
- Image directories — if images are part of your content strategy or you care about Google Image Search, blocking
/images/removes you entirely from it
The Most Dangerous Robots.txt Mistakes Subu Has Seen
In no particular order, from fourteen years of auditing things that should not exist:
The Staging Disaster — Disallow: / copied from a staging environment into production. The single most common, single most devastating robots.txt error. Happens more than you’d believe. Always check before launch.
The Wildcard Massacre — A developer trying to block one folder accidentally writes Disallow: /p — which blocks every URL starting with /p. The /products/, /pages/, /press/, /portfolio/ folders. Gone.
The CSS/JS Block — Disallow: *.js or Disallow: /wp-content/ — blocking all JavaScript and CSS files. Google renders your pages without styling or functionality and has absolutely no idea what it’s looking at.
The “Someone Mentioned Duplicate Content So I Blocked Everything” — a site blocks an entire /blog/ section from crawling because “someone said duplicate content is bad.” This is not how you fix duplicate content. Please don’t do this.
The Ghost robots.txt — a large website with no robots.txt at all, allowing every crawler including the aggressive and irrelevant ones to crawl every session ID, every filtered URL, and every internal search result page without restriction. Crawl budget destruction in real time.
How to Test Your Robots.txt Right Now
Three ways, in order of effort:
Method 1: Subu’s Robots.txt Checker
Yes, I built a tool for this. Because I got tired of watching people find out about their blocked websites the hard way.
Paste your website URL into Subu’s Robots.txt Checker and it’ll instantly fetch your live robots.txt file, highlight any Disallow rules that might be blocking important pages, and flag the classic disasters — including the Disallow: / death sentence — before they cost you three months of rankings.
Takes about four seconds. Go do it now. I’ll wait.riority.
Method 2: Just Go Look At It
Type yourwebsite.com/robots.txt into your browser. If you see the file, read it. If you see anything that looks like Disallow: / — stop everything.
Method 3: Google Search Console
Go to Settings → Robots.txt in Search Console. Google shows you the live file and provides a tester where you can input any URL to see if it’s currently blocked. Use this. It’s free. It’s authoritative.
Crawlability Beyond Robots.txt
Robots.txt is the most famous crawlability lever but it’s not the only one.
Other things that prevent Google from crawling your pages:
- Noindex tags — these don’t prevent crawling but prevent indexing; a crawled-but-noindexed page costs crawl budget while delivering nothing
- Broken internal links — pages only discoverable through links that return 404s become orphaned and invisible
- JavaScript-gated content — navigation or links that only appear after user interaction can’t be followed by Googlebot (we covered this in the JavaScript SEO section of The Tech Manual)
- Login walls — any content that requires authentication before serving is invisible to Google by design
- Redirect chains — Google may give up following a long redirect chain before reaching the final destination
Full crawl health is the intersection of all of these working correctly at once.
The TL;DR
- Robots.txt tells Google what it’s allowed to crawl — it’s the first thing Googlebot checks on your site
Disallow: /blocks your entire website from Google — check your robots.txt before and after every launch or site change- Block admin panels, internal search, and checkout pages — never block CSS, JavaScript, or your main content
- Test using Google Search Console’s built-in robots.txt tester or Screaming Frog
- Crawlability problems extend beyond robots.txt — broken links, JavaScript issues, and login walls all affect Google’s ability to reach your pages
Your robots.txt has enormous power over whether your website exists in Google’s eyes. Treat it accordingly.
Sitting in your Search Console right now wondering why Google can’t see half your site? Drop the URL in the comments. Subu will take a look.
— Subu, SEO by Subu
