Technical SEO: The Foundation You’re Probably Ignoring
Okay, story time.
A client came to me once with a brand new website. Six months in the making. Not a small amount of money spent. Beautiful homepage. Great content. The kind of website that gets you compliments at dinner parties.
Three months after launch, zero organic traffic.
I ran a crawl. Found the issue in four minutes flat.
User-agent: *
Disallow: /
That means: “Dear Google, please don’t crawl any page on this website. None of them. We’d like to remain invisible, thanks.”
The developer had forgotten to remove the staging crawl block before going live. For ninety days, Google had been politely walking away from their front door, because the sign on the door said GO AWAY.
When I told the developer, they literally said:
“But the site loads fine on my browser.”
Yes. Yes it does. That is not the problem.
Welcome to Technical SEO, the layer of your website that Google actually cares about before it ever reads a single word of your content. The part that your developer doesn’t think about and your content writer doesn’t know exists. The part that, when broken, makes everything else you do completely pointless.
This is The Tech Manual. By the end of it, you’ll know exactly how to stop your website from silently sabotaging itself.
What Is Technical SEO, Actually?
Technical SEO is the process of making sure Google can find your website, crawl it, understand it, and decide it’s worth indexing and ranking. That sounds simple. It is not simple.
Find & Crawl
Can search engines discover your pages through links and navigate your site architecture without getting trapped?
Understand & Rank
Is your code rendering correctly so Google knows what it’s about, and is the site fast, secure, and mobile-friendly enough to win?
Here’s the thing about Google’s crawlers, they’re not humans. They don’t see your beautiful homepage the way you do. They don’t admire your font choices or appreciate your hero image. They’re bots. They read code, follow links, and make decisions based entirely on what your server tells them.
If your server is confusing or accidentally blocking Google, you don’t rank. Doesn’t matter how good your content is. Technical SEO is fixing that foundation.
It covers everything from how your site is structured and how fast it loads, to whether your HTTPS certificate is valid, whether your sitemaps are accurate, and whether your schema markup is telling Google the right story about your content.
It is, without question, the least glamorous part of SEO. It is also the most important.
What’s In This Guide
This is your complete Technical SEO command centre. Each section below covers one critical pillar of technical health. Read the overview, then hit the full guide for the deep dive.
🛠️ Subu’s Toolkit: Free Technical SEO Audit
Before you read the guide, find out exactly what’s broken on your site right now. Run a complete technical and on-page analysis in seconds.
Let’s go.
1. Crawlability & Robots.txt
The part where you make sure Google is actually allowed inside.
Let’s start with the most fundamental question in all of SEO: can Google even get to your website?
Before Google can read a single word, rank a single page, or send you a single visitor, it has to be able to crawl you. Crawling is how Google discovers your pages. Its bots (called Googlebots) follow links, read your pages, and add them to Google’s index. No crawl? No index. No index? No rankings. No rankings? No traffic. No traffic? We have a very serious conversation.
Your robots.txt file is the first thing Googlebot checks when it visits your site. It’s a small text file that lives at yourwebsite.com/robots.txt and tells crawlers which parts of your site they’re allowed to visit, and which parts they should skip entirely.
Hover or tap to block the crawler
Used correctly, robots.txt is a powerful tool. It can protect staging environments, keep admin panels private, and prevent crawlers from wasting time on low-value pages. Used incorrectly, as we’ve established, it can block your entire website from Google for three months while you wonder why nobody is showing up.
Here are the common robots.txt disasters I have personally witnessed across 14 years of doing this.
Disallow: /pushed live from a staging environment (the classic)- High-value pages accidentally blocked because someone got a wildcard wrong
- CSS and JS files blocked from crawlers, making pages render completely differently to Google than to users
- No robots.txt whatsoever on a site with hundreds of URLs that absolutely should not be indexed
The golden rule: check your robots.txt before you do anything else. It takes ten seconds. Go to yourwebsite.com/robots.txt right now. I’ll wait.
🛠️ Subu’s Toolkit: Robots.txt Tester
Are you accidentally blocking Google? Paste your URL below and test your robots.txt file for syntax errors, logic flaws, and accidental blocks.
2. XML Sitemaps
Your website’s GPS. Without it, Google is just guessing.
Once Google is allowed in, you want to make its job as easy as possible. That’s what a sitemap does.
An XML sitemap is a file that lists every important URL on your website, your pages, posts, product listings, whatever, and tells Google: “These are the pages I want you to know about. These are the ones that matter.”
Think of it as the table of contents for your website, specifically written for search engines. Without a sitemap, Google still finds your pages, it’s good at following links. But with a sitemap, you’re proactively telling it where to go, which pages exist, when they were last updated, and how they’re prioritized. You’re not waiting for Google to stumble across things. You’re guiding it.
Hover or tap to parse sitemap.xml
This matters especially for large websites with thousands of pages, new websites with limited backlinks, or content buried deep in your site architecture that crawlers might not reach naturally.
Common sitemap sins I’ve seen in the wild:
- Sitemaps that include 404 pages, redirected URLs, and noindexed pages, you’re advertising your broken pages to Google
- Sitemaps that haven’t been updated since 2021 and are missing half the site
- Sitemaps that exist but were never actually submitted to Google Search Console
- No sitemap at all on a 10,000-page e-commerce site (we had a long conversation that day)
Your sitemap should be clean, current, and submitted. That’s the whole job.
🛠️ Subu’s Toolkit: XML Sitemap Generator
Don’t have a sitemap? Let’s fix that right now. Instantly generate a clean, Google-friendly XML sitemap for your website.
3. Core Web Vitals & Page Speed
Because if your website takes 8 seconds to load, your users left at 4.
Google made it official in 2021: page experience is a ranking factor. And at the centre of page experience are Core Web Vitals, three specific metrics that measure how your page actually feels to a real user loading it.
The three metrics:
- LCP (Largest Contentful Paint): How long does it take for the biggest visible element on the page to load? Google wants this under 2.5 seconds. If your hero image takes 6 seconds to appear, users have already bounced and taken their click-through rate with them.
- INP (Interaction to Next Paint): How quickly does your page respond when someone interacts with it? Clicking a button and staring at nothing is maddening. Google agrees. INP replaced FID in 2024 and measures the full range of interactions across a user’s session.
- CLS (Cumulative Layout Shift): Does your page jump around while it’s loading? That button you tried to click that moved just as your finger touched it? That’s a CLS problem. Google hates it. Users hate it. It’s a universally despised experience and it’s also very measurable and very fixable.
Hover or tap to simulate a bad layout shift
Why does this matter for SEO? Because Google’s job is to send users to pages that give them a good experience. Slow, janky, unstable pages are a bad experience. Sites that consistently give users a bad experience get quietly deprioritised.
And here’s what people miss: Core Web Vitals also directly affect your bounce rate, your session duration, and your conversion rate. This is not just an SEO problem. It’s your entire business.
Read the Full Guide: Core Web Vitals & Page Speed →4. Mobile-First Indexing
Google looks at your mobile site first. If it’s broken on mobile, you are broken.
Here’s something a surprising number of website owners still don’t fully understand: Google indexes the mobile version of your website first. Not the desktop version. The mobile version.
This has been the default since 2023. If your mobile site has missing content, broken images, blocked resources, or a navigation menu that only works with a mouse, that’s what Google is evaluating. That’s the version it’s using to make ranking decisions.
Hover or tap to simulate Googlebot Mobile
I cannot tell you how many times I’ve audited a website where the desktop version looks perfect, and the mobile version is, to put it professionally, a disaster. Missing structured data. Blocked JavaScript. Images that refuse to load. Pop-ups covering the entire screen the moment a user arrives. A menu that requires hovering to operate on a device that doesn’t support hover.
What mobile-first indexing means in practice: Your mobile site needs the same content as your desktop site. All the structured data must exist on mobile. The same robots.txt rules apply. And page speed matters even more.
The shift to mobile-first wasn’t a surprise. Google announced it years in advance. And yet here we are, in 2026, still encountering websites where mobile was clearly built as an afterthought, squeezed in at the end of the project when the budget was almost gone.
Your users are on mobile. Google is on mobile. Time to join them.
Read the Full Guide: Mobile-First Indexing →5. HTTPS & Site Security
The padlock isn’t just for users. Google is checking too.
Short version: if your website still runs on HTTP and not HTTPS, you have a problem. A ranking problem, a trust problem, and a user experience problem, all packed into one unencrypted connection.
HTTPS means your website has an SSL certificate installed. The data between your site and your visitors is encrypted. The padlock appears in the browser. Users feel safe. Google feels safe. Everyone goes home happy.
HTTP (Unsecure)
Hover or tap to intercept
HTTPS (Secure)
Hover or tap to intercept
Google confirmed HTTPS as a ranking signal in 2014. That was over a decade ago. And yet in 2026, I still find HTTP websites. I discover them in the same archaeological layer as Flash animations and “best viewed in Internet Explorer” notices.
Beyond the ranking signal, there’s a more immediate issue: Chrome actively displays “Not Secure” next to HTTP URLs. That warning is not subtle. It is front and centre in the browser bar, actively telling your visitors that this website might not be safe. Watch what that does to your bounce rate.
Site security beyond basic SSL also covers:
- Mixed content issues: your page loads over HTTPS but pulls in images, scripts, or stylesheets over HTTP, triggering security warnings
- Hacked content: if your site gets compromised and starts serving malware or spam links, Google will flag it and warn users away
- Security headers: HTTP response headers that protect against cross-site scripting and clickjacking
- Server reliability: if your site is consistently down when Googlebot visits, that is noted
HTTPS is not an advanced technique. It is not optional. It is the bare minimum entry requirement for being taken seriously in 2026.
Read the Full Guide: HTTPS & Site Security →6. Structured Data & Schema Markup
Telling Google exactly what your content means, not just what it says.
Google can read your content. But reading and understanding are two different things.
When you write “Subu has been doing SEO since 2012,” Google reads it. But does it know that Subu is a person? That 2012 is the start of a career? That this is a professional biography rather than a sentence in a novel?
Hover or tap to render JSON-LD
Schema markup, also called structured data, answers those questions. It’s code you add to your page that explicitly tells Google: “This is a Person. This is a Product with a specific price. This is a review with a 4.7 rating. This is a recipe with fifteen ingredients and a 45-minute cook time.”
🛠️ Subu’s Toolkit: Schema Generator
Don’t write JSON-LD code by hand. Generate perfect schema markup for your pages in seconds.
When Google understands your content at that level of specificity, it can display it differently in search results. Those gold star ratings beneath product listings? Schema. The FAQ accordions that expand directly in search results? Schema. Recipe cards with timing and calorie counts? Schema. Event listings with dates and venues? Schema.
These enhanced displays are called rich results. They make your listing larger, more visually prominent, and more informative, before a user has clicked a single thing.
Schema types that matter most:
- Article / BlogPosting: for editorial content
- FAQPage: triggers the accordion dropdowns directly in search results
- Product: price, availability, reviews, for e-commerce pages
- LocalBusiness: name, address, phone, hours (essential for Local SEO)
- BreadcrumbList: shows your navigation path in the search result
- Person: author markup that directly supports E-E-A-T signals
Structured data isn’t a magic ranking bullet. But it makes your content machine-readable at the deepest level possible. And Google consistently rewards pages that make its job easier.
7. Canonicalization
When the same content lives on multiple URLs, someone needs to be in charge.
Here’s a scenario. You have a product page accessible at:
- yoursite.com/products/red-shoes
- yoursite.com/products/red-shoes/
- yoursite.com/products/red-shoes?color=red
- yoursite.com/products/red-shoes?ref=homepage
To you, these are all the same page. To Google, these are potentially multiple different pages with identical content. Google doesn’t know which one to rank. It might split the link equity between all of them. It might index the wrong one. It might rank none of them with any real confidence.
Hover or tap to consolidate equity
A canonical tag fixes this. It’s a single line in your page’s <head> section that tells Google: “I know this content appears at multiple URLs. But this specific URL is the real one. The canonical one. Please index and rank this one.”
You’re not hiding the other versions. You’re simply appointing a boss.
Where canonical tags go wrong, and they go wrong often:
- Canonical tags pointing to URLs that are themselves redirects or 404s
- Pages with no self-referencing canonical at all
- Canonical tags directly conflicting with hreflang tags on multilingual sites
- Paginated content where every page canonicalizes to page 1
The canonical tag is one line of code. But when it’s missing or wrong, it quietly undermines the ranking authority of your most important pages for months before anyone notices. I have audited sites where mis-configured canonicals were silently leaking equity for over a year.
Read the Full Guide: Canonicalization →8. Redirects (301, 302, and Redirect Chains)
Moving a page without a redirect is like moving house and not telling anyone your new address.
Pages move. Sites get restructured. URLs get cleaned up. Domains change. This is normal. It is also fine, as long as you tell Google where everything went.
A redirect is the forwarding notice. You’re telling Google and your users: “This URL has moved. Here’s where it lives now. Please update accordingly.”
The two you need to know:
- 301 Redirect (Permanent): “This page has moved for good. Update your index. Transfer all ranking signals to the new URL.” This passes the vast majority of link equity from the old URL to the new one.
- 302 Redirect (Temporary): “This page has moved for now. Keep the original indexed. We’ll be back.” The critical mistake: using a 302 when you actually mean a 301. It silently prevents link equity from transferring, sometimes for years.
Then there’s the disaster scenario: redirect chains.
A redirect chain is when URL A redirects to URL B, which redirects to URL C, which redirects to URL D. Each hop loses link equity. Each hop slows the page load. Enough hops and crawlers simply give up and walk away.
Hover or tap to send crawler through the chain
I’ve untangled redirect chains six links deep. I’ve fixed redirects pointing to pages that themselves return a 404. I’ve corrected 302s used for site migrations that happened in 2019 and were never revisited. Redirects are genuinely easy to get right. They are also remarkably easy to get wrong.
🛠️ Subu’s Toolkit: Redirect Chain Checker
Don’t guess if your link equity is leaking. Track the exact path of your URLs and uncover hidden redirect chains instantly.
9. JavaScript SEO
When your beautiful dynamic website is a completely blank page to Googlebot.
This section is for the modern builds. The React deployments, the Next.js sites, the Angular SPAs, the Vue frontends. The beautiful, fast, dynamic websites that load their content via JavaScript.
Here is the problem: Googlebot is not particularly good at JavaScript. It is improving. But it is not good.
Dynamic Content Loaded!
This is all the critical text, pricing data, and internal links that Google entirely missed on its first pass because they were locked behind client-side JavaScript execution.
Hover or tap to process the Rendering Queue
When Google crawls a traditional HTML page, it sees all the content immediately. When Google crawls a JavaScript-rendered page, it first sees a near-empty HTML shell. Then it has to queue the page for a second wave of processing, wait for the JavaScript to execute, wait for the content to render, and then read it.
That second wave? Google calls it the “rendering queue.” Pages can sit in that queue for days. Sometimes weeks. And plenty of pages never get properly rendered at all.
I once saw a brand launch a 4,000-page Next.js product site relying purely on client-side rendering. For six weeks, their organic impressions were zero because Google couldn’t see a single product. JavaScript SEO mistakes are very, very severe.
What to actually do:
- Server-Side Render (SSR) or Static Site Generate (SSG): all critical content, titles, meta tags, H1s, body content, and internal links should be present in the raw HTML.
- Test what Google actually sees: using the URL Inspection tool in Google Search Console → “Test Live URL” → “View Tested Page” → check the rendered HTML.
- Avoid lazy-loading critical content: lazy loading images is fine; lazy loading your main body content is asking for crawling problems.
- Crawl with a JavaScript-disabled crawler: to see what Google’s first pass looks like. What’s missing is what Google potentially never sees.
If your site is built on Next.js specifically, you’re actually in luck: Next.js natively supports SSR and SSG. The technology is there. The question is whether your development team configured it correctly.
Read the Full Guide: JavaScript SEO →10. Crawl Budget Optimization
Google doesn’t have unlimited time for your website. Don’t waste a second of it.
Every website gets a crawl budget, Google’s way of deciding how many of your pages to crawl, and how frequently. For small websites with a few dozen pages, this isn’t something you think about. For large sites, e-commerce stores, news sites, aggregators, anything with thousands or tens of thousands of URLs, crawl budget management is mission-critical.
Hover or tap to send Googlebot through unoptimized faceted navigation
Your crawl budget gets eaten by:
- Faceted navigation: filter pages on e-commerce sites creating thousands of URL combinations like
/shoes?color=red&size=9&brand=nike. Each combination is a unique crawlable URL. - Session IDs in URLs: tracking parameters appending endless duplicate versions of the same page.
- Infinite scroll or pagination without proper HTML links: Google can’t follow a “load more” button.
- Soft 404 pages returning 200 status codes: Google obediently crawls pages that tell users they don’t exist.
- Duplicate content at multiple URLs: every duplicate version is a crawl budget hit with zero return.
The goal of crawl budget optimization isn’t to increase how often Google crawls you. The goal is to make sure the budget Google does allocate goes exclusively to your valuable pages, not to filter combinations, session IDs, and ghost URLs.
Read the Full Guide: Crawl Budget Optimization →11. URL Structure
Your URLs are public. Make them readable by humans and machines alike.
A URL is not just an address. It’s a signal. To users, it communicates credibility and relevance. To Google, it provides context about page hierarchy and content type. A good URL structure is clean, logical, and self-explanatory. A bad one looks like it was generated by a randomness machine at 3 AM.
Hover or tap to decipher the ugly URL
The deciphered URL tells me, and Google, exactly what I’m getting before I even click. It’s in the right category. It describes the content. No guesswork required.
URL Best Practices: Use hyphens, not underscores. Keep them lowercase always. Make them short and descriptive. Include your primary keyword naturally. Avoid dynamic parameters where possible.
Think of your URL structure as city planning. Get it right from the beginning and everything flows logically. Redesign after the city is built and you’ll have confused residents and broken roads for years.
Read the Full Guide: URL Structure →12. Duplicate Content
Two pages saying the same thing. Google picking neither. You wondering why.
Duplicate content is exactly what it sounds like: the same or substantially similar content appearing at more than one URL on your website. Or sometimes, on completely different websites.
Google’s official line is that duplicate content doesn’t result in a manual penalty in most cases. What it does result in is Google choosing which version to rank, and it may not choose the one you want. Link equity gets split between duplicates. Ranking signals dilute. Pages that should be strong become average. Pages that should be average become invisible.
Hover or tap to consolidate duplicates
The main sources of duplicate content (most of them unintentional):
- HTTP vs HTTPS: http:// and https:// versions both being accessible and indexed
- WWW vs non-WWW: [www.yoursite.com](https://www.yoursite.com) and yoursite.com treated as separate sites
- Trailing slash vs no trailing slash: /about and /about/ both live, both indexed
- Product variants: same product, different colour or size, served at different URLs with near-identical descriptions
- Self-Plagiarism: boilerplate service text copied across dozens of location pages
Most of these are fixed through canonicalization (covered in section 7), proper 301 redirect configurations, or simply deciding which version is the canonical version and enforcing it universally across the site.
Read the Full Guide: Duplicate Content →13. International SEO & Hreflang
When you have multiple languages or regions, and want Google to show the right version to the right person.
International SEO is where technical complexity reaches its peak. If your website serves multiple countries, multiple languages, or both, you need to tell Google explicitly which version of a page is meant for which audience. Without this, Google guesses. Google’s guesses are often wrong.
Hover or tap to route users via Hreflang
The tool for this is the hreflang tag, a small annotation you add to each page that says: “This is the English version for US users. This is the Spanish version for Spanish-speaking users. This is the Portuguese version for Brazilian users.”
When implemented correctly, hreflang ensures a user in Brazil sees your Portuguese version, a user in Spain sees your Spanish version, and a user in the US sees your English version. Correct localisation, delivered automatically.
When implemented incorrectly, which is most of the time, you get UK users seeing the US version with dollar prices, Spanish users getting the English page, and Google completely ignoring the tags because they contain errors.
The absolute rules of hreflang:
- Every variant must reference every other variant: if you have an English and a Spanish version, both pages must reference each other. One-directional hreflang is invalid.
- Every page must include a self-referencing hreflang: a page must reference itself as well as its alternate versions.
- The x-default tag designates your fallback version: shown to users whose language or region doesn’t match any specific variant.
- No conflicts with canonical tags: a page canonical-ing to another URL while also having a hreflang pointing to itself is a contradiction Google will resolve unpredictably.
🛠️ Subu’s Toolkit: International SEO Hub
Hreflang is not beginner territory. Don’t write it manually. Generate perfect tags, or test your existing setup to find hidden errors.
14. The Full Technical SEO Audit Checklist
Everything above, condensed into one place. Bookmark it. Use it.
You’ve read the manual. Now here’s the field checklist, the one you run before a site launch, after a migration, following a traffic drop, or just as a quarterly health check.
🛠️ Subu’s Final Tool: The Complete Audit
Don’t want to check all of this manually? Run your domain through the full Technical & On-Page SEO Audit tool to generate a complete report instantly.
Run this checklist on every site before and after launch. Run it after every major update. Run it after your developer says “I just made a small change to the template.” Especially then.
Read the Full Guide: The Technical SEO Audit Checklist →The TL;DR
You’ve made it to the end of The Tech SEO Manual. Here’s the condensed version for the skimmers and the sleep-deprived:
- Technical SEO is the foundation. Content and links built on a broken technical foundation don’t rank. Fix the base first.
- Robots.txt controls what Google can crawl, check it, protect it, never accidentally block your entire site with it.
- XML sitemaps guide Google, keep them clean, current, and submitted.
- Core Web Vitals are ranking factors, slow and janky sites lose in both dimensions.
- Mobile-first indexing means Google evaluates your mobile site, if it’s an afterthought, your rankings will reflect that.
- HTTPS is the minimum standard in 2026, not optional, not negotiable.
- Schema markup makes your content machine-readable and unlocks rich results.
- Canonical tags tell Google which version of a page to rank when duplicates exist.
- Redirects need to be the right type (301 vs 302) with no chains and no loops.
- JavaScript SEO is the most dangerous trap, if your content only exists after JS executes, Google may never see it.
- Crawl budget on large sites is finite, make sure it’s spent on pages that actually matter.
- URL structure should be clean, keyword-present, and locked in before launch.
- Duplicate content dilutes signals, consolidate, canonicalize, or noindex.
- Hreflang for international sites is non-negotiable and must be implemented bidirectionally.
Technical SEO isn’t glamorous. Nobody’s going to write a song about your robots.txt configuration or get emotional about your canonical tag implementation. But every single page you’ve read about in this guide? Every content strategy, every backlink you build, every keyword you target, it all depends on this working correctly underneath.
Get the tech right. Then build everything else on top of it.
Got a technical SEO nightmare you’re staring at right now? Drop it in the comments. Subu has genuinely seen worse, and Subu will help.