Technical SEO diagnostics: trace a page from request to index

Start with the crawl path

Begin with the exact URL and ask whether a crawler can reach it. Read the site’s robots.txt at the root of the host. A Disallow rule controls whether a crawler may request a path; it is mainly a traffic-management tool. It does not reliably remove an HTML page or PDF from Google Search, because another page may expose the URL even when its content cannot be fetched. A page that must stay out of search needs a noindex directive or access control, and the crawler must be allowed to fetch the directive.

Treat the edge as part of this first check. A WAF, bot rule, rate limiter or login wall can return a challenge, a 403 or a different response to Googlebot, Bingbot or a browser. A request with a familiar User-Agent is only a smoke test: it does not prove that the real crawler’s network is allowed. Review WAF security events, verified-bot handling and origin logs with the person who owns the edge configuration. Test an important URL, its stylesheet and its image assets, because blocking a required resource can change what a renderer sees. Record the timestamp, host, path, response and edge action before changing a rule.

Check the HTTP response before reading the page

Next, inspect the response status and redirect chain. A 200 means the server returned a representation, not that the page is useful or indexable. A 301 or 308 can be correct when an old URL has an equivalent permanent destination; test the old URL, every hop and the final URL. A 404 or 410 is appropriate for content that is genuinely gone. A 500, 502, 503 or repeated timeout is a serving problem, and prolonged 5xx or network failures can cause crawling to slow and existing URLs to disappear from the index.

Use a HEAD request for a quick header check, then a GET request for the actual body. In a terminal, curl -I https://example.com/page shows status and headers, while curl -L -A Mozilla/5.0 https://example.com/page follows redirects and lets you save the returned HTML for inspection. Do not treat a successful command as proof of every user-agent path; compare a normal browser, a search crawler test where appropriate and the provider’s logs. Look for content type, cache behavior, compression, X-Robots-Tag and the final URL. A branded error page returning 200 is a soft 404: people see an error, but the status tells a crawler that a real page exists.

Separate noindex from canonical

Noindex and canonical answer different questions. A robots meta tag with noindex, or an X-Robots-Tag header with noindex, asks Google not to include that URL in search. It does not stop a visitor from opening the URL, and it only works when the crawler can fetch the response. If robots.txt blocks the path, Google cannot reliably see the noindex rule. Search for every robots meta element in the original HTML and every X-Robots-Tag value in the response headers; an application or plugin may add one later.

A canonical link identifies the URL you prefer among duplicate or near-duplicate versions. It is a signal, not a command that overrides every other fact. Put a single, absolute, self-referencing canonical on the preferred page, keep internal links and the sitemap consistent with it, and do not point an indexable page at an unrelated homepage. Do not use noindex to choose a canonical within a group when the page should remain searchable. Check the canonical in the source HTML, the rendered HTML and Search Console’s Google-selected canonical. A JavaScript change that disagrees with the original HTML creates an avoidable diagnosis.

Use the sitemap as an inventory

A sitemap should contain the absolute URLs you want discovered and considered for search, normally the canonical versions. It is a hint, not a submission receipt or an indexing guarantee. Compare its rows with the route inventory, the internal links and the pages that the business actually wants to keep. Remove staging hosts, parameter variants, redirected URLs, noindex pages and thin utility states unless there is a deliberate reason to list them.

Open the sitemap over HTTPS and confirm a successful response, valid XML and a consistent host. Check that each listed URL returns the intended status and canonical, and that translated pages point to their own language version where that is the project’s design. If there are more than 50,000 URLs or the uncompressed file exceeds 50 MB, use an index and split the child files. Submit the sitemap through Search Console after the server and page signals agree, then monitor discovered, crawled and indexed counts. A submitted sitemap can reveal a mismatch; it cannot repair one.

Compare initial HTML with rendered HTML

Inspect two documents. The initial HTML is the body returned by the first page request, before scripts run. The rendered HTML is the DOM after the browser or a crawler renderer has loaded scripts and resources. A server-rendered or pre-rendered page can expose its title, headings, copy, links, canonical and structured data immediately. An app-shell page may deliver almost no useful text until JavaScript calls an API. Google can render many JavaScript pages, but rendering is a later step, takes resources and is not a capability every search engine or social crawler has.

Open View Source or the saved GET response and search for the main heading, meaningful copy, internal links, canonical and robots directives. Then use browser inspection or Search Console’s live URL test to view the rendered HTML and loaded resources. If a paragraph, link or metadata field exists only after a failed API call, a consent gate or a hydration error, the source verification has found a real indexing risk. Keep the rendered version consistent with the source; use JavaScript for enhancement, not as the only delivery mechanism for the page’s meaning. Fix missing server data, blocked assets or runtime errors, then repeat both inspections.

Follow a small decision tree

Write the diagnosis as a decision tree that another person can run:

1. Can the intended crawler reach the URL and its required assets? If no, inspect DNS, TLS, authentication, robots.txt, WAF events and network errors. Restore access or document the intentional block.

2. Does the final response have the intended status? If it is a redirect, test the destination. If it is 4xx, confirm whether the content should exist. If it is 5xx or times out, fix serving capacity and retry. If it is 200 with an error message, return a real 404 or repair the page.

3. Does the response contain noindex? If yes, decide whether that exclusion is intentional. Remove it only when the page is genuinely eligible; do not hide a staging or private page by accident.

4. Is there one canonical, and does it agree with redirects, links and the sitemap? If no, choose the correct representative URL and make every signal agree.

5. Is the useful content in the initial HTML or reliably present after rendering? If no, repair server rendering, API access or JavaScript errors, then test the rendered result.

6. After the fix, does Search Console’s live test show a crawlable, indexable page? That is evidence of access and detected signals, not a promise that Google will index or rank it. Keep the old response, the fix, the retest and the owner in the issue record.

Turn evidence into a fix order

Prioritize failures that prevent the page from being fetched or understood: DNS, TLS, authentication, WAF blocks, server errors and accidental noindex. Then correct conflicting canonical, redirect and sitemap signals. After that, improve initial HTML and rendering reliability, followed by internal links, metadata and less urgent enhancements. Pair every finding with the URL, user-agent or tool used, timestamp, raw status or header, screenshot or saved HTML where useful, and a reproducible retest. This avoids the common mistake of declaring technical SEO healthy because one browser loaded a pretty page. A stable diagnosis tells the next developer what the crawler received, what the visitor saw and which remaining uncertainty still needs measurement.