Session

200 Standard success — full content returned.

304 Not Modified — bot already has this page cached, server skipped sending content. Saves bandwidth but counts as a crawl hit.

— solid edge — URL was reached via a referer link from another page in this session.

- - dashed edge — No referer. Bot fetched this URL from its own crawl queue, not via a link.

dashed border node — Entry point. No other page in this session linked to it.

grey node — External referer. Not part of this session — it's the page that triggered the crawl.

301 Permanent — passes link equity to destination. Use for page moves.

302 Temporary — does not reliably pass link equity. Avoid for permanent moves.

307 Temporary (method-safe) — same SEO implications as 302.

308 Permanent (method-safe) — like 301 but preserves POST method.

400 Bad Request — malformed URL or request.

401 Unauthorised — login required, bot can't access.

403 Forbidden — server blocked the bot (firewall, IP, robots).

404 Not Found — page doesn't exist. High volume = broken internal links.

410 Gone — page permanently removed. Tells Google to deindex faster than 404.

429 Too Many Requests — bot rate-limited. May reduce crawl frequency.

500 Internal Server Error — unhandled exception on the server.

502 Bad Gateway — upstream server (e.g. PHP-FPM) returned an invalid response.

503 Service Unavailable — server overloaded or in maintenance mode.

504 Gateway Timeout — upstream server took too long to respond.