Website decisions

Keep staging and preview websites out of Google search

Choose access controls and indexing rules for preview websites, then check the live site has the right settings at launch.

Hands typing website code on a laptop screen.

The short answer

Choose access controls and indexing rules for preview websites, then check the live site has the right settings at launch.

  • Use access controls for content that should be private.
  • Robots.txt alone does not reliably keep a URL out of search.
  • Check both preview protection and production indexability at launch.

Decide whether the preview must be private

A staging site may contain unfinished copy, test forms, customer material or a future offer. If people should not be able to access it, use an appropriate authentication or access-control mechanism. A search setting does not make a publicly accessible page private. Keep an inventory of preview domains, temporary hosting addresses and old test sites. Record the owner, access policy and retirement date for each. An abandoned preview can remain discoverable long after the new website launches.

Understand the three different controls

Google's documentation distinguishes crawling rules from indexing rules and access protection. Choosing the wrong control can produce an unexpected result.

Google specifically warns that blocking crawling can prevent it from seeing a noindex instruction. A blocked URL may still appear based on links to it. Ask the developer to select controls that match the actual purpose and verify the response rather than assuming a file name proves the outcome.

  • Authentication or password protection: restricts access to content that should be private.
  • Noindex: tells a supported search crawler not to index an accessible resource when it can read the instruction.
  • Robots.txt: controls which URLs a crawler may fetch; it is not a reliable mechanism for keeping every URL out of search results.

Test the preview from outside the signed-in session

A developer's browser may be authenticated automatically. Check what an ordinary unauthenticated request receives, including the actual temporary hostname. Look for exposed articles, assets, form endpoints and alternate preview URLs within the agreed scope. For an intentionally public preview using noindex, inspect the actual response and page instructions. For a private preview, confirm access is genuinely restricted. Do not describe a client portal as protected merely because a search engine is asked not to list it.

Make the launch switch a two-sided checklist

It is possible to protect staging correctly and still launch production with an accidental noindex instruction or authentication requirement. Review the real production domain as well as the preview configuration. Confirm the intended live URLs return the right content, have the correct canonical addresses and indexing instructions, and appear in the appropriate sitemap. Check forms use the intended live destination and test submissions are labelled. Keep protection on previews that remain in use after launch.

Keep evidence of the actual state

Record the preview host, test time, unauthenticated result and relevant response instructions. Separately record production checks and any later Google observation. Technical eligibility, a Google crawl and an indexed URL are different pieces of evidence. Repeat the launch checks when changing hosting, preview systems or site-wide SEO settings. Avoid deleting a useful production page simply because an old preview was exposed; fix the URL and environment that actually caused the problem.

Common questions

Does robots.txt make a staging site private?

No. Use appropriate access controls for private material. Robots.txt is a crawler instruction, not an authentication boundary.

Should production be checked too?

Yes. Confirm the real domain has the intended public access, content, canonical and indexing settings after the launch switch.

Sources & further reading

Official references supporting the platform guidance in this article.

Make it work for your business.

Turn a good first impression into a clear next step.

A custom website built around your customers. Clear messaging, considered design and a simple path to enquire, with optional ongoing care.

About Demandify

Demandify helps Australian businesses connect lead generation, Meta ads, websites and SEO. Founded by Brandon Cutajar, with a background in finance, automotive and marketing. Our guides explain how we approach the work; a proposal sets the scope for your business.

Meet the founder