Google indexing

559 articles

Troubleshoot crawling, indexing, robots directives and canonicals when pages are missing from search.

ProcrustesPR
How should you handle redirects and minimize search traffic loss when restoring an old website?

If the old pages are definitely coming back online but search results still show the new URLs introduced in the redesign, first remove the rules that…

How can you address a mismatch between website content quality and its appearance in search?

When website content quality and its appearance in search do not align, first distinguish between three possibilities: the page does not answer users’…

What are cache busting and the standard sitemap update mechanism used for?

Cache busting and the standard sitemap update mechanism address different problems. Cache busting prevents browsers from using outdated cached versions of…

What Is a Video Sitemap For? Required Tags and When to Use Optional Ones

According to the supplied Google documentation, a video sitemap helps Google find and understand videos on a website, especially newly added content or…

Which SEO elements should you check when building a new website?

When building a new website, check the testing environment, URLs and page markup, search engine crawling settings, and plans for tracking after launch. If…

ProcrustesPRProcrustesPRProcrustesPRUnder what circumstances might backlinks affect rankings?
What can cause “Crawled – currently not indexed,” and how should you address it?

“Crawled – currently not indexed” means Google has fetched a page but has not yet included it in its index. It does not mean the page has been penalized. If…

Good content and plenty of backlinks, but still no rankings or organic traffic? Check these four things first

Believing your content is high quality and having plenty of backlinks do not mean your site should rank. These public discussions suggest four things to…

Does Google consider collapsible content less valuable?

The fact that content requires a click to expand does not, by itself, mean Google considers it less valuable. The public discussions referenced here answer…

Do small websites need a sitemap? First, check whether their pages are easy to find

Small websites do not necessarily need an XML sitemap. If a site mainly contains company information, a contact form, and a few news articles, and search…

How does blocking crawling affect page removal?

Blocking crawling does not remove content that has already been collected. Its effects depend on which crawler is blocked and where you want the content…

How Do Absolute URLs and Relative Links Affect SEO?

Choosing between absolute URLs and relative links comes down to whether links point to the right pages, how easy they are to maintain, and where they are…

Why might secure-server pages not be indexed by search engines? Check login requirements, redirects, and actual URLs first

If secure-server pages are not indexed by search engines, do not assume the secure connection is the cause. First check whether accessing the content…

ProcrustesPRProcrustesPRProcrustesPRWhat risks remain with large numbers of links, even from topically relevant websites?
How do you verify crawler identity and enforce access policies?

Verifying a crawler requires more than checking its User-Agent name. Follow the provider’s verification method to check against official IP lists or perform…

How can you check whether content is unique or duplicates other content?

To check whether content is unique, first identify the article or pages you want to examine and decide whether to compare them with content on your own site…

How can you validate and troubleshoot AMP content?

When validating and troubleshooting AMP content, make three separate checks: whether the content is valid, whether the pages can be crawled, and whether…

How can you prevent or handle unwanted duplicate HTTPS pages?

To handle unwanted duplicate HTTPS pages, first identify which URLs serve duplicate content and which version you want to keep. Then fix the links pointing…

How can you limit a webpage’s appearance in featured snippets?

To limit a webpage’s appearance in Google featured snippets, first decide whether you can accept losing regular search snippets too. If you need to reliably…

How can you restrict or verify Googlebot?

To verify Googlebot, you cannot rely solely on the User-Agent in the request headers. Google recommends performing a reverse DNS lookup on the source IP…

How can you limit content appearing in Google Search?

To limit content appearing in Google Search, first decide whether you want to restrict indexing, snippet display, or access to the content . Use noindex to…

ProcrustesPRProcrustesPRProcrustesPRUsing Web 2.0 Sites for SEO: Content Exposure and the Effects and Risks of Link Manipulation
How can you reduce the risks of changing a website’s domain?

Reducing the risks of a domain change requires more than sitewide 301 redirects and submitting a change of address. You also need to check where each old…

How can you prevent URLs with specific parameters from being indexed by search engines?

To prevent URLs containing the widgetCacheId parameter from being indexed, have them return noindex and allow search engines to crawl them and read the…

How do you deploy and maintain video structured data?

To deploy video structured data, first add VideoObject to pages where users can watch the video, include all required properties, and test the markup. Then…

How can you prevent website settings from accidentally blocking content from being indexed?

To prevent website settings from accidentally blocking indexing, first confirm that important pages can be crawled. Then check whether robots.txt and…

How to keep specific pages out of search results: first check the page’s status and who controls it

To keep a specific page out of search results, first determine whether you control it and whether it still needs to be publicly accessible. For pages you…

How do you choose and implement robots tags?

When choosing robots settings, first distinguish between crawling, where crawlers read content, and indexing, where content is included in a search engine's…