Website security
89 articles
Address HTTPS, permissions, updates and maintenance to reduce common website risks.
When checking security headers in a technical SEO audit, the key is to distinguish embedding restrictions that may affect how content appears in search from…
What are common web crawlers used for, and how should websites set access rules?Websites should not allow or block all crawlers indiscriminately. Instead, first determine whether a crawler is used for search indexing, model training, or…
How do .htaccess and robots.txt differ when blocking bots, and can you use just one?You can use just one, depending on whether you want bots to respect crawling restrictions or need the server to deny access. robots.txt may be enough for…
Why might secure-server pages not be indexed by search engines? Check login requirements, redirects, and actual URLs firstIf secure-server pages are not indexed by search engines, do not assume the secure connection is the cause. First check whether accessing the content…
How do you verify crawler identity and enforce access policies?Verifying a crawler requires more than checking its User-Agent name. Follow the provider’s verification method to check against official IP lists or perform…
How can you restrict or verify Googlebot?To verify Googlebot, you cannot rely solely on the User-Agent in the request headers. Google recommends performing a reverse DNS lookup on the source IP…
How do you choose, set, and check webpage link attributes?When choosing webpage link attributes, start with three questions: does the link involve payment or user-submitted content, how should it open when clicked,…
What are the benefits and risks of switching a website to HTTPS?The main benefit of switching a website to HTTPS is that it encrypts data transmitted between the browser and server. This is particularly important for…
What is website health, and how should it be measured and continuously monitored?Website health describes how well a site functions, how easy it is to use, and whether it has the foundations needed for search visibility. Measuring it…
How can you keep addressing technical SEO issues?Ongoing technical SEO means making regular audits, fixes prioritized by severity, and continuous performance monitoring part of website maintenance—not…
How do you create, check, and safely use robots.txt?To create a robots.txt file, first decide which crawlers may access which paths. Then save the rules as a plain-text file in the website’s root directory…
How to block or restrict AI crawlers: identify their purpose, then choose the level of controlTo restrict AI crawlers that follow the rules, you can use robots.txt. To reject requests from crawlers that ignore those rules, or requests to fetch…
How to address manual actions and security issues on your website: check the notices first, then fix each issue separatelyTo address manual actions and security issues on your website, first check the notices for both in Search Console, then fix the issues based on the reasons…
How can you check whether a website uses SSL and whether Google has indexed HTTPS or HTTP?Two things need to be checked separately: whether the website currently serves pages over HTTPS, and which URL version Google has indexed. For the first,…
How can you test whether a crawler follows the rules? Check the logs, then compare allowed and disallowed pagesTesting whether a crawler follows explicitly stated rules requires more than checking whether it fetched disallowed pages. This article focuses on…
How to keep content out of search results: the differences between noindex, crawl restrictions, and access controlsTo keep publicly accessible content out of search results, you can set noindex through a page's meta tag or an HTTP response header. Search engines must be…
When should you block crawlers, and how?Consider blocking crawlers only when a website is still being tested, is not intended to be public, or has specific content you do not want crawled. Pages…
What Is an HTML Redirect, and When Should You Use One?An HTML redirect uses a meta refresh tag on a page to tell the browser to send visitors automatically from the original URL to another URL. It is suitable…
What harm can content farms cause?Content farms can do more than waste readers’ time. They may spread misinformation that affects readers’ health decisions and judgments about public issues.…
Which SEO practices are unethical, and what risks can they pose?Unethical SEO centers on deceiving users, manipulating search systems, or maliciously undermining competitors to improve rankings. Common practices include…
How Does Trust Affect SEO and User Behavior?Trust may influence whether users click, continue browsing, and become customers, and it is also related to search performance. However, users trusting a…
What Is Negative SEO, and How Much Impact Can It Have?Negative SEO involves malicious efforts to harm someone else’s search visibility—for example, making a website appear to violate search engine policies or…
How do SEO companies address technical and operational website issues?The issues an SEO company can help with should first be clearly defined: inconsistent URL and link settings, insufficient guidance when pages cannot be…
What factors should you consider when choosing a data center?When choosing a data center, first identify where your website’s main visitors are located, your service requirements, and your future growth needs. Then…
How should you maintain a website after choosing a CMS?After choosing a CMS, schedule ongoing website audits, updates, and backups, and check for technical errors, SEO issues, and overall site health. If the…
