Bing On SEO for subscription content and paywall

Fabrice Caneau post an article on SEO best practices for sites that use subscription and paid content. While Google offers something called flexible sampling, previously a first-click free program, Microsoft Bing does not.

Bing basically wants you to detect that BingBot is trying to access this subscription and paywall based content and then give BingBot full access to this content. In a sense, show Bing all the content and only show users a snippet of the content. Bing says it’s okay, it’s not a form of cover-up and it’s allowed.

Fabrice wrote “the first step is to allow search engines, like Bing, to see the full content that normally resides behind a paywall or subscription. By accessing the full content, search engines will be able to index more text, which will help match more customer requests.” “You can identify Bingbot by referencing the IP address to Bingbot’s public list of IP addresses,” he added. The list of BingBot IP addresses was released by Microsoft recently.

Next, Fabrice explained that the next step is to “prevent search engines from exposing publisher subscription-based or paywall-based content in search engine cache pages.” It helps “publishers can control whether search engines show a cached page of a document by using a special bot meta tag in the header section of the web page or, alternatively, by using a client HTTP response header returned to search engine crawler for each URL.”

His blog post explains how to make this Robots Meta Tag and/or X-Robots-Tag.

He then explains why they don’t take Google’s approach:

Discussion forum on Twitter & WebmasterWorld.