While Google deprecated the old AJAX crawling scheme, they have always recommended progressive enhancement and recently updated their recommendations at Google I/O to include dynamic rendering where server-side rendered content is shown to bots, rather than attempting a complete client-side approach.
But What About Bing?
Everyone forgets about Bing, which is fair enough. Bing also powers the Yahoo search results and their market share in early 2018 was around 14% in the UK, but a more significant 24% in the US (according to recent Statista data).
Although in November 2014 when they introduced new mobile search bots , Lee Xiong from the ‘Bing Crawl Team’ discussed their advances in rendering –
“In all of these examples, the user agent strings containing “BingPreview” refer to crawlers that are capable of “rendering” the page, just like a user’s browser would. It is therefore paramount that you allow our crawlers to not only find the core content of the URLs themselves, but that you also allow them access to the necessary resources needed to load each page, that is, including any CSS, script, and image files.”
If you run some searches (alongside a site: query) in Bing for Bartosz’s JS test site, you can see that Bing is still not indexing any of the content today.
As usual with anything Bing related, there was barely any discussion or coverage.
Internally we brushed this off at the time as the only evidence we had ever seen, was them making a complete mess of it when a client tested a client-side approach last year for an SPA, before moving to server-side rendering.
Show Me The Money
A website we have been aware of and have been monitoring that relies purely on client-side rendering with React is Prose (https://prosehair.com/). If you check a site: query in Bing, you’ll see 11 URLs have been indexed (which doesn’t itself mean anything, as URLs can be indexed by sitemaps etc).
At first glance, you’ll see they all have the same page title (which is in the initial HTML response on every page) and typically you may assume this means Bing isn’t indexing the content in the rendered HTML, where there are unique.
However, if you look again, you’ll notice all the snippets are unique, and the initial HTML response (view source) doesn’t have meta descriptions or content.
The interesting thing here is that Bing is using the static HTML title still in the SERPs.
The title is from the original HTML (which you can see from viewing the source) –
The page titles in the initial HTML are overriding the rendered, more unique and descriptive page titles in the SERPs (which you can see using ‘inspect element’ in Chrome).
Bing is using the rendered HTML meta descriptions, where there aren’t any in the initial HTML response.
To verify the site wasn’t dynamically rendering to search bots, we spoofed the user-agent string, and we were able to check the fetched HTML in Google Search Console, not Bing Webmaster Tools (but presumably it would be spoofed there, too).
While Bing wouldn’t show a cache of the page at all, Google’s cache is as you would expect for a client-side JS site (showing the initial HTML, rather than the rendered HTML).
A site query in Bing shows 90 results that have been indexed.
And content which is only available in the rendered HTML is being returned under search.
Although, again Bing is using the title element returned in the initial response, rather than the rendered HTML.
One More For Luck
Another website which loads all content using an external JS file is NS Tecnologia (http://nstecnologia.com.br/). Their initial HTML is literally 8 lines (a clean approach!) –
A site query pulls back their blog at the top results (which doesn’t rely on JS), which again, might make you assume the homepage hasn’t been indexed.
However, a URL query shows it is actually indexed – running searches for content from the page in the rendered HTML also returned the page.
The results also suggest that Bing may have a two-phase indexing process, much like Google. However, if Google renders a page for indexing, we generally find that Google then uses that rendered HTML version for indexing and scoring. Bing’s approach appears to be far more mixed, between content in the initial response, and rendered HTML, which makes it less convincing and reliable.
We know a lot now about Google’s WRS (Chrome 41), which was always present in log files. If Bing is accurately matching their user-agent strings to their WRS, then they might well be using IE11 (released in October 2013).
It’s worth noting, we have seen lots of sites being discussed and shared as client-side when in reality, they actually use the old AJAX crawling scheme or are dynamically serving pre-rendered content to search bots. As a caveat, we analysed the sites above as best we can, to ensure that they were being delivered entirely client-side, but it’s difficult to be 100% sure without having direct access to them.
Finally, it’s worth reiterating that if you care about SEO and sleeping at night, don’t rely on client-side rendering.