How does Google crawl Infinite Scroll pages?


#1

Hi! I’ve always wondered how Google crawls pages that have infinite scrolling? Posts on the Google webmaster blog such as this one (Understanding Dynamic Webpages) suggest that Google understands JavaScript and parses it like a user agent. So if that is the case, wouldn’t it get stuck in a big loop while crawling, and keep on going through all posts on a blog everytime it landed on the homepage? It must take a lot of time and network traffic as well. So what are the optimal SEO settings for such a set-up (infinite scroll)?


#2

Nice question I had a similar query in mind related to paginated pages. Waiting for mohamamd to clarify this.


#3

The good news is that from May 25, 2014 onward, Google has started parsing and indexing Javascript. Which is a big relief for dynamic sites like “Blogger Help Forum” that uses Ajax and JSON to load dynamic content and also uses infinite scroll Page (aka ajax loaded content) to keep users engaged on a single page.

There is standard method while developing JavaScript application. Previously we would need to create two sets of data one for sever side and one for search robots enclosed inside <noscript> tags but now Google has matured enough to index and crawl javascript content by itself.

To make sure that search engines can crawl individual items linked from an infinite scroll page, we make sure to produce paginated series. The image below from Google webmaster blog gives an easy explanation:

SEO for infinite scroll pages

Infinite scroll page is made search friendly when converted to a paginated series, each component page has a similar *<title>* with rel=next/prev values declared in the <head> tag that tells search robots that all these paginated pages are chunks that are closely interlinked and not duplicated items. This makes your application search friendly and puts it on a safe side.

As long as you are adopting the above discussed method, your content will stay indexed and easily crawled.

For more in-depth details, you can read the following comprehensive post by

Let me know if you needed more help understanding this easy implementation.