Google crawler not seeing my nav

by Will Haley   Last Updated December 19, 2017 17:04 PM

I have created a trivial/demo SPA with Ember.js. It has no meaningful content. Just filler. I created this specifically to try and understand how Google may crawl a production Ember SPA of mine.

Google says...

In 2009, we made a proposal to make AJAX pages crawlable. Back then, our systems were not able to render and understand pages that use JavaScript to present content to users.

...

Times have changed. Today, as long as you're not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers.

I have not created a sitemap.xml or tried to enhance the HTML semantics in any specific way for Google. I just threw this together in about an hour. I was hoping to understand how Google crawls the links on a site like this.

The app renders without issue in production, and is backed by fastboot running server-side, which can route non-root URLs without a hashbang. So a server-side request for something like /about or /api is handled properly and renders the SPA with the correct route/pushState.

enter image description here

It's all fine and dandy, except when I ask Google to crawl the root (/) "and linked pages". For reasons unclear to me, Google shows that it completely misses the nav bar when it crawls...but, the style is there, the dynamically loaded content is there.

So, where's the navigation?

enter image description here

I didn't think there was anything odd about the navigation. It's not hidden by my styles. It's certainly not ideal as far as I'm concerned. I'd rather see this as an li, but again, I threw this together in an hour. I didn't imagine that having the navigation set up like this would make it completely hidden from the crawler for whatever reason.

<div class="nav-items">
  <div><a href="/api" id="ember257" class="ember-view">API</a></div>
  <div><a href="/about" id="ember266" class="ember-view">About</a></div>
</div>

A result of this seems to be that Google search results are not showing my /about page, but do show my /api page. The /api page is linked to from both the body and the nav, but the /about page is only linked from the nav. Since the nav isn't rendered, it seems like search results from Google do not know about my /about page.

I'm sure that a sitemap.xml and other changes would fix this, but my curiosity lies in why the nav bar is hidden. The "Fetching" tab in "Fetch as Google" shows that the nav bar HTML is present, but it seems like it is not rendered from Google's perspective. Why is the Google Crawl rendering process seemingly ignoring it?



Related Questions



Single Page Application, Web Crawlers and SEO

Updated August 16, 2017 08:04 AM


What would be the best way to index react app?

Updated September 25, 2017 04:04 AM

How do I increase traffic to my single page app?

Updated July 03, 2016 08:01 AM