Hacking the Console Log To Spy on Googlebot

Googlebot uses Chrome 41 to render webpages. I wanted to see if I could write code that would catch if Googlebot outputted interesting items to the console log during the rendering of pages in their Google Search Console > Fetch As Google tool.

Digging a bit I was able to find this Github repo that listened for console updates and allowed me to hook in to output the log messages to a previously appended <pre> element at the top of the page. Unfortunately, I could not find a way to hook into all of Chrome's logged messages, only the ones directly called via console.(log|warn|error)(). I did learn about a monitorEvents function in Chrome Developer Tools that allows you to output all types of events to the console. That will be helpful for future debugging sometime. Nevertheless, some log information did get diplayed and I had a decent script to see what else I could output from the rendering process.

I added an output dump of the window object's valid properties, and also iterated over the window object's on{blah} events and added event listeners to output that information to the log. That at least gave me a sense that I was getting pretty much all I could get, and brought the realization that the script could be very useful to debug potential rendering issues for Googlebot at some point.

You can see a live demo of the output here.

Note: Since it logs all events, it will be very difficult to scroll to the bottom of that page. You've been warned.

From there, I tested in Google Search Console.

Here is an output from Googlebot Desktop

desktop-googlebot-1

Here is an output from Googlebot Mobile

mobile-googlebot

It was interesting to see that perhaps Google just drops in some default information for the window.timing properties. For example, responseStart and reponseEnd always seem to be different by 10 "milliseconds". I know from Tom Anthony that Google uses a sped up clock in javascript, so I am not really sure how to refer to the timing difference, so I will just call them 10 snits.

Mobile Googlebot's screensize is 431 X 731 and Google resizes to 12,140 pixels high, while the desktop version is 768 X 1024 but Google only resizes to 9,307 pixels. Also, Google has the cookiesEnabled flag set to true, so even if it doesn't store them between sessions, they may be doing something with them.

You can grab the javascript code to put into the footer of your site here. I make no warranties about this code. Use at your own risk.

Update: Below is the updated view of the events that Googlebot fires (eg. message, load, pageshow, resize).

Mobile

mobile-gbot-2

Desktop

desktop-gbot-2