Why Chrome DevTools Is Inaccurate for Mobile Testing

In Mobile Testing, Technical Brief by Chris Strahorn

Last Updated January 10, 2018

It’s the middle of the night and you’re staring at your PC probing for bugs. Since you’re using Google Chrome, you hit F12 and pull up Chrome DevTools to help with your mobile website debugging. Up pops a sweet suite of developer tools, which includes a mobile testing emulator that lets you imitate the properties of any phone or tablet under the sun. It’s Google so it must be completely accurate, right?

Now here’s the news you didn’t want to hear: Chrome’s mobile emulator often gets it wrong, delivering a number of unwelcome surprises. And these surprising mobile testing results are a no-fun basket of “Easter eggs.” Instead, they could seriously mislead your team about your website’s performance on mobile, just when you’re trying to engage and convert customers.

Lurking Dangers in Chrome DevTools for Mobile

Chrome’s developer tools offer an emulator, not an actual phone or tablet, to simulate your website on any given mobile device. burger king chrome device modeThis emulation is based solely on changing the screen resolution to match the dimensions of the device. Consequently, Chrome’s display of your website on mobile may be completely inaccurate. That’s right, Google may be getting it totally wrong!

Chrome’s departure from actual phones and tablets may result in the dreaded “false positive”: you think your newly designed website is bug-free on the latest iPhone 6s plus and ready to go live because Chrome’s device mode emulator shows you it is. But serious bugs infecting your code or poor optimization for mobile may be delivering a poor UX, painfully slow load speeds, and decreased conversion rates. Yes, you are soaring with confidence over your website’s sure-to-be successful launch, even while Chrome’s inaccuracies are setting you, your team, and your brand up for a fall.

Why Chrome DevTools’ Mobile Emulator Delivers Inaccurate Results

DevTools is an example of an emulator, which is an application that imitates the operating properties of mobile device hardware and operating systems and displays the results on your desktop. Sounds perfect, right?

But since emulators do not replicate the software or the hardware of the actual physical mobile devices, they have serious limitations, as Google acknowledges. Simply put, these limitations result in serious testing errors.

Google even admits their emulator fails to accurately duplicate the behavior of the GPU and CPU in the mobile device hardware. But here are two additional big deficiencies in the Chrome mobile emulator that Google doesn’t mention:

  1. Chrome mobile emulator uses a different rendering engine than either Safari on iOS or the native Android browser. The rendering engine is what takes some HTML and turns it into a web page that you can see and interact with on the screen.
  2. Chrome mobile emulator uses a different JavaScript engine from Safari in particular. Chrome uses V8 while Safari runs JavaScriptCore/Nitro.

Chrome’s mobile emulator deviates significantly from components of actual phones and tablets. In fact, Chrome’s web dev testing tools not only employ different rendering engines and different JavaScript engines, but also different hardware — arguably everything that’s important in how a web page actually renders.

Finally, additional inaccuracies are introduced by Chrome DevTools, since they often use the latest OS for each device, but that differs from consumers’ use in the real world. For example, Android 4.x is still the most popular and most used OS, operating in more than 50% of Android devices, still trumping the later 5.x and 6.x combined.

Chrome Mobile Device Mode vs. Mobilizer
Round 1: Taco Bell on the iPhone 5c

Let’s look at actual examples of mistaken mobile test results from Chrome’s F12 and why their mobile emulator failed to perform with 100% accuracy.

Chrome Device Mode

Mobilizer Mobile Testing Lab

taco bell's website on google chrome device mode

On the left, Chrome DevTools shows a perfectly rendered mobile site. On the right, the Mobilizer test result from a real iPhone 5 located in the Mobilizer mobile device lab reveals that the background imagery on the Taco Bell website fails to appear, leaving only black.

The Chrome emulator is imitating the properties of the wrong OS and thus fails to expose the glitch. Had Taco Bell’s marketing, development and QA teams relied solely on Chrome mobile emulator, they would not have seen the error and would not have known that their customers were unable to view their delicious burrito background.

This error is remarkably common. Older operating systems account for more than 50% of traffic on Android devices and 5% of traffic on Apple devices. So, if you are using Chrome DevTools, you are likely failing to spot this issue with your website’s mobile performance and thereby delivering a poor UX to a significant share of your mobile traffic.

Chrome F12 Mobile Device Mode vs Mobilizer
Round 2: Paypal’s Snags on the iPad Mini

Chrome Device Mode

Mobilizer Mobile Testing Lab

paypal's webiste on chrome devtools vs mobilizer

Chrome DevTools (on the left) makes it seem like PayPal’s site displays great on the iPad mini. However, as Mobilizer shows on an actual iPad mini, PayPal’s text is badly misaligned, with parts chopped off and obscuring the top navigation links. The background image is displayed much more zoomed out than Chrome’s mobile emulator leads you to believe, and the misalignment of the call to action button blocks out the women, so the customer can’t see how she is happily completing her PayPal payment. Chrome’s mobile emulator shows a perfectly laid-out page, misleading PayPal’s developers, designers, QA and marketing teams.

Chrome F12 Mobile Device Mode vs Mobilizer
Round 3: Ducati on Samsung’s Galaxy S6

Chrome Device Mode

Mobilizer Mobile Testing Lab

ducati chrome device mode vs. mobilizer

Similar to the issues Chrome F12 has with iOS devices, on Samsung’s Galaxy S6, an Android device, critical errors can also occur. F12 misses glitches on the Ducati site, as shown on a real device (on the right) in Mobilizer, including the top image that is badly cropped, decapitating key messaging along with the motorcyclist’s head – ouch!

Chrome F12 Mobile Device Mode vs Mobilizer
Round 4: Orbit Gum on iPhone 5c

Chrome Device Mode

Mobilizer Mobile Testing Lab

Orbit Gum chrome device mode vs mobilizer mobile testing lab

Chrome F12 tools can claim snafus exist when none occur on the actual mobile devices. For example, the Orbit Gum’s website appears flawlessly on the iPhone 5c, but Chrome’s mobile emulator thinks there are a plethora of bugs that cause the page to barely render.

Chrome Mobile Device Mode vs Mobilizer
Round 5: Cloudera on the iPad mini

Chrome Device Mode

Mobilizer Mobile Testing Lab

cloudera chrome emulator for mobile

On the left, Chrome devtools depicts Cloudera’s website as perfectly rendering on the iPad. But in fact, on an actual iPad mini the background image does not appear — as tests in Mobilizer’s device lab demonstrate. Those in charge of Cloudera’s website may not know their mobile customers aren’t receiving their full message of “togetherness,” which should be (but isn’t) emphasized by the imagery of the rowing team working in perfect unity.

Although the issue visually appears to be the same as the issues in Taco Bell’s mobile display, the hidden problem is actually quite different. Because Chrome devtools employs a different rendering engine, it doesn’t display the missing background image. Specifically, Chrome’s mobile emulator on the desktop is parsing and displaying results different than the actual browser on the iPad tablet.

A Comparison: Mobilizer vs. Chrome DevTools for Mobile

Let’s consider an alternative: Mobilizer’s online mobile testing platform, which delivers fast screenshots of your website on actual market-leading phones and tablets. Both Chrome device mode and Mobilizer offer distinct advantages for marketing teams. This chart highlights how the two tools offer complementary functionality to ensure easy, bug-free mobile development and a glitch-free, superlative user experience.

Advantages and Utility

Mobilizer Chrome DevTools (F12)
● Renders web pages on actual market-leading devices, ensuring 100% accuracy

● Tests across device OS versions

● Side-by-side display of devices enables fast identification of errors

● Simple, intuitive interface makes it easy for the whole marketing team, not just the developers

Mobile-focused analytics can reveal performance issues and suggest where to focus your team’s time

● Enables workflow collaboration with easy-to-share mobile screenshots, reports, and Jira tickets

● Automates delivery of renders of web pages and proactive notification of issues

● Chrome Extension app for fast mobile testing of web pages

● Emulates any device resolution, but only the viewport size is guaranteed to be accurate

● Allows for interactivity

● Examines code segments in detail and their display

● Offers a wide variety of tools that are useful to developers in building web applications

● Built into Chrome

● Free

When an Emulator Just Won’t Do

Chrome’s web developer tools are an excellent addition to the digital marketer’s toolset. But just as every tool has its special purpose for which it is perfectly adapted, so too the Chrome emulator best accomplishes only certain tasks. It can accelerate mobile web development by displaying your web page on any device next to its code. But development and QA for the mobile web are dangerously incomplete if you deploy only an emulator.

Before your website is set to go live and be seen by thousands, if not millions, of prospects and customers, you want 100% accurate testing and easy, fast inspection. You need to test your crafted new page design on the real thing, whether an online mobile lab or a phone sitting in the palm of your hand, not a simulated imitation.

Mobile1st is a leader in marketing technology and site optimization solutions. More than prioritizing and executing industry leading CRO practices, we do what makes the money; optimizing our Client’s digital commerce product toward consistent financial growth and ROI. All achieved through excellence in Lean UX, Analytics, Research, Experimentation and Development.
Contact Mobile1st Experts Now!