Mobile1st offers mobile optimization services to increase your mobile website’s revenue/conversion by decreasing your shopping cart abandonment rate, reducing your page load speed, improving mobile UX and SEO, analyzing your mobile analytics, and more. Our team of mobile experts analyzes and improves your website across 35+ factors, A/B tests ideas, and can even (optionally) implement the design/code changes.
Get a free mobile audit of your website today!

BrowserStack is a popular online tool that lets developers test their website on a remarkable range of browsers in combination with multiple versions of operating systems. That’s a welcome relief for WebDev and eCommerce teams trying to ensure a great user experience across an impossibly vast range of browsers and devices.

Recently, BrowserStack extended their testing capabilities from desktops to mobile, adding value for hard-pressed designers, developers and marketers. And we wondered: Exactly how good is the BrowserStack tool for testing mobile performance?

The short answer – offered by both leading research organization Gartner and on-the-ground website developers – is: BrowserStack Mobile, like most mobile emulators, is “useful for early functional testing, but is not a substitute for access to real devices.”

Or as Tom Maslen at Smashing Magazine reports, “[Emulators] really help you to understand how your code behaves on different platforms, but they don’t tell you everything you need to know about the UX.“ That’s why Tom advocates setting up your own device testing farm and Gartner recommends utilizing an online testing lab (yes, one like Mobilizer).

---

BrowserStack is upfront. Its mobile tests results are based on mobile emulators, not real devices. They’re only a simulation. BrowserStack lets you test on “the official Google Android emulator, Apple iOS simulator and Opera Mobile emulator.”

“It’s great to have access to almost every conceivable OS and browser combination, but [BrowserStack] can be very sluggish and hard to test interactions. I usually resort to using it for bug fixing rather than regular website testing. BrowserStack relies on mobile device emulators, so it can’t be relied on 100% for a native experience.”Jordan Lewis, Front End Developer, Envato

And they add, “This is the closest that one gets to a real device.”

But is “close” good enough? What if close hides failure? What if less than perfect rendering fails to disclose your website’s bugs, glitches, and errors? Emulator-based tools use an application, typically supplied by the OS manufacturer, which are coded to simulate the operations of the mobile devices, right on your desktop.

Here’s why mobile emulators like BrowserStack fall short. Emulators don’t necessarily test the same rendering engine as used on the actual device. That rendering engine is used by the browser to interpret the HTML and CSS code, and consequently, mobile emulators can miss serous design flaws and coding bugs. You must check your website on the actual physical devices to see what your site visitors will truly experience.

Furthermore, emulators don’t always take into account the details of the native processing on each device. For example, the iPhone 6 Plus has a “downsampling” process that’s not used on most other devices and it certainly impacts how a page is displayed. Again, critical variables in the mobile UX are left out by emulators.

It’s simple. There’s no replacement for testing on real devices. But what kinds of errors do mobile emulators generate?

1. Defective Default Elements

Each browser and operating system renders default elements, such as buttons and scrollbars, differently. Emulators neglect these differences, misrepresenting the way your site appears. While the differences can be minor, they could have a huge effect on your users’ experience, including fundamentally disrupting branding.

Such deviant presentations of the brand are doubly problematic in this era where the typical customer journey (67% says Forbes) traverses multiple platforms — from desktop to tablet to mobile.

[It’d be good to show an example of a BrowserStack fail, not sure that’s possible]

Nike on Galaxy Tab Pro 8.4 on Mobilizer

2. Failing to Reveal Missing Videos

Viewing videos on mobile phones has never been bigger! Yes, customers, whether business or consumers, love video. One small problem – many phones don’t have the player installed (i.e., Flash) or have a hard time playing it with the default browser. Emulators unfortunately may fail to catch this frustrating defect.

BrowserStack’s emulators, even if provided by Apple, Google, etc., aren’t the actual devices, and that allows identifiable errors to creep in. Worse, by spending many an hour testing your site on BrowserStack’s mobile emulators, your guard may be lowered. You may be lulled by a “false positive’ — believing your website is free of bugs and glitches on mobile when sadly it’s not.

Beyond its failure to accurately render your website on mobile, BrowserStack possesses other drawbacks that make it less than an ideal tool. Web developers and QA teams complain, sometimes bitterly, about their user experience.

---

1. Slow. “BrowserStack can be very sluggish,” says developer Jordan Lewis. Even BrowserStack recognizes this issue. Consider this FAQ from their site:

• Q: Why is my remote browser display taking so long to respond?
• A: As we provide remote access to browsers, your experience is not going to be the same as working with browsers installed on your own machine. For mobile testing, we provide official emulators for iOS, Android and Opera mobile devices, which are inherently slow in nature.

But some BrowserStack users are a bit more harsh in their criticisms.

---

2. Complicated. More browsers than you can count and multiple operating systems, but usually simpler is better for getting things done.

3. Not focused on mobile.

4. Limited functionality. BrowserStack Mobile doesn’t come with:

• Mobile data analytics that can help you spot which mobile devices are delivering suboptimal performance, potentially caused by bugs. In contrast , Moblizer details such metrics as load time, bounce rate, sales conversion rate, etc. broken out by each mobile device alongside the visual renders.
• Alerts warning you when there are sudden shifts in your website’s performance on mobile
• Side-by-side scrolling through your website on different mobile devices to facilitate the identification of errors

BrowserStack is a good developer tool, facilitating testing across most every browser and device. Unfortunately, it can be slower than molasses. And you need to treat it with caution when you are doing QA or optimizing your website on mobile. BrowserStack’s delivery of false positives — the illusion of a well-functioning site when there still remain bugs to be squashed — can lull the unwary digital marketer into complacency. In the final analysis, there’s no substitute for real devices for final testing and fully accurate results.

---

TRY MOBILIZER TODAY

Mobile1st is the company behind Mobilizer, the world’s first mobile performance platform. Mobilizer helps companies increase revenue and engagement by enabling them to see exactly what their mobile users see, quickly identify display issues, monitor metrics by device, and optimize the mobile customer experience.