Through the build of Mobilizer and interviewing peers in the digital community, I’ve realized collectively we still suck at mobile testing. And believe me, it pains me to say that because I feel that in general the digital community does a great job of doing things the right way.
I confessed in my talk that I hate QA. It’s at the end of the line in the process and historically I haven’t given it the proper attention. What’s worse, is that even after all of my years in digital, I knew my developers sometimes only tested on emulators and I didn’t know how to efficiently fix the problem.
What’s a PM to do you ask?
First things first, do you know the difference in testing tools? Here a basic categorization of types of testing tools.
While all of these product examples have uses and you can’t get through the process with just one tool, the important thing to understand and challenge your development team on is if they are only using mobile emulators, you can’t ship the product. If your dev team disagrees, check out this article that our CTO wrote with examples of how an emulator can’t replace real device testing.
Trust no one – you’re the final sign off
I joke frequently at the office that I don’t trust anyone and what I mean by that is it’s the PM’s responsibility to do the final check before launch. Yes, you should be able to count on your team but the PM knows all of the small minor details and the big picture vision so it makes sense that you should have some high level involvement with the QA process.
Testing on devices one by one is time consuming and expensive – that’s not the solution for a PM
Having an internal device lab has value and nothing will ever replace the full experience of interacting with the device. But the problem that I faced was the cost of purchasing new devices every year (roughly 5 – 20k), keeping their software updated (average hourly rate $150 * hours spent maintaining devices) and finding them at the testing station instead of at someone’s desk not charged.
To add to that complexity is every product has a different demographic. In some instances you will need to primarily test on the latest devices with the latest OS and in other cases you will need to test on older devices with older OS. So even if you purchased a set of the most popular devices and kept them updated, that may not be thorough enough for comprehensive testing. You also have to consider popular devices by country. If the user base is international, that exponentially increases the cost of device acquisition.
You can also opt to use a device lab service like Perfecto Mobile but you are still forced to test on one device at a time.
The problem that the Mobilizer Device Lab solves is a quick and easy way to see how the pages are rendered.
We wanted to provide a solution that allowed you to spot check in a quick and efficient way. We created a device lab and coded an iOS and Android app that pulls up the website you have requested and returns a full page screen shot for your review (I only included a partial screen shot below).
You’re then able to download a report to send off to stakeholders as a final delivery item to show how sites are rendering across devices. You can also pass the report along to developers in the QA process to show them exactly what issues you are seeing. We include render times and other meta data in the report.
We’re still building out the product and getting feedback from our peers on how to continue to solve the pain points of mobile testing. We welcome any thoughts and suggestions you have to help us product road map out the next set of features so we can better serve the awesome digital community we’re apart of.