Device Testing in Mixed Environments

Are you a device test manager working for a carrier? Are you tasked with testing in a mixed environment with multiple manufacturers across multiple models? Perhaps your environment is full of high-, mid- and low-tier devices?

If these parameters are true for you and you are testing on Android, chances are you might be struggling to compile accurate data—especially if you are working in a manual environment and have gleaned that Android devices are greatly wide-spread and multi-faceted.

Thus, your foundation and approach is paramount to your task(s) being successful and not appearing so daunting.

By using MCellBlock (MCB), a device dense solution that tests up to 32 mobile phones or 16 tablets (up to 8”) at one time, you create a systematic, highly organized data set that helps you present a clear portfolio of test data to your executive management. This helps them ultimately understand the ROI received from MCellblock—and, you will look like a rock star.

Here’s how.

How does MCellBlock help my testing process?

Speed and ease is key – With MCB, once you configure the web UI, it’s as simple as pushing a button and  running test scripts across multiple devices, across tiers, and against multiple manufacturers. This process can be done as quickly as ten minutes- barring normal testing challenges.

Data Accuracy – With MCB, there are baseline scripts available to you upon procurement of the product. The manufacturer makes v. –  version 7.0  and later baseline test scripts. You will be supplied with scripts created to run across various devices and various operating systems.

Space allocation for testing – MCB is designed to fit into your current data center, alongside other rack mountable components. With its 4U-size dimensions, the hardware doesn’t take up much space in lieu of how dense it is. And because of its multiple onboard temperature sensors, MCB ensures the devices contained within it are maintained at a constant temperature.

Friendlier to a budget – MCB fits into budgets easily by reducing the need for multiple teams of manual testers. The reduction in costs for manual testing labor is realized very rapidly; When you purchase MCB on your cap ex budget and claim depreciation over time, you affect your ultimate bottom line. MCB takes the place of formerly four to five engineers handling test scripts and devices, so one engineer can run and share access to other teams from a remote location via the web interface for even greater cost savings.

The Five-Step Approach to Benchmarking Devices

1) Evaluate your devices to be tested and divide them into tiers. For example, categorize all mid-tier devices together, lower-tier devices together, and all of the high-tier devices together—this is regardless of manufacturer. Then, divide the devices into tiers to create an apples to apples evaluation.

2) Decide what you are looking to get out of your test plan and determine a baseline set of test scripts.
Construct a number of (let’s just say ‘five’) test scripts that you know you can run against ALL of the low, mid, and high-tier devices. Then, adjust accordingly per criteria regardless of different screen sizes, cameras, chip size, or other variables. Build one set of baseline scripts so that your evaluation of the various devices will be meaningful.

Challenge: We know It becomes difficult at times to categorize in this way because as Android evolves, the predecessors can become obsolete. To that end, the test scripts can also be challenging to script because of the aged versions. However, even if it is not completely doable to craft something that tests across the board –you may need to simply craft a test script from v. 6.0 of Android and later.

3) Start to add more test scripts per capability or device type after you’ve collected your baseline data and begin to test the limits.
Decide your order of events and begin to graph or chart accordingly per hierarchy; e.g. – We recommend first testing low to mid, then mid to high, and so on. Begin to answer how each device performed within those specific tiers based on individual criteria: memory, chipset, speed, or other variable.

Know that your higher capability devices will likely produce ‘better’ results as you move up the chain. However, you can still see how far you can push the lower tiered devices by taking a mid-tier script and running it on the low-tier device (if you want to present even more complete data).

4)  Drill down even further.
Within various tiers, and/or within various manufacturers, begin to compare meticulously. For example, if you were comparing chipsets, you might compare the Qualcomm and Media Tech chipsets or the Media Tech and the Samsung chipsets. How did Brand A device perform with these particular chip sets versus Brand B device?

If you can say that Brand A device performed really well or, conversely, performed really poorly – can you attribute the performance back to the chip set? If so, you can determine what the optimal chipset is (or optimal memory or speed rate) to give your executive team leverage to negotiate with the manufacturers. This can result in driving customer satisfaction rates up later. Remember, your company wants to compete with the next carrier at the same price point. It is your job to help them determine the performance of each specification accurately.

5) Wrapping up your testing, reporting on your results.
After you’ve used MCellBlock to help you collect your test data across the various devices and you have compared the results, you can stack rank the devices across each category and/or manufacturer in the various tiers. MCB allows you to export your test data and import it into your favorite data tool, e.g. Excel, so that you can create tables and graphs of your benchmark data. This also allows you to be able to save the data in your preferred location so you can access it later to easily show executive committee if you’re required.

With MCB, you no longer need to dread device testing in mixed environments. Because speed, ease, data accuracy, space allocation, and budget friendliness are in your corner, you can execute on your five-step process knowing a sound framework backs you the whole way.

Thoughts and approach in this post offered by contributing blogger, Paul Tedrow 



Leave a Reply

Your email address will not be published. Required fields are marked *