Nexus and Android Benchmarks - Part I

In a three-part series, I'm going to compare the relative performance of the original and new Google Nexus 7 tablets. The new Nexus 7 comes with Android 4.3; the 2012 Nexus 7 had Android 4.2 on it, but these tablets will be getting an over-the-air (OTA) update during the next couple of weeks. Unlike most other comparisons you'll see, I spent a little extra time to try and extract some statistically relevant data.

For these tests, I will be using the AnTuTu benchmark app. Why this app? Well, it breaks the results into CPU, GPU, Ram, and I/O, so you can see how the different components of the system are performing. Also, it's a free download and it has a good reputation for producing reliable benchmark results.

In this post, I'm going to compare the 2012 Nexus 7 with Android 4.2 to itself, before rebooting and after. Before rebooting, the tablet had been on and in daily use for over a month. One difference between iOS and Android devices is in the way they handle multitasking and what I wanted to see was if having other apps running in the background resulted in a drop in performance.

For each configuration, I ran the benchmark ten times. For the Nexus 7 before rebooting, the average of the benchmark results was 11,910 with a standard deviation of 100. After rebooting, the average was 12,019 with a standard deviation of 134. So, they look comparable, but the reboot seems to have helped a bit.

What I did then was performed a t-test, which is a statistical method for determining how much of the difference between two sets of measurements is attributable to the variable that changed. The example that is typically given is that if you have two sets of cancer patients and you give one set an experimental drug, you want to tell how much of their improvement is attributable to the drug. In this case, I wanted to see how much of the change in performance was attributable to the reboot, as opposed to statistical variation in the measurements.

The average difference between the two trials was 109 (with a standard deviation of 141) or roughly a 1% performance increase.  However, the t-test gave a result of 2.44 (with a probability of 96%), which is a .02% performance increase; the 1% increase that the average shows is mostly statistical noise. So the actual performance increase is completely negligible.

The takeaway message is that Google has done a very good job in implementing multitasking in Android and rebooting your device won't help you eek out any additional performance.