In part I, I compared the 2012 Nexus 7 before and after a reboot. In part II, I compared the 2012 Nexus 7 running Android 4.2 and with the same tablet running Android 4.3.
Today, I compare the 2012 Nexus 7 running Android 4.3 with the brand new Nexus 7.
The new Nexus 7 features a new CPU and GPU, a refined design, the latest Android version, both front and rear cameras, and a bunch of other improvements. Personally, I'm impressed with how good it feels to hold; the slightly narrower bezel really does make a huge difference. But, I digress... onto the benchmarks!
For these tests, I will be using the AnTuTu benchmark app.
For
each configuration, I ran the benchmark ten times. For the original Nexus 7, the average of the benchmark results was 11,977 with a
standard deviation of 178. The new Nexus 7 scored an impressive average of 20,249 with a
standard deviation of 240. The new Nexus 7 is clearly much faster!
As for the subsystem sections of the results, the CPU scored on average 28% faster. The RAM was 42% faster and the I/O was 7% faster. But the GPU really took the crown with performance that was 169% faster. I could see the difference in the graphics benchmarks: whereas before, a couple of the tests had such low framerates that it looked like a slideshow, now those same tests are much smoother.
It's not strictly necessary in a case like this where the performance is clearly improved, but for fun I went ahead and performed a t-test, which is a statistical method
for determining how much of the difference between two sets of
measurements is attributable to the variable that changed. The average difference between the two trials was 8272 (with a standard
deviation of 295) or roughly a 69% performance increase. And the t-test confirmed that the >8000 point difference was due to the actual difference between the tablets (and not just statistical noise) with a >99% probability.
Wednesday, July 31, 2013
Sunday, July 28, 2013
Nexus and Android Benchmarks - Part II
Part I is here.
In this post, I'm going to compare the 2012 Nexus 7 with Android 4.2 to itself with Android 4.3. Android 4.3 offers a few performance enhancements, most relating to OpenGL. Specifically, the new version includes OpenGL ES 3.0. This benchmark only runs ES 2.0 tests, but I'm hoping to see at least some improvement in the GPU tests.
For each configuration, I ran the benchmark ten times. For the Nexus 7 with Android 4.2, the average of the benchmark results was 12,019 with a standard deviation of 134. After installing Android 4.3, the average was 11,977 with a standard deviation of 178. The results are similar, but the new version scores slightly worse.
What I did then was performed a t-test, which is a statistical method for determining how much of the difference between two sets of measurements is attributable to the variable that changed. In this case, I wanted to see how much of the change in performance was attributable to Android 4.3, as opposed to statistical variation in the measurements.
The average difference between the two trials was -42 (with a standard deviation of 141) or roughly a .5% performance decrease. However, the t-test gave a result of <1, which is completely negligible.
The takeaway message is that you're not going to see an improvement performance-wise in going from Android 4.2 to 4.3. Instead, the new version appears to focus more on new features, such as restricted profiles and OpenGL ES 3.0.
In this post, I'm going to compare the 2012 Nexus 7 with Android 4.2 to itself with Android 4.3. Android 4.3 offers a few performance enhancements, most relating to OpenGL. Specifically, the new version includes OpenGL ES 3.0. This benchmark only runs ES 2.0 tests, but I'm hoping to see at least some improvement in the GPU tests.
For each configuration, I ran the benchmark ten times. For the Nexus 7 with Android 4.2, the average of the benchmark results was 12,019 with a standard deviation of 134. After installing Android 4.3, the average was 11,977 with a standard deviation of 178. The results are similar, but the new version scores slightly worse.
What I did then was performed a t-test, which is a statistical method for determining how much of the difference between two sets of measurements is attributable to the variable that changed. In this case, I wanted to see how much of the change in performance was attributable to Android 4.3, as opposed to statistical variation in the measurements.
The average difference between the two trials was -42 (with a standard deviation of 141) or roughly a .5% performance decrease. However, the t-test gave a result of <1, which is completely negligible.
The takeaway message is that you're not going to see an improvement performance-wise in going from Android 4.2 to 4.3. Instead, the new version appears to focus more on new features, such as restricted profiles and OpenGL ES 3.0.
Saturday, July 27, 2013
Nexus and Android Benchmarks - Part I
In a three-part series, I'm going to compare the relative performance of the original and new Google Nexus 7 tablets. The new Nexus 7 comes with Android 4.3; the 2012 Nexus 7 had Android 4.2
on it, but these tablets will be getting an over-the-air (OTA) update
during the next couple of weeks. Unlike most other comparisons you'll see, I spent a little extra time to try and extract some statistically relevant data.
For these tests, I will be using the AnTuTu benchmark app. Why this app? Well, it breaks the results into CPU, GPU, Ram, and I/O, so you can see how the different components of the system are performing. Also, it's a free download and it has a good reputation for producing reliable benchmark results.
In this post, I'm going to compare the 2012 Nexus 7 with Android 4.2 to itself, before rebooting and after. Before rebooting, the tablet had been on and in daily use for over a month. One difference between iOS and Android devices is in the way they handle multitasking and what I wanted to see was if having other apps running in the background resulted in a drop in performance.
For each configuration, I ran the benchmark ten times. For the Nexus 7 before rebooting, the average of the benchmark results was 11,910 with a standard deviation of 100. After rebooting, the average was 12,019 with a standard deviation of 134. So, they look comparable, but the reboot seems to have helped a bit.
What I did then was performed a t-test, which is a statistical method for determining how much of the difference between two sets of measurements is attributable to the variable that changed. The example that is typically given is that if you have two sets of cancer patients and you give one set an experimental drug, you want to tell how much of their improvement is attributable to the drug. In this case, I wanted to see how much of the change in performance was attributable to the reboot, as opposed to statistical variation in the measurements.
The average difference between the two trials was 109 (with a standard deviation of 141) or roughly a 1% performance increase. However, the t-test gave a result of 2.44 (with a probability of 96%), which is a .02% performance increase; the 1% increase that the average shows is mostly statistical noise. So the actual performance increase is completely negligible.
The takeaway message is that Google has done a very good job in implementing multitasking in Android and rebooting your device won't help you eek out any additional performance.
For these tests, I will be using the AnTuTu benchmark app. Why this app? Well, it breaks the results into CPU, GPU, Ram, and I/O, so you can see how the different components of the system are performing. Also, it's a free download and it has a good reputation for producing reliable benchmark results.
In this post, I'm going to compare the 2012 Nexus 7 with Android 4.2 to itself, before rebooting and after. Before rebooting, the tablet had been on and in daily use for over a month. One difference between iOS and Android devices is in the way they handle multitasking and what I wanted to see was if having other apps running in the background resulted in a drop in performance.
For each configuration, I ran the benchmark ten times. For the Nexus 7 before rebooting, the average of the benchmark results was 11,910 with a standard deviation of 100. After rebooting, the average was 12,019 with a standard deviation of 134. So, they look comparable, but the reboot seems to have helped a bit.
What I did then was performed a t-test, which is a statistical method for determining how much of the difference between two sets of measurements is attributable to the variable that changed. The example that is typically given is that if you have two sets of cancer patients and you give one set an experimental drug, you want to tell how much of their improvement is attributable to the drug. In this case, I wanted to see how much of the change in performance was attributable to the reboot, as opposed to statistical variation in the measurements.
The average difference between the two trials was 109 (with a standard deviation of 141) or roughly a 1% performance increase. However, the t-test gave a result of 2.44 (with a probability of 96%), which is a .02% performance increase; the 1% increase that the average shows is mostly statistical noise. So the actual performance increase is completely negligible.
The takeaway message is that Google has done a very good job in implementing multitasking in Android and rebooting your device won't help you eek out any additional performance.
Subscribe to:
Posts (Atom)