Wednesday, July 31, 2013

Nexus and Android Benchmarks - Part III - 2012 vs. 2013 Nexus 7's

In part I, I compared the 2012 Nexus 7 before and after a reboot. In part II, I compared the 2012 Nexus 7 running Android 4.2 and with the same tablet running Android 4.3.

Today, I compare the 2012 Nexus 7 running Android 4.3 with the brand new Nexus 7.

The new Nexus 7 features a new CPU and GPU, a refined design, the latest Android version, both front and rear cameras, and a bunch of other improvements. Personally, I'm impressed with how good it feels to hold; the slightly narrower bezel really does make a huge difference. But, I digress... onto the benchmarks!

For these tests, I will be using the AnTuTu benchmark app. For each configuration, I ran the benchmark ten times. For the original Nexus 7, the average of the benchmark results was 11,977 with a standard deviation of 178. The new Nexus 7 scored an impressive average of 20,249 with a standard deviation of 240. The new Nexus 7 is clearly much faster!

As for the subsystem sections of the results, the CPU scored on average 28% faster. The RAM was 42% faster and the I/O was 7% faster. But the GPU really took the crown with performance that was 169% faster. I could see the difference in the graphics benchmarks: whereas before, a couple of the tests had such low framerates that it looked like a slideshow, now those same tests are much smoother.

It's not strictly necessary in a case like this where the performance is clearly improved, but for fun I went ahead and performed a t-test, which is a statistical method for determining how much of the difference between two sets of measurements is attributable to the variable that changed. The average difference between the two trials was 8272 (with a standard deviation of 295) or roughly a 69% performance increase. And the t-test confirmed that the >8000 point difference was due to the actual difference between the tablets (and not just statistical noise) with a >99% probability.

Sunday, July 28, 2013

Nexus and Android Benchmarks - Part II

Part I is here.

In this post, I'm going to compare the 2012 Nexus 7 with Android 4.2 to itself with Android 4.3. Android 4.3 offers a few performance enhancements, most relating to OpenGL. Specifically, the new version includes OpenGL ES 3.0. This benchmark only runs ES 2.0 tests, but I'm hoping to see at least some improvement in the GPU tests.

For each configuration, I ran the benchmark ten times. For the Nexus 7 with Android 4.2, the average of the benchmark results was 12,019 with a standard deviation of 134. After installing Android 4.3, the average was 11,977 with a standard deviation of 178. The results are similar, but the new version scores slightly worse.

What I did then was performed a t-test, which is a statistical method for determining how much of the difference between two sets of measurements is attributable to the variable that changed. In this case, I wanted to see how much of the change in performance was attributable to Android 4.3, as opposed to statistical variation in the measurements.

The average difference between the two trials was -42 (with a standard deviation of 141) or roughly a .5% performance decrease.  However, the t-test gave a result of <1, which is completely negligible.

The takeaway message is that you're not going to see an improvement performance-wise in going from Android 4.2 to 4.3. Instead, the new version appears to focus more on new features, such as restricted profiles and OpenGL ES 3.0.






Saturday, July 27, 2013

Nexus and Android Benchmarks - Part I

In a three-part series, I'm going to compare the relative performance of the original and new Google Nexus 7 tablets. The new Nexus 7 comes with Android 4.3; the 2012 Nexus 7 had Android 4.2 on it, but these tablets will be getting an over-the-air (OTA) update during the next couple of weeks. Unlike most other comparisons you'll see, I spent a little extra time to try and extract some statistically relevant data.

For these tests, I will be using the AnTuTu benchmark app. Why this app? Well, it breaks the results into CPU, GPU, Ram, and I/O, so you can see how the different components of the system are performing. Also, it's a free download and it has a good reputation for producing reliable benchmark results.

In this post, I'm going to compare the 2012 Nexus 7 with Android 4.2 to itself, before rebooting and after. Before rebooting, the tablet had been on and in daily use for over a month. One difference between iOS and Android devices is in the way they handle multitasking and what I wanted to see was if having other apps running in the background resulted in a drop in performance.

For each configuration, I ran the benchmark ten times. For the Nexus 7 before rebooting, the average of the benchmark results was 11,910 with a standard deviation of 100. After rebooting, the average was 12,019 with a standard deviation of 134. So, they look comparable, but the reboot seems to have helped a bit.

What I did then was performed a t-test, which is a statistical method for determining how much of the difference between two sets of measurements is attributable to the variable that changed. The example that is typically given is that if you have two sets of cancer patients and you give one set an experimental drug, you want to tell how much of their improvement is attributable to the drug. In this case, I wanted to see how much of the change in performance was attributable to the reboot, as opposed to statistical variation in the measurements.

The average difference between the two trials was 109 (with a standard deviation of 141) or roughly a 1% performance increase.  However, the t-test gave a result of 2.44 (with a probability of 96%), which is a .02% performance increase; the 1% increase that the average shows is mostly statistical noise. So the actual performance increase is completely negligible.

The takeaway message is that Google has done a very good job in implementing multitasking in Android and rebooting your device won't help you eek out any additional performance.






Friday, April 5, 2013

Worry Less About Moore's Law

You can't read more than a couple of articles about upcoming CPU's or chipsets without running into one claiming that Moore's Law is coming to an end. Moore's Law is usually (and incorrectly) stated as "microprocessors double in speed every 18 months". They then claim that with the end of Moore's Law, we'll stop seeing improvements in computer technology.

The first thing wrong with Moore's Law is that it's not a law, but an observation. Laws are unbreakable, but observations are. Second, Gordon Moore said that it was the transistor density that was doubling, not the speed. And third, he noticed that it was doubling every two years, not eighteen months. And fourth, the observation was made regarding silicon semiconductor technology.

Interestingly, Moore made the observation in 1965 using data from 1958 to the then-present. From there, the Law has actually held up pretty well for the last fifty years. The reason for this may simply be a case of the prediction leading the technology. If transistor densities have doubled like clockwork and you know your competition is also working on the next doubling, then you are well-motivated to make it happen.

Even so, Moore's Law is coming to an end. Silicon transistors can only get so small; single atom transistors are as absolutely small as you could ever hope to get and for silicon, that's about 0.25nm. Right now, Intel's latest chips use 22nm technology (which is half of the feature width). So, there is a limit and we are approaching it. There is no way that transistor densities can keep doubling for the next 20 years without having to have transistors smaller than a single atom.

But I'm not worried and here's why. First, Moore's Law makes a prediction about transistor size, not processing speed. Current chips have over a billion transistors on them and when you have that much wiring, there is invariably room for improvement. In fact, the current battle between Intel's x86 instruction set and ARM's RISC instruction set is evidence of that. If there was a "right" way to hook up a billion transistors, Intel would do it and be done with it. But the fact that they have to compete with ARM and AMD shows that there is still competition in the architecture regime.

Also, even for a fixed transistor density, you can increase the number of transistors by increasing the chip area or going three-dimensional. All chips today are made in nearly flat 2D layers, but research has been done into layering transistors. If this were accomplished, you would have an overnight doubling of transistors on a chip. And since each layer is only a few nanometers thick, you can have thousands of layers before the chip becomes appreciably thicker. Of course, there are a lot of challenges in this arena, specifically in heat dissipation and signaling, but it is a way forward.

Moore's Law is also an observation on silicon semiconductor integrated circuit technology. The original electronic transistors were vacuum tubes, which were then replaced by individual transistors. Those were replaced by silicon integrated circuits which we use today. It is inevitable that someday we will move away from silicon to something better. We may figure out how to make transistors at the subatomic level, or quantum computing might finally mature. But most likely, it will be some technology that we haven't even heard of yet that will replace it.

In the short term, Moore's Law looks like it still has a bit of life left in it. I expect that within this decade we will see the first signs of a transition away from silicon to a similar technology (such as gallium-arsenide transistors). Then, shortly thereafter we'll see a transition away from the integrated circuit technology that has been the center of computing and technology for more than half a century.

If you feel like checking it out for yourself, Intel has made a copy of the original 1965 paper available on it's website.






Sunday, March 24, 2013

Review: Visual C# 2008 in 24 Hours

Sams Teach Yourself Visual C# 2008 in 24 Hours

Even though were a few months into 2013 and Visual Studio 2012 has been out for a year, I picked up James Foxall's Visual C# 2008 as a quick way to start working with C# for a job-related assignment. I have a ton of C, C++, and Java experience, but no experience with C# and I haven't worked with the Visual Studio IDE's in quite a while. I needed a jump start into the language and IDE, and that's what I got.

One concern I had was how relevant the book would be, considering that it is five years old and is specific to a two-versions-old edition of Visual Studio. It turns out that it wasn't a problem at all. The visual style of the IDE has changed a little, but the locations and names of the elements have not. The code samples all compiled and ran without modification. In one of the later chapters, he covers code automation using Microsoft Word and Excel. He coded for Office 2007, but I ran these samples against Office 2010 without a problem.

One possible age-related problem is that the link to the sample code given in the text is no longer correct. You can find it at both http://www.informit.com/content/images/9780672329845/examples/Examples.zip and http://www.jamesfoxall.com/downloads/TYCS2008samples.zip. With the exception of a couple of images used in the projects, I didn't find this file very helpful. After all, learning to code is not a spectator sport and if you're just copying code samples, then you're not doing it right.

As for the book, it's fairly well laid out and it does quite a bit of hand holding. It assumes very little coding experience. As is the case with all of the books in Sams Teach Yourself ___ in 24 Hours series, it is split up into 24 chapters, or "hours". Not a single hour took me the full hour to complete. Some took as little as fifteen minutes and most averaged around 30-45 minutes.

There is a good deal of breadth in the text (but not a lot of depth). The book covers Visual Studio and designing applications graphically, objected oriented programming, basic programming techniques (like if-statements and for-loops), manipulating the registry, managing files, working with databases, controlling other applications with automation, and creating an installer. Of course, none of these subjects is covered in much detail. For example, the database chapter assumes that someone else has designed a database for you; the text only shows you how to pull records out and insert new ones.

For me, the book was exactly what I needed. An experienced programmer who wants a jump-start into C# development will find what they need here. If this is you, I would suggest that you just skip chapters 10 through 14; oddly he places the basic programming techniques here in the middle of the text. This book is supposedly geared toward new programmers, but I think being thrown into the deep end for the first 200 pages and only then being told what a for-loop is would be confusing. It's an odd choice and for that reason, I wouldn't recommend this book if you don't already know the basics.


Tuesday, March 19, 2013

Self-Destructoid

The gaming website Destructoid published a piece last week where their founder, Niero Gonzalez, bemoaned the loss of revenue due to ad-blockers. His basic argument was that the future of his site and sites like his are in danger because so many users are blocking all ads by default that they can no longer receive enough revenue to keep in business.

Micropayments have never really taken off and subscriptions only seem to work if you have a major news website like the New York Times or Wall Street Journal. For the smaller players, it seems that ads are the only realistic source of revenue. And if too many users start to use ad-block, then you're surely going to be driven out of business.

Niero bases most of his argument on data collected by a third-party tool called BlockMetrics and it showed that 42-46% of his viewers are blocking ads. BlockMetrics also allows you to enter your current CPM (which is a measure of the revenue you get per 1000 ads shown) and then it calculates how much money you have lost on non-ad-viewing readers. For example, if you show 100,000 ads daily at $1.20 CPM, you're making $120 per day. And if 45% of your readers have ad-block enabled, then BlockMetrics would calculate that you could be showing 181,800 ads daily and therefore you're effectively losing $98.18 in revenue every day.

Lets imagine for a moment that there was a magic switch and Niero could force all of his users to see ads. On day 1, he would get nearly a doubling of ad revenue, but things would surely change from there.

Websites sell ads on the basis of the number of views. Advertisers buy ads on the basis that they will cause people to buy their product. There is a disconnect between what the website is selling and what the advertiser is buying and this is resolved through the CPM. Websites that can drive clicks (and ultimately purchases) to a product can charge a higher CPM; ones that can't watch their CPM fall. If Niero was able to force the 45% of his audience who doesn't want to see ads to see them, would these people really click on ads at the same rate that the non-ad-blockers do? They would click some, but it only stands to reason that the rate would be somewhat less, which would drive down his CPM. Showing ads to people who don't want to see them is not going to lead to more sales for the advertiser.

The other consideration is that the people who are blocking ads are doing so because ads were so repulsive to them that they were willing to spend time and effort to find, download, install, and setup an ad-blocker. Admittedly, if you know what you're doing, this doesn't take even five minutes, but still these people were bothered enough to do it. If they are forced to view ads on one website, but not on another, which site are they going to view? Gaming websites, like Destructoid, are numerous and if you don't like one, it's not hard to find another. If Niero forced his audience to view ads, then his audience would surely shrink. And since most traffic to websites is driven by people sharing on Facebook, tweeting on Twitter, posting to links on blogs, et cetera, the loss of the ad-blocking audience would surely result in some drop-off in the non-ad-blocking audience as well.

Unfortunately, BlockMetrics data only shows you part of the picture and it makes it seem like you're leaving money on the table. No one likes to think that half of their income is being stolen. But the truth is that an ad-block rate of 0% would be mostly canceled out with a decrease in CPM and a decrease in readership. Without ad-blockers, Destructoid's revenue might go up, but it would be a modest increase at best.

My advice? Don't go chasing every nickle. You'll make yourself mad trying to do so. It's like Google's Amit Singhal said last week about SEO: "If you build high-quality content that adds value, and your readers and your users seek you out, then you don't need to worry about anything else." Niero should focus on making his content the best it can be and the advertising revenue is sure to follow.

Sunday, March 17, 2013

Review of TP-Link 200Mbps Powerline Ethernet Starter Kit

Today, I'm going to post a quick review of the TP-Link TL-PA2010KIT AV 200Mbps Nano Powerline Adapter Starter Kit. A video review I made follows and it includes some setup instructions and other details not in this review, so its worth watching.

TP-Link Nano Powerline Ethernet Adapters
Pros: In the box are the two adapters and two 2-meter cables. Take each adapter, plug it into the wall, plug the cable into it and the computer or router and you're done. A lot of devices say "get started in seconds", but for this one, it's really true. There are no configuration settings to play with. The units are glossy white, fairly small, and they don't block the other plug. The pins are unpolarized, so you can plug it in upside-down - this lets you plug both units into the same outlet (but there's no practical reason to do this, other than for testing). The front of the unit has a synchronization button, if the units ever become unpaired. There are three blue lights on the front to help you diagnose a problem, if you ever have one.

It works flawlessly with Windows and Linux and the computers see the cable as they would any other Ethernet cable - the computer has no idea that its using powerline adapters.

Cons: This device will not work on an outlet with a surge protector. I tried that initially and I was getting around 80% packet loss.

Other Thoughts: The advertised speed on these is 200Mbps, which is 25MBps. Ethernet uses 5/4 encoding, which cuts the practical speed to 20MBps. But that's duplex and in most cases you'll only be transferring a file one direction at a time, so you really should expect to see 10MBps max. With me so far? If I have both units plugged into the same outlet, I get a file transfer rate of about 9.2MBps, pretty close to the theoretical max. With the units plugged into outlets on the opposite ends of my house, then I get 3.6MBps, which is still fairly fast. Pings were comparable to wireless pings, hovering around 1.5ms from computer to router across the house.



Saturday, March 16, 2013

How many people are on the Internet?

How many people are on the Internet? It seems like it should be an easy on to answer, but it's deceptively simple. However, attempting to answer it illustrates many of the problems in making supposedly simple measurements.

First, the world's population is a bit over 7 billion people, so this establishes a nice upper bound. No matter what, we know that the answer lays between zero and 7 billion.

Now, every device on the Internet has a unique IP address associated to it. It should be a simple matter to count up the number of addresses and get a better estimate. However, this is the first problem: IP addresses are unique to devices, not people. Two people sharing a computer counts as one IP address and one person with a cell phone, computer, tablet, and Internet-connected TV counts as four. However, lets ignore that problem for the moment.

Most IP addresses use what's called IPv4 and currently IPv4 addresses are "exhausted"; in other words, every address has been claimed by someone. IPv6 is the newest addressing system, but its usage is fairly small, so we'll ignore it. IPv4 addresses are a 32-bit number, so the number of addresses is 2^32 = 4,294,967,296 or approximately 4.3 billion. That means that there is an upper limit of 4.3 billion devices on the Internet.

Not all IPv4 addresses are in use due to inefficiencies in the distribution system. The first users of the Internet were given large blocks of addresses for them to dole out to their users, however many of these initial blocks were extremely large. For example, Stanford University was allocated 16.7 million addresses. That's a bit of overkill for a university with less than 20,000 people on campus. They later returned the unused addresses, but these sorts of inefficiencies remain at a smaller scale. For example, many ISPs maintain smaller blocks of unused addresses so that they can give them to new customers. With this in mind, the 4.3 billion device count was an overestimate.

However, Network Address Translation (NAT) can cause underestimates. NAT allows many devices to share one IP address. Most home users have one IP address for their Internet connection and every Internet-connected device (computer, laptop, tablet, cellphone, TV, Bluray, etc) shares that IP address using NAT. So, our 4.3 billion device count is also an underestimate.

So, the number of people on the Internet is somewhere around 4 billion. Possibly more due to people sharing devices and NAT. Possibly less due to people with multiple devices and unused IP addresses.

But what got me thinking about this was a comment from a video game publisher about the number of pirated copies of their latest game. If we can't even figure out reasonably well how many people are on the Internet, how can we possibly answer more sophisticated questions like how many pirated copies of a particular game are there?