iOS 5 update tested

Once again, I used GeekBench 2 ($0.99), Gauge Mathematical Tool (free), BenchTest ($0.99), SunSpider (free) and V8 (free) to test the performance of each OS version on each device. I tested the original iPad 1 Wi-Fi, the iPhone 4 (GSM), the iPod touch 4th Gen, and the iPad 2 Wi-Fi+3G (GSM).

Conclusions

If you use the web more than anything else on your iOS device, then you should be eager to apply each and every update that Apple puts out, since they all improve Safari’s performance every time. iOS 5 is no exception in that regard.

Considering all of the new features you get with iOS 5.0, the hit on performance in other areas is in most cases negligible.  In fact, with the exception of the original iPad (the older device that was tested), all devices actually gained in performance in most areas. That is simply amazing for the first release of a major OS update. This is nothing like the 4.0 release, where we saw incremental performance gains over the lifetime of 4.0 to restore what was lost to older device; iOS 5.0 is a solid release for all where performance is concerned.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Tablet market to hit over 377 million units by 2016
• 2012: Data, spectrum and the race to LTE
• Connected world: the consumer technology revolution

iOS 4.3 Update Tested

The only concern I had was that the benchmark apps were all iPhone apps, not true iPad apps, but it shouldn’t affect the overall results.  I used GeekBench 2 ($0.99), Gauge Mathematical Tool (free), BenchTest ($0.99), SunSpider (free) and V8 (free) to test the performance of each OS version on the iPad. I performed all the tests on the same device, both before and after the upgrade.

Results for the Wi-Fi iPad

For the most part, the test results showed that performance was slightly better on iOS 4.2.1 than on iOS 4.3 on the iPad.  But overall, the results were close enough to consider the upgrade to iOS 4.3 on the iPad simply worthwhile.  That is, until you look at the JavaScript test results. Mobile Safari on the iPad was significantly faster on iOS 4.3.  The only problem I ran into was that the Google V8 JavaScript tests would not complete on the iPad.  The test would either hang or crash mobile Safari.  I had a similar issue on the iPhone 3G when trying to execute the same tests.

Results for the iPhone 4

While I couldn’t upgrade my iPhone 3G, I could apply the latest update to my iPhone 4 for comparison to prior results and see how it was affected by by the upgrade.  Unlike on the iPad, I’ve been maintaining benchmark results on my iPhone 4 for some time now, so I have some historical data to compare the latest tests to:

Over time, you can see where the performance of iOS on the iPhone 4 has slowly degraded with each release. 4.3 is no exception to this trend.  Keep in mind that this performance degradation is not nearly as dramatic as that which occurred with the iPhone 3G.  Note also the continual increase in performance noted in the JavaScript performance with each release.  The 4.3 update is the most significant yet in that regard on the iPhone 4.

Conclusions

For what you get in the form of features withe the iOS 4.3 update, and for the almost insignificant general performance hit on the iPad, the upgrade is overall a great one.  Any time you get more features for little or no degradation in performance is a good thing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Report: A Mobile Video Market Overview
• In Q3, the Tablet and 4G Were the Big Stories
• Mobile Operators’ Strategies for Connected Devices

CNET ran the iPad 2 through the SunSpider benchmarking tool, which measures Javascript performance. It found that the iPad 2 had the best overall scores of devices it tested, with a score of 2097. The original iPad running iOS 4.2 scored 8321, and lower scores are better when testing with SunSpider. The iPhone 4 on iOS 4.2 scored 10414, and the Nexus S and Galaxy Tab had scores of 6128 and 7066 respectively.

In addition to the iPad 2′s dual-core A5 processor, a lot of the increase in performance can be attributed to changes made to the iPad’s operating system in iOS 4.3. CNET also ran the SunSpider test on an original iPad and iPhone 4 running iOS 4.3′s latest beta, and those numbers showed huge improvements. The original iPad on iOS 4.3 scored 3261, and the iPhone 4 scored 4151. That’s great news for users who aren’t planning to upgrade, since you should experience much snappier browsing even with just a software update.

Android fans shouldn’t feel too badly about the poor performance of the Galaxy Tab and Nexus S, since SunSpider tests performed using the upcoming Galaxy Tab 10.1 found that it scored 2256, which is nearly on par with the iPad 2.

Bottom line, Apple mobile device users are about to get yet another improvement to the overall iOS experience, with or without the hefty price tag of buying new hardware.

Related content from GigaOM Pro (sub req’d):

• Why iPad 2 Will Lead Consumers Into the Post-PC Era
• App Developers: Are You Ready for HTML5 and Metered Data?
• How to Market Your iPhone App: A Developer’s Guide

Geekbench 2, the latest version of Primate’s testing software, only measures processor and memory performance, and doesn’t take into account video card performance. Results are drawn from the Geekbench user community, and higher scores are better. Partial results are shown below; click on the image to see the full results.

As you can see in the full results, the new MacBook Pros outperform the previous models by a wide margin, which is to be expected when jumping from a dual- to a quad-core configuration. Note that even the 13-inch MacBook Pro outperforms the previous 15- and 17-inch Pro powerhouses released in early 2010.

The new notebooks even outperform some current Mac Pros, according to Primate’s general Mac benchmarks: definitely an impressive display of raw muscle from a portable computer.

That said, the ultimate test of any computing device is whether or not it meets your actual usage needs. Accordingly, I’m still inclined to wait until the next MacBook update rolls around. Do these performance scores influence your decision one way or another?

Related content from GigaOM Pro (sub req’d):

• Transient Apps: The Consumer Influence on Enterprise Mobility, Part 2
• Why Apple Hasn’t Sewn Up the Tablet Market — Yet
• Rogue Devices: The Consumer Influence on Enterprise Mobility, Part 1

iOS 4.2.1 Update Tested

To test how iOS 4.2 affected performance, I took an iPhone 3G and ran it through the same series of performance tests I did when iOS 4.1 was released. Just like last time, I used GeekBench 2 ($1.99), Gauge Mathematical Tool ($1.99), BenchTest (99 cents), SunSpider (free) and V8 (free) to test the performance of each OS version on the 3G.

Also like last time, the 3G I was testing was still running with a lot of features disabled, and was being used more as an iPod touch than as an iPhone. I am reusing the same data collected last time for comparison. The following results are only a subset of the data collected, and focus on the data points that changed the most from version to version:

Results for iPhone 3G: The 4.2.1 Update

While some things seem to have improved slightly, like JavaScript and drawing new content when you switch views in an app, other tests have dropped off considerably. Memory performance as measured by GeekBench, as well as the executions of Gague’s Prime Number test are especially bad. These would affect apps that are doing much more than just displaying or editing a simple screen, like an RSS reader, for instance.  It would have a greater impact on apps performing more advanced computational functions, such as Documents to Go.

Just informally trying out several core apps like Apple’s Notes, Safari, Calendar and Contacts revealed more than the benchmarking tests indicate, however. Scrolling speed and data entry appear to be much improved. It seemed as though my performance tests were contradicting my real world experience.

One More iPhone 4 Test: The 4.2.1 Update

Testing on the iPhone 4 was the best way to find out if the performance degradation I was seeing on the iPhone 3G in the tests was really something to worry about. I reused the same benchmarking data collected in earlier tests for comparison. I was looking for the same degradation in performance in the same areas.

Since both the GeekBench Stdlib Allocate test as well as the Gague Prime Number test degrade on the iPhone 4 as they did on the iPhone 3G, I had to conclude that it was an issue with how the test was seeing changes made to the iOS firmware in version 4.2.1, and not necessarily a reflection of actual device performance.

Conclusions

While data entry, scrolling, pinching and zooming have improved slightly, the 3G is still not up to par with its newer siblings. Where we see the most gains on both the iPhone 3G and the iPhone 4 is in the JavaScript results.  Whatever the Webkit/Safari team is doing, it is having a positive effect with each release.

Bottom line: If you’re still using iOS 3.1.3 and use sophisticated apps that are heavily computational, then it’s probably best to avoid 4.2 (that’s easier if the apps you’re using don’t require 4.0 or higher). If you’re content with the apps Apple provides and mobile Safari, you should be fine. If you need the more demanding apps, and those apps need iOS 4.0 or higher, it might be time to consider that hardware upgrade.

Online Benchmark Results

• GeekBench 2 – iPhone 3G running iOS 4.2.1
• SunSpider – iPhone 3G running iOS 4.2.1

Related content from GigaOM Pro (sub req’d):

• App Developers: Are You Ready for HTML5 and Metered Data?
• HTML5’s a Game-Changer for Web Apps
• How to Market Your iPhone App: A Developer’s Guide

Thinner, yes. But better?

I am one of the few, proud MacBook Air (MBA) owners in the world. Every article you’ve seen from me in the past was written on the petite beauty. In one last month, I wrote:

I’m only here to make a case that anyone that hasn’t used one should give it a shot and that Apple needs to throw some marketing dollars behind it so everyone else knows what we MacBook Air owners have known for a long time: It’s the best notebook Apple makes and it also happens to be the smallest.

This is exactly what Apple did yesterday, and yet, I was severely disappointed. It seems as if Apple is finally throwing marketing dollars behind the MBA, and I watched my Twitter stream as thousands of geeks’ heads exploded at how thin it is. But I felt a little weird reading this on my MacBook Air and thinking out loud, “I know. It’s been that way for years.” It’s thinner, yes, but not significantly. That’s the whole story in a nutshell.

Pricing & Marketing

On the whole, the reason Apple’s MacBook Air wasn’t a hit before now was price. Imagine the iPad started at $999. Fewer people would own one, and those that did and sang its praises it to friends would be immediately be greeted with the response that it was simply too expensive for a tablet. Basically, Apple just dropped the price. The MacBook Air of today is largely unchanged from what we had before, and that’s disappointing.

11.6″ Model Versus the Sony TZ (circa 2008)

Consider Engadget’s live-blog from Macworld 2008, when the Macbook Air was introduced. Steve initially compared the MacBook Air to Sony’s TZ series. His biggest gripe was that the processor speed maxed out at 1.2GHz Core 2 Duo, and the display was only 11 inches.

The 11.6-inch MacBook Air is “too cramped” and “too slow” according to the Steve of 2008. To be fair, the Air isn’t the same as the TZ in every way. I haven’t used the new keyboard, but Apple claims it’s full-size and the body is still a bit thinner.

Let’s Talk Speed

Current MacBook Air owners won’t see a significant speed bump if they own the Revision C model (1.8/2.13GHz) released last year. In fact, not much has changed about the machine at all.

MacBook Air Revision C (Jun. 2009)

• 1.86 ($1499) or 2.13GHz ($1799)
• 3.0 Pounds
• 6MB of L2 Cache w/ 1066Mhz Frontside Bus
• 120GB HDD or 128GB SSD
• 2GB RAM (DDR3-8500 at 1066Mhz)
• 13.3-inch screen w/ 1280×800 resolution
• NVIDIA GeForce 9400 w/ 256Mb of VRAM (shared with main memory)

MacBook Air Revision D (October,2010)

• 1.86Ghz ($1,299) or 2.13Ghz ($1699)
• 2.9 pounds
• 6MB of L2 Cache w/ 1066Mhz Frontside Bus
• 128GB SSD or 256GB SSD
• 2GB/4GB RAM (DDR3-8500 at 1066MHz)
• 13.13″ screen w/ 1440×900 resolution
• NVIDIA GeForce 320m w/ 256Mb of VRAM (shared with main memory)

Of course, I’m only comparing the 13-inch models, but you’re getting a machine that’s basically the same speed with an option to add more ram and more storage,  plus a high-resolution screen which isn’t something I’d recommend to everyone. Some (especially those over 50) will prefer a lower resolution. Even I had trouble using a 17-inch MacBook Pro and eventually sold it.

What you are getting is a drop in price. Last year, a 2.13GHz machine with 2GB of RAM and a 128GB SSD would cost $1799. Today, that same machine costs $1699 with double the storage, and it’s only $100 to upgrade to 4GB of RAM.

Real World Performance

Will HD Flash movies on YouTube perform better? No. Will Photoshop CS5 filters apply faster? Maybe with a RAM upgrade. Nothing about this upgrade is impressive if you’re a current MacBook Air owner looking for more speed.

Why can’t Apple squeeze more juice out of this machine? Apple didn’t talk speed at all. They didn’t throw their famous “2x Faster” graphic up on the “buy now” page. Apple does tout the 2.9x faster graphics card, but I can guarantee you the Air released yesterday is only marginally faster than last year’s model, despite the introduction of the 320M, because that’s a video card using shared RAM, meaning the graphics performance goes down as you open more applications.

I was underwhelmed by today’s announcement, but the MBA was already a good notebook. I was hoping Apple would make it even better. It didn’t. “One more thing” was simply a minor evolutionary product refresh, and that’s too bad.

Related content from GigaOM Pro (sub req’d):

• Mobile Operators’ Strategies for Connected Devicess
• Strategies for the Future of Digital Content Storage
• Company Profile: Apple

So has the recently released 4.1 update finally closed the door on the issue of iOS 4 running sluggishly on the iPhone 3G?

iOS 4.1 Update Tested

To test how the iOS 4.1 affected performance, I took a pair of iPhone 3Gs and ran them through a series of performance tests. I used GeekBench 2 ($1.99), Gauge Mathematical Tool ($1.99), BenchTest (99 cents), SunSpider (free) and V8 (free) to test the performance of each OS version on the 3G.

I started out with one iPhone 3G running 3.1.3, and the second running 4.0.2. I must admit that both 3Gs were dumbed down quite a bit, and were not being utilized as iPhones, but more like iPod touches. The iPhones used were purchased at the same time, shipped in the same shipment, and have very similar Serial, IMEI, Wi-Fi and Bluetooth numbers assigned to them. The following results are only a subset of the data collected, and focus on the data points that changed the most from version to version:

Results for iPhone 3G: 3.1.3 vs 4.0.2

It’s not what the primary results did say, it’s what they did not say. As I executed the tests side by side, I noticed that even when similar end results were returned by the benchmark tests, the UI of each iPhone would behave differently. The 3G running 3.1.3 would behave as expected, but the 3G running 4.0.2 did not. Progress bars wouldn’t update smoothly on 4.0.2, but would jump to 100 percent at the end of the test. As tests were executing, it was as if the user interface on 4.0.2 was somehow sporadically frozen during the tests. This behavior is consistent with other video reports online that iOS 4 is slow on 3G devices. It also explains why certain activities, namely games that don’t utilize Apple’s stock UI components, run pretty much the same on each OS version. If you now use your 3G primarily as a game device as I do, you may not have noticed all the differences in performance.

If, on the other hand, you’ve continued to use the device as a smart phone, you’ve most likely noticed the degradation in performance. What was surprising at first was that the JavaScript results were much faster on iOS 4. This makes some sense, since the version of Safari on iOS 4 most likely has the latest JavaScript engine under the hood. So while some UI elements may be slower, and most hardware-based benchmarks remained relatively constant, the JavaScript results actually improved from 3.1.3 to 4.0.2.

Results for iPhone 3G: The 4.1 Update

I then upgraded the 3G running iOS 4.0.2 to the new iOS 4.1. At this point, I had one iPhone 3G running iOS 3.1.3, and the other running iOS 4.1. This time around, the UI did appear to behave more as one would expect on 4.1, with smoother progress bars and a zippier response from the UI. It felt faster, but was it as fast as 3.1.3? No, not really.

While certain numbers did seem to bounce back, namely the performance of the  Stdlib Allocate test within the GeekBench test suite, most numbers remained the same as in 4.0.2. The BenchTest results for Drawing Into View and File to Filesystem did seem to bounce back a little as well. It was only the JavaScript results (as per the SunSpider test) that really seemed to improve even further. However, it’s important to note that the iPhone 3G (on both versions of the OS) never passed the V8 test. Safari always crashed before completing the test.

One More Test: iPhone 4 on 4.0.2 vs. 4.1

So before drawing any conclusions, I wanted to see if the subtle performance increases noticed between 4.0.2 and 4.1 on the 3G were the same performance increases noticed on the new iPhone 4. Finding the same performance increases on the iPhone 4 as compared to the iPhone 3G would point to an overall performance boost based on OS updates that weren’t specific to any one hardware platform. Luckily, I happen to have two iPhone 4s in the house as well. This time, I performed benchmarks on the exact same hardware before and after the upgrade. I was very surprised with the results of the performance tests on 4.0.2 compared to 4.1 on the iPhone 4.

Conclusions

It’s still very likely that Apple did focus on performance issues related to each one of its own internal apps that are distributed only via OS updates. Apart from the JavaScript benchmark results — which prove that Safari on iOS 4 outperforms Safari on iPhone OS 3 — the boost to overall performance based solely on the test results listed above is marginal.

These tests, however, are looking at the OS itself, not individual app performance improvements. Surprisingly, there does appear to be a platform-specific focus on performance issues related to the iPhone 3G, as performance does appear to have degraded on the iPhone 4 with the 4.1 update. I’d been focusing heavily on the iPhone 3G, and had run several iterations of tests comparing 3.1.3 to 4.0.2, and was prepared to collect the same amount of data following the 4.1 update on the iPhone 3G. I was much more meticulous with the details on executing the tests.

The tests against the iPhone 4 were an afterthought, and would require a more through examination before claiming that the iOS 4.1 update did indeed degrade the performance of the iPhone 4. Just as we struggled through the iPhone OS 3 updates on the iPhone 3G last year, there will likely be more updates to iOS 4 in the months to come. While I still don’t feel iOS 4.1 is back to the same performance levels of iPhone OS 3.1.3, there do appear to be performance gains in 4.1 that prove that Apple is serious about supporting the iPhone 3G on the iOS 4 platform.

Online BenchMark Results

• GeekBench 2 – iPhone 3G running iPhone OS 3.1.3
• GeekBench 2 – iPhone 3G running iOS 4.0.2
• GeekBench 2 – iPhone 3G running iOS 4.1
• SunSpider – iPhone 3G running iPhone OS 3.1.3
• SunSpider – iPhone 3G running iOS 4.0.2
• SunSpider – iPhone 3G running iOS 4.1

Next Gen Customers on Previous Gen Devices

The most likely recipients of older iPhone 3G devices will be the siblings of the owners of the devices which have replaced them with the new iPhone 4. This is a very important class of user that Apple cares a lot about. The new owners of these older iPhone 3G devices will be potential Apple customers throughout their academic life. And it appears that students are serious about Apple. Providing a less than awesome experience now could jeopardize their loyalties later.

One iOS to Rule Them All

The current burden on developers targeting Apple’s mobile devices is that there are too many software versions to manage; 3.1, 3.2, and 4.0, to name a few. The number of devices on each of these three major platforms is still significant. With the upcoming iOS 4.1 update, Apple has the opportunity to converge more of these devices to just one OS platform. That is, if they can convince iPhone 3G owners that performance is no longer an issue.

Focusing on multiple platforms does take time away from adding features and releasing new titles. Giving developers the tools they need is only part of the equation. Switching between simulators and managing several different devices takes time. Cutting down on the number of device and OS combinations that developers need to support will only strengthen the quality of the apps for devs that don’t test across the matrix of combinations, and give back some development time to those that do.

Engineering Pride

Apple has been quite public about the engineering capabilities of other software development shops. Apple was also adamant all along that iPhone 3G devices would be supported on iOS 4, minus some features. Nonetheless, Apple proclaimed support for these devices. Taking on its own challenge, stepping up to the plate and delivering an awesome experience to its next generation of customers is what Apple’s core values are all about.

Related GigaOM Pro Research: Rogue Devices: The Consumer Influence on Enterprise Mobility, Part 1

The architecture of the new iPhone 4 places the antenna structure within the stainless steel band that wraps the new phone. And the problem many users have noted is that applying normal pressure along specific points of the phone causes the signal quality indicator to drop. Seems like just a hardware issue, right? Well before you go old-school and wrap your phone in aluminum foil in a desperate attempt to boost the signal strength, there’s more to the story.

The issue of signal strength has been picked apart across the Internet; some people can reproduce the problem, while others cannot. A few days after these issues first came to light, Steve Jobs said in response to an email:

Gripping any phone will result in some attenuation of its antenna performance, with certain places being worse than others depending on the placement of the antennas. This is a fact of life for every wireless phone. If you ever experience this on your iPhone 4, avoid gripping it in the lower left corner in a way that covers both sides of the black strip in the metal band, or simply use one of many available cases.

It’s an interesting response, but it seems to speak to Apple’s industrial design decisions, since such sensitive hardware is in a region of the phone that’s naturally likely to be handled. Regardless, since Jobs’ comments, there have been daily rumors concerning Apple’s solution to the issue, whether it’s a software update to “fix” the problem or providing customers with an iPhone 4 bumper case that avoids placing pressure on the antenna. Most recently, fabricated emails have come to light suggesting that Jobs told a customer “calm down” and that it’s “not worth it.” While that turned out to be a fake email according to Apple, the company finally released a statement regarding the issue altogether.

According to the statement, the algorithm for calculating the signal strength has just been inaccurate, showing more bars than it should in some cases. In an example provided by Apple, what is displayed on your iPhone could be two bars higher than the actual signal strength. The statement also suggests your real signal strength never changes, so when you see the lower signal strength as a result of placing pressure along one of the antennas, you’re really seeing a more accurate representation of the signal.

Okay, so it’s a software issue. Or is it?

Apple says this problem of inaccurate signal strength has been present in every iPhone since launch. (Thanks, Apple!)

So if it’s an issue of inaccurate signal strength, how does that affect performance? Look at the video that Cameron Hunt posted to Vimeo that shows how Safari simply stops loading when he touches one of these antenna points along the device. If you apply Apple’s logic to his scenario, when you watch the bars begin to drop, his actual signal strength shouldn’t be dropping. Yet it does, because Safari cannot finish downloading the page. Clearly, there’s still some degree of a hardware issue involved.

Apple says a free software update for iPhone 4, iPhone 3G and 3GS users will be available in the next few weeks to address the problem, and will cause your iPhone to display a more accurate signal strength. Additionally, the update will make the first three signal bars “a bit taller” and “easier to see.” Just remember, they’re only taller to make them more visible, not because the signal strength is any better.

For the technically minded out there, it’s been mentioned that the field test mode in iPhone 4 has disappeared. That’s too bad, as it would have been a great way to see what’s really happening to the signal strength. Does anybody know how to access it on the new iPhone 4?

Do you really believe Apple’s response that the issue is software-related and the reality is that the network reception is actually much lower than what your iPhone displays? Is there really anything wrong? Or is nothing wrong? Or is there something wrong but the problem is just normal of cell phones and it’s just time for us Apple users to drink the Kool-Aid again? Share your thoughts in the comments.

Before the holidays, Apple put its foot down and delayed shipment on all new larger iMac orders until after Christmas. The ordering page for the 27-inch machine changed today to three weeks, extending delays further still. Maybe it’s just a backlog resulting from the initial delays, but reports of problems don’t seem to be slowing down, so that doesn’t seem likely.

Let’s review the storied past of the iMac that held so much promise when it was announced. Very shortly after that announcement, as soon as the first units started shipping, reports came in from users about problems with Flash performance, hard drive spindowns, bad Snow Leopard installs and permissions issues. So it tripped a bit out of the gate, but this is a distance race, not a sprint, right?

But as the race continues, the iMac in question doesn’t seem to be improving its performance. The display seemed to be the primary cause of concern as time went on. People reported cracked displays on delivery, screen flicker that drove a whopping number of customers crazy, and visual noise and dead pixel issues. The latest and most talked about complaint is significant screen yellowing and gradient change from top to bottom of displays on the 27-inch model.

The screen isn’t the only problem, though. Out of the box, some users are reporting DOA machines that just would not boot, and more complaints than any other about processor performance. A friend of mine had to return theirs twice for the same issue, and still hasn’t received a problem-free unit. Apple has tried software fixes, too, but they haven’t worked for everything.

The question isn’t really whether or not the 27-inch iMac is a lemon, it’s how Apple let this happen to begin with. Cupertino is not known for a tendency to rush machines into production before they’re ready, but this has all the earmarks of that exact situation.

Perhaps Apple was distracted by the tablet it’s apparently been developing, and if so, I’m not optimistic about the future of the Mac line. With the iPhone and a brand new similar platform in the tablet, will QA suffer in other areas? Apple’s reputation depends on two things: innovation and reliability. I, for one, am not willing to sacrifice one for the other. Are you?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why iPad 2 Will Lead Consumers Into the Post-PC Era
• The Near-Term Evolution of Social Commerce
• Content Farms: The Players, The Benefits, The Risks