Adding Vellamo to our Mobile Benchmark Suite - Six Android Phones Testedby Brian Klug on November 16, 2011 10:43 AM EST
Real web browsing performance is a unique combination of system performance, the Android browser itself, and what contributions or customizations (if any) the OEM has made in the shipping browser build. This is to say nothing of the numerous third party browsers out there. Qualcomm's Innovation Center recently made public a tool for gauging overall browser performance that it has used for a while to checkpoint and gauge performance in house and in collaboration with OEMs. We've used a subtest or two from it in a few reviews already, and it's named Vellamo.
Left to right: T-Mobile SGS2, Motorola Atrix, AT&T SGS2, HTC Sensation, LG Optimus 3D, Xiaomi Mi-One
First off, Qualcomm sent us five dual-core smartphones, almost all of which we've seen and tested before. Just to be sure, we verified that nothing has been tampered with and that all were running the latest shipping OEM ROM and started fresh with a complete reset. We also added another, the T-Mobile SGS2 bringing that total to six devices.
|T-Mobile SGS2||Motorola Atrix||AT&T SGS2||HTC Sensation||LG Optimus 3D||Xiaomi Mi-One|
|SoC||1.5 GHz APQ8060||1.0 GHz Tegra 2||1.2 GHz Exynos 4210||1.2 GHz MSM8260||1.0 GHz OMAP4430||1.5 GHz MSM8260|
|RAM||1 GB LPDDR2||1 GB LPDDR2||1 GB LPDDR2||768 MB LPDDR2||512 MB LPDDR2||1 GB LPDDR2|
|OS||Android 2.3||Android 2.3||Android 2.3||Android 2.3||Android 2.2||Android 2.3|
The selection we've made isn't meant to be totally comprehensive, but gives an idea for just how much performance varies from one OEM to another, largely as a function of changes made to the browser on their respective versions of Android. These devices are all dual core and what I'd personally consider of the latest generation (dual core, Android 2.3, e.t.c.). We're going to be updating our Smartphone Bench category soon (something I've been behind on, as of late) with Vellamo scores for devices as well.
If you've been reading our reviews, basically every device except for the T-Mobile SGS2 (which we're going to review shortly) and the Xiaomi Mi-One should be relatively familiar. The Mi-One is from a Chinese OEM we haven't seen devices from before called Xiaomi, but is a surprisingly decent performer and has one of the most unique (and surprisngly speedy) skins I've ever seen on Android.
Vellamo Test Breakdown
First among these tests is See the Sun Canvas, which renders increasingly demanding stages of a 2D rotating butterfly in an HTML5 <canvas> element. The first stage starts out with cubes, then moves to circles, then circles of different color, and finally alpha-blended for the maximum load.
The Xiaomi Mi-One is a new device for us, but delivers strikingly fast performance thanks to the work that Xiaomi has done to the stock Android browser. I found the SGS2's backing store pretty decent as well, but you can see how much difference there is in performance between the two despite packing the same 1.5 GHz dual core snapdragon (APQ8060 and MSM8260 differ solely in baseband - APQ lacks one).
The next test is Deep Sea Canvas which tests almost the same parameters as See the Sun, except tests are non-sequential here and are an attempt to fully load the platform.
Interestingly enough, Exynos 4210 comes out on top of the pack.
Next up is Aquarium canvas which tests the number of HTML5 canvas objects that can be rendered simultaneously. In addition, this test runs in a native Android Java implementation for comparison with the HTML5 <canvas> implementation in the browser.
Vellamo calculates a score for each subtest alongside FPS and dropped frames, and we've chosen to report FPS wherever possible, but this test makes more sense being reported in base score. Interestingly enough the Sensation 4G edges out the faster T-Mobile SGS2.
Pixel Blender is next which draws multiple images over a static background and applies alpha blending. Vellamo calculates the graphics throughput of the browser in pixels per second to arrive at memory bandwidth for blitting and blending.
Again there's a pretty shocking disparity between Samsung's browser and the Xiaomi, we're reporting the base score here as well.
Where I feel Vellamo really shines however is this next suite, which we've shown scores from before, and this is the User Experience Test category. This is the area that until now has proven challenging to qualitate without using video or other subjective measures - things like scrolling performance and responsiveness.
I have to admit that Vellamo's numbers here closely mirror my own subjective scrolling performance perceptions when using the respective devices. The three scrolling tests represent a way to finally codify one of the more elusive (and yet easily perceptible) browser performance aspects.
The final test is a networking loader suite which loads a page stored locally and times how long it takes in milliseconds. This is done in both hot and cold mode (with the caches cleared and not cleared) and is designed to test local performance.
Finally Vellamo also reports a total score which is a summation of all the base scores from each subtest.
The total score subjectively lines up with where I would mentally rank the devices - we've seen how competent Samsung's browser optimizations are (especially their browser's backing store), and how much sometimes HTC's Sense tends to weigh the platform down. Motorola's browser on the Atrix and LG's on the Optimus 3D take a more stock approach, and the Xiaomi surprises us with some serious optimizations from a company I doubt many in the US have heard of.
We're going to be adding (and simultaneously giving the same treatment) a few more benchmarks in preparation for a revamped testing suite for 2012 and an updated smartphone Bench as well.
Post Your CommentPlease log in or sign up to comment.
View All Comments
B166ER - Friday, November 18, 2011 - linkMy Cyanogen'd (7.1b) Atrix running 2.3.7 did 835. What's that over stock?
juicytuna - Friday, November 18, 2011 - linkJust ran this on my Galaxy Nexus. Unfortunately it failed all three flinger tests but it did score 896 overall.
juicytuna - Friday, November 18, 2011 - linkSorry, that should have been a V8 total score of 1335.0
ChronoReverse - Friday, November 18, 2011 - linkMy HTC G2 is running CM7.1 with a patch that enables GPU acceleration in the web browser.
With the single core CPU overclocked to 1.5GHz, my score is 1090.
Jamezrp - Sunday, November 20, 2011 - linkI've been using Vellamo on a number of devices for testing, and at first I thought that it was great...but then I found the problem, which is similar to NenaMark (1&2), Neocore, and several other benchmarks: results are very much based on screen resolution. That is to say, if you have an Android phone with a low-res screen, you will have an inflated score.
Case in point: I have something like 7-9 handsets I'm testing right now, with screen resolutions varying from 320x480 to 720p. The low resolution LG Doubleplay manages to get an average of 810, while the Galaxy Player gets an average of 620, the HTC Rezound gets 920, and the Galaxy S II (well I had a score, but my frickin spreadsheet lost it's data while upgrading my GPU for some reason...)
As I continue testing, I'll add my findings, but it looks like the more I test the more relevant screen resolution is to the test, which inherently limits it. It may be well rounded, but the overall scores may not matter in the long run. Individual tests, however, may.
Jamezrp - Sunday, November 20, 2011 - linkI've gone over the testing here, and after some serious testing on my own, I have to conclude that Vellamo isn't a good benchmark at all. There's a number of reasons:
1) Results don't match. I've tested across six phones now, and even on the same device results don't match up. I get 80-point spreads, which is a joke. Why does the Rezound have a low score of 856, but a high of 930? Why does the benchmark run better when other applications are running, and worse when nothing else is running? Why does it take over an hour for the scores to pick up (ie, for the phone to "warm up")? The results, on several devices (though not on all) are not accurate and don't seem to level out.
3) Most of the tests are frankly pointless and don't show useful results. As someone who will use the benchmark scores to chart data, points are fine. But they don't really say anything besides what the points are. The tutorial that shows what each test is and why is great, but ultimately the data is what's important, and the data is flawed from the start. In an attempt to simplify it, the data just doesn't make any actual sense.
4) As I mentioned in a post earlier, scores rely heavily on display resolution, which is not good. If smartphone displays are going to vary as they do now - from 320x480 to 720p - then the scores aren't going to match well in any way. I have a few phones scoring higher than others solely because of this. The LG Doubleplay, for instance, a dumbphone by nearly all regards except that it runs Android 2.3.5, averages 200 points higher than the Samsung Galaxy Player in an average of 10 tests. What the hell.
I'll keep playing with it, but Vellamo thus far doesn't look like a benchmark worth using, at least not in the long run.
ickyboo - Saturday, February 4, 2012 - linkDon't forget it's also written by Qualcomm, So all scores will be biased towards their own chipsets.
sunwufan - Tuesday, November 22, 2011 - linkI am proud of using it now and hopefully they could get the 4.0 ROM ready soon.
branwell1 - Friday, December 23, 2011 - link嗨,大家好，我来自中国上海。一个星期前，我买了一部小米手机，仅仅花了我1999人民币(约合316.64美元)，用了一个星期，小米非常棒。希望想要这部手机的人能够尽快得到。
Hi! I am from Shanghai China.I bought a Xiaomi one week ago.It costs me ￥1999(about $316.64).I have used it for one week.It's really excellent！Hope you who wants one could get and enjoy Xiaomi as soon as possible. My e-mail is email@example.com can get more informatio from me.
branwell1 - Friday, December 23, 2011 - linkHi! I am from Shanghai China.I bought a Xiaomi one week ago.It costs me ￥1999(about $316.64).I have used it for one week.It's really excellent！Hope you who wants one could get and enjoy Xiaomi as soon as possible. My e-mail is firstname.lastname@example.org. You can get more informatio from me.