Entries tagged with search - MIX Online

Smartphone Carnage Far From Over

Joshua Allen — Mon, 21 Jul 2008 21:56:00 GMT

I can still remember meeting with people from RIMM 10 years ago, as they were a relative upstart compared to Palm, and were feeling out Microsoft’s intentions in the smart phone space. Since then, a lot has happened. Lots of new players and threats have emerged, and many have fizzled out after great hype.

All of the recent hype about iPhone 3G would make you think that the Smartphone market is Apple’s to win (from RIMM). Heck, much of the coverage makes it sound as if the iPhone is the first phone to have a popular developer ecosystem. CNET even went so far as to predict that the iPhone would replace Facebook and MySpace as the preferred social network targeted by widget vendors and advertisers! As in the past, the popular consensus is bound to look myopic in hindsight.

In this industry, there are 4 types of players who compete and cooperate:

· Carriers – companies like AT&T or Orange who sell you communications bandwidth for your phone. They own spectrum, and lease that spectrum to you. They want you to A) pay as much as possible for as long as possible, B) not cost them a lot in customer support. Carriers have the control, since spectrum is a scarce physical resource. This is why everyone else is interested in “network neutrality” legislation.

· Handset makers – these are companies like Nokia who make phones and want to get a profit on the hardware sale.

· Phone OS providers – companies like Microsoft who try to license their software to handset makers.

· Service providers – companies like Google who make profit when people use their service from a mobile phone.

I won’t take the risk of making predictions that could come back to bite me, but it’s safe to predict that the carnage is far from over. For the next 5 years at least, the following 5 companies will all be very relevant with significant share. For each one, I talk a bit about the business model and strategy to help understand how things might play out:

Google

Google Android aims to be an open-source operating system for Smartphones; free to all handset makers. Google like to say that they “do everything out in the open”, so it was fun to see them get caught being secretive. But it’s all a bit unfair, since Google is more open than Apple, Microsoft, or RIMM – and in fact that openness may be their downfall.

You see, Android is basically a strategic hedge by Google, to ensure that nobody can tie up the mobile platform with a closed, proprietary system. Google benefits when everyone develops using web standards and targeting Google services. Google’s business model is to profit on the services (like search) and use that profit to subsidize the phone user experience. This is a really strong strategy; and Google’s absolutely dominant ability to monetize their services will make them relevant on all mobile phones, regardless of how many Android units have shipped in 5 years.

Google’s weakness is that their strategy is fundamentally opposed to the strategy of the carriers. Phone Carriers want you to keep paying your bill, and to use data plans that are more profitable. They don’t want you calling them with an expensive support call about some 3^rd party application that they didn’t even write. Carriers may not mind open source, but they want a controlled developer ecosystem.

Symbian (Nokia)

Symbian is the dominant smartphone platform outside of the USA. The big news recently was that Nokia has purchased Symbian and will open-source the platform.

Unlike Google, who makes money from the services, the handset makers get their profit from selling the hardware. An open-source Symbian means that they don’t see a sustainable business model in licensing the handset OS. While open-source Symbian and Android are a blow to Microsoft’s Windows Mobile strategy (at a minimum, creating some pricing pressure), open-source Symbian is also blow to Google’s Android plans. The handset makers are wary of Google and want to keep their options open.

RIMM

RIMM owns a proprietary handset, operating system, and services. Because they control everything except the carrier, they can offer seamless end-to-end experience. This is why BlackBerries are so strong in the USA. RIMM knows how to work with carriers, too.

Apple

Apple is essentially copying RIMM’s strategy, controlling a proprietary handset, operating system, and services. Just like RIMM, they provide a seamless end-to-end experience. Just like RIMM, they make carriers happy by providing a sexy device that makes it easier for the carrier to sell expensive contracts.

Much has been made about how “innovative” the iPhone SDK and store is, but people apparently forget that Windows Mobile, Palm, and even RIMM have had additional applications available for a long time. The sort of applications, and the download trends, look a lot like other platforms. When people mention that Facebook is the #3 download from the App Store, they forget that Facebook released an app (web-based) for iPhone long before the SDK was released, and it was immensely popular. Windows Mobile recently got two Facebook apps, and installs of the Facebook app for BlackBerry still outnumber iPhone.

What Apple is doing differently is important, though. By centrally controlling the application store, they give an improved user experience. And more importantly, they provide a visible brand where people wanting support can call *instead* of calling the carrier. Apple’s app store will certainly increase expense for the carriers, but less so than the more open strategies of Google or Microsoft.

Apple business model is to profit up-front on the hardware, and break even on the services. They take a 30% cut of app store revenues and charge a subscription for mobile me, but their primary strategy is to profit on hardware. This gives them the free cash flow up front.

Microsoft

Microsoft’s business model historically has been to make money licensing our proprietary operating system. As a platform, we offer C++, .NET, or Silverlight, as well as AJAX.

As the entire company moves toward a software plus services strategy, our mobile strategy combines operating system with services. This is what the Danger acquisition was about, and it is no mistake that the Live Search app is one of the most popular applications for Windows Mobile.

Search is the Lever

Common wisdom says that this is about Apple and RIMM, but I think this is actually about Google and Microsoft. It’s true that Google hasn’t shipped a single unit yet, and Microsoft’s primary revenue stream (licensing the OS) seems threatened by open-source Symbian and Android. And neither company sells a sexy handset to drive cash flow-positive revenue.

But both companies control search engines, and search service is far more monetizable than any of the other services. The end-to-end experience using the WLS app on Windows Mobile is the sort of experience Google would love to have on all mobile handsets, and you can bet that they will. Apple getting $0.30 every time someone installs the “Flash Light” application is cool, but the revenues and margins of app store and iTunes store won’t be able to compete with search. Like iTunes and app store, the Mobile Me service is an attempt by Apple to protect their high hardware margins.

Search will be a critical component of RIMM or Apple experience anyway. Search is a really hard market to enter, and none of the other contenders will be able to afford the infrastructure necessary.

Of course, when anyone makes money, the carriers want to take a cut. So the carriers are the wildcard here. This is a fact that Google and Microsoft have known for a long time, and both companies will need to get better at making carriers’ lives easier to make inroads against Apple and RIMM.

BOSS Ain't Bad

Joshua Allen — Thu, 10 Jul 2008 20:02:00 GMT

Yahoo! just announced that they are opening up their search engine for people to re-rank and augment search results to create custom search engines. I saw an early version of this feature (named Yahoo! BOSS), and have been anticipating the release. Unsuprisingly, I think that some of the popular blogs are wrong about BOSS.

Contrary to what the popular blogs are saying, I don't think this move is as radical or revolutionary as Search Monkey. It's definitely cool, but Google, Yahoo!, and Microsoft have all had search APIs for a long time. The news here is not about a search API, but about the terms of use. The terms of use for BOSS are admittedly more generous than Google's, but not radically so. I previously explained why I think Search Monkey (a product which the popular blogs largely dismissed) is truly innovative. BOSS is a good addition, but the really creative parts are still just vapor.

The most perplexing coverage is both CNET and ZDNet somehow confusing BOSS with open source. Nothing could be further from the truth. Not only do you not get the source code to Yahoo!'s search algorithm, you are forbidden from trying to reverse-engineer it. It is true that Yahoo! contributes various technology to the open-source community, and a Yahoo! engineer recently won the TeraSort benchmark using the open-source platform upon which Yahoo! search runs. But the actual ranking and indexing algorithms are Yahoo! search's crown jewels, and they would be crazy to give those away. Perhaps people are just confused because "BOSS" ends in "OSS"?

Finally, a common theme in the blogs is that this is a "Hail Mary" play of desperation brought about in response to the recent drama with Microsoft. But I'm not convinced. BOSS is a lot less crazy than Search Monkey, and is really just an increment on what Google are already doing. The terms of use may be dangerously liberal, but they have plenty of room to learn and change. And it's not the kind of feature that someone can crank out in a few days. They have been working on this for awhile.

SearchMonkey is Disruptive

Joshua Allen — Tue, 24 Jun 2008 18:07:00 GMT

OK, after being initially skeptical, I’ve become convinced that Yahoo! SearchMonkey has the potential to really change the game in search. The evidence is mounting that they have really thought this through, and that they intend to disrupt the existing order. The plan is somewhat crazy, but this just might work.

When SearchMonkey launched, about 6 weeks ago, it seemed that the news was primarily lauded by proponents of RDF who believed Yahoo!’s endorsement of RDF would resurrect their beloved but anemic Semantic Web (with a big “S”) standard and give it legs to finally dethrone the small-“s” semantic web technologies like tags and microformats. To understand why they were so excited, you need to understand that it is the search engines who strangled RDF in the first place.

7 years ago (when Google was still a serious underdog), I argued that the search engines completely control the fate of “semantic web” standards, and explained that the major search engines have very little business incentive to support such standards. You can read the whole whitepaper, but the summary is simple:

1) Search engines are the dominant way that people query for entities on the web, and it’s nearly impossible to get authors to add the semantics and bootstrap the semantic web if search engines ignore the semantics or promote competing semantics.

2) When it is difficult to extract semantics from documents, it gives advantage to incumbents with massive scale data centers who can extract semantics from natural language. It creates barriers to entry for new competitors.

3) As a top search engine, you want the most useful semantic information stored in a format that your competitors cannot utilize.

For 7 years, my thesis held. At the time, I lobbied both Google and Microsoft to start indexing RDF (and later microformats). My hope was that their desire to disrupt the (then) dominant Yahoo! search position would lead to a more open web. But for 7 years, no search engine was crazy enough to truly adopt open standards for semantics. In fact, Google even dropped support for meta tag’s rudimentary semantics during that time period.

Then came SearchMonkey. For the reasons outlined above, indexing RDF and microformats is a pretty crazy underdog disruptive strategy, so I was skeptical. At first, my skepticism seemed to be justified:

1) At first, they supported only a handful of partners. See point #3 above.

2) The functionality was totally opt-in by consumers, and Yahoo! was doing nothing to evangelize it to average users. It looked like a silly PR stunt to curry favor with the RDF and microformats camps, and clearly Yahoo! was not putting any wood behind it.

3) Semantics can only be added by document owners, on their own subdomain. This immediately favors large incumbents. See the whitepaper for a description of why author-created metadata is a very weak form of semantics.

In the past 2 weeks, however, the first two reasons for my initial skepticism have been obliterated. The SearchMonkey gallery has expanded, and there are a number of interesting services already available. It appears that Yahoo! is promoting services which are not necessarily created by the site authors, which is huge. Check out the Wikipedia Topics entry, for example. And the PHP API entry is a perfect example of why opt-in by default was a good choice – I may want my search results to show PHP API entries, but most people do not. In addition, Yahoo! has started to promote the gallery from the home page of search.yahoo.com, under the customize button.

This isn’t a PR stunt. These guys are serious. Yahoo! took the single thing that drives publisher behavior (search engine exposure) and tied it squarely to open semantic standards.

Now, let’s contrast this with the Google approach. Google were the very first to offer “blended” search results, and much was made of the fact that Google Maps returns microformats on search results page. But spitting up microformats from your proprietary index is the opposite of consuming microformats to enrich your index. And the mechanism by which Google attaches semantics to the “plus box” is notoriously opaque. Watching people beg Matt Cutts for information, insinuate that blended results on SERP amounts to paid placement, or speculate about the algorithm as it changes under their feet (did Google “plus box” really just start scraping hCard?) makes you appreciate the way that Yahoo! does it out in the open.

Google did pay lip service to “out in the open”, when they launched Google Base to much fanfare and started integrating Google Base results into the main search results page. But Google Base still required publishers to store their content in Google’s servers, and the prominent listing on the search results page quickly became a distant memory and Google Base a black hole with little influence on the main search page.

I think people were a bit confused when Yahoo! claimed that SearchMonkey is a “long tail” strategy. But the discussion of Google’s contrasts should have made it clear by now that they are right. Yahoo!’s model of user opt-in makes room for both the default mass-appeal plugins (like Flickr) and the more niche plugins like PHP APIs.

Overall, this is very strong progress in just 6 weeks. To keep up the momentum, Yahoo! needs to continue promoting to end-users, and should be more aggressive about influencing search results ordering when SearchMonkey plugins are installed. For example, I have opted-in to the Yelp plugin, but perfectly good Yelp tresults often get pushed off of the page by CitySearch and others. Random samplings of users who haven’t tried any customizations should be shown enhanced search results pages and offered the chance to customize.

In addition, Yahoo! should allow SearchMonkey plugins to customize results for other pages. For example, I should be able to see the IMDB information next to a search result for a blog page that reviews a movie. This would truly bootstrap the use of microformats, since adding a microformat to your page would automatically make it more useful to anyone using Yahoo!’s search engine. Google tried something similar, with less than stellar results, when they started using scraped addresses from around the web to enhance their map “plus box”. When Google scraped restaurant addresses from old and outdated sites, the search results page “plus box” started directing diners to the wrong location, leaving restaurant owners bewildered as they tried to figure out where the wrong data was coming from. Yahoo!’s approach mitigates against this, since people opt-in to the provider, and they know where the data is coming from. In Google’s approach, you get whatever plugins Google gives you, and you have no idea where they are getting the data.

If Yahoo! share stabilizes or increases, I would expect Google to respond by being more aggressive with their “plus box”, and perhaps embracing and extending, with an eye to extinguishing SearchMonkey. SearchMonkey will encourage the greatest proliferation of microformats yet seen on the Internet, and as more microformats are available, Google will certainly start to leverage this information more in building their index.

A Data Center Built With Shipping Containers

Joshua Allen — Wed, 02 Apr 2008 20:06:00 GMT

Yesterday, Microsoft became the first in the industry to announce a data center designed entirely with shipping container-based server modules. It's a sizable data center, supporting somewhere between 150,000 an 440,000 servers. Mike Manos revealed some specifics about the Chicago data center at a talk at Data Center World.

James Hamilton has some commentary on his blog. James has been evangelizing the advantages of shipping container-based data centers for years, inside the company and within the industry. As he points out, several of the largest equioment companies are now providing containers, and you can be sure that we are not the only company buying them. But this is the first time anyone has talked publicly about a real industry-scale data center designed from the ground up for containers. I'm particularly interested by the fact that the containers use angle parking instead of being arranged at right angles.

This is a big accomplishment, and no doubt a sign of where the rest of the industry will be moving in coming years. Congratulations to Mike, James, and the rest of the team!

Greg Linden: Trends in Collective Intelligence and Centralization

Joshua Allen — Tue, 18 Mar 2008 22:02:00 GMT

Rich user experiences are going mainstream on a wider variety of devices and putting pressure on web standards. This trend is changing the face of the web as we know it, and we've covered this trend extensively here and at the MIX conferences. But there is another trend that is also changing the web forever, and Ray Ozzie's keynote at MIX08 was a shot across the bow regarding Microsoft's response to this seismic shift.

In his keynote, Ray talked about the "mesh", and the inexorable shift of services into large centralized data centers. We announced some cloud storage services, and discussed our philosophy of keeping control at the edges. This is really just the beginning, and we'll be having a deep conversation with the industry over the next year.

To kick off the conversation here on MIX Online, I asked Greg Linden to share his broad industry perspective about some of these topics. Greg led development of Amazon's ground-breaking recommender systems, created Findory, and recently joined Microsoft to work on some top-secret incubation projects. He continues to run the popular "Geeking with Greg" blog, where he riffs on large-scale centralized computing, data mining, and "collective intelligence".

Just a few of the topics we talked about:

Is MapReduce/Hadoop really as good as SQL?
What are the limits of social search?
What good is collective intelligence, anyway?
Will all of the world get sucked into one or two datacenters?