It’s kind of amusing to see this back and forth about the open web and people suddenly being concerned about whether challenges to Google’s position of dominance are good for the “open web”. I do think it’s very possible that Google is entering a phase similar to that of Microsoft circa the late 90’s but, if so, we won’t really know for sure for another 10 years, and a lot can change. However, I do think we’re seeing the decline of the search-driven page-oriented web as it’s existing for the last decade. The reason is pretty simple and I’ll try to summarize it as best as I can, although I’m a long-winded kind of guy, so bear with me.
Now, if the enemies of the search-driven web are personalization (user-specific dynamic content) and client-side interactivity (highly programmatic user-driven content), then in the age of social and mobile, we can see that things are going to change in some significant ways. First of all, search engines are only really good at retrieving content that they have in their own indexes. The metacrawlers and federated search systems were only ever really good for specialized searches. For things like Google+ to work the way the user wants and to include Facebook and Twitter, Goggle would need to effectively copy the entire content set and the social graph and access control policies of any social content it wishes to use in it’s results. I’m sure some sort of escrow agreement could be negotiated between all the parties involved to allow this, and I’m sure such efforts have been made, but I could see it not being very simple, and it’s certainly not scalable and not very “open web”. Still, it wouldn’t surprise me if Google came up with some sort of standards or service for letting social apps upload their data to Google, so that Google could get the next Facebook or Twitter to already be feeding their social content into Google. If you think no one sane developer would trust Google with this, I’d counter that plenty of developers who compete with Google already trust Google with even more sensitive information such as their email.
More of a challenge is that applications have leapfrogged in interactivity to now becoming full client applications, most specifically in mobile. The content that the apps are accessing just isn’t being designed for web page use, it isn’t being rendered in a way that a search engine can use it in any useful way. Apps are powered by web services and databases full of small snippets of content tied together by relationships and queries that are changing based on any number of dynamic inputs, such as your location. The semantics are opaque to anyone but the programmer who builds them and the designer who crafts the experience for the user. Android has, unsurprisingly, a nice search mechanism for apps to be searchable, but this moves most of the search experience to the client, so I’m not sure that it benefits Google to the same degrees that web search does.
But, back to the point at hand, the issue (as always) comes down to vicious and virtuous circles. Social network optimization appears to deliver better ROI than search engine optimization. Facebook is optimized for social dynamic content and for apps and it generates proven results (traffic) for those. Twitter, to some extent, does as well. So, the effort that you could have put into mirroring all your dynamic content into static search SEO landing pages will be instead put into pushing status updates to Twitter and Facebook and using Twitter and Facebook’s APIs. For the user, Facebook increasingly becomes where you learn about apps and personally relevant content, rather than Google. For developers and publishers, the result is that Facebook delivers even more to you and Google delivers even less. Since Google and Facebook are essentially discovery marketplaces, you end up with both consumers and suppliers moving from one marketplace to the other.
Now, this really has nothing inherently to do with “open” versus “closed,” but it has everything to do with whether we can continue on a model where one or two companies deep mirror the entire web into a database and use it as the mechanism for discovery and whether we expect that the trend towards deeply personalized and social, highly-interactive and primarily client-side executed applications continues. The jury is still out on the former, but I’d certainly bet on the latter.