Discoverability and the Dynamic Web

Ed Anuff —  February 5, 2012 — Leave a comment

It’s kind of amusing to see this back and forth about the open web and people suddenly being concerned about whether challenges to Google’s position of dominance are good for the “open web”.  I do think it’s very possible that Google is entering a phase similar to that of Microsoft circa the late 90’s but, if so, we won’t really know for sure for another 10 years, and a lot can change.  However, I do think we’re seeing the decline of the search-driven page-oriented web as it’s existing for the last decade.  The reason is pretty simple and I’ll try to summarize it as best as I can, although I’m a long-winded kind of guy, so bear with me.


When dynamic websites first started to become the preferred method of publishing (via everything from high-end CMS to blogging systems), the challenge was how to make sure that your content was search-engine optimized.  This had an unfortunate side effect of killing one of the original reasons people started building dynamic websites, which was personalization.  If you wanted Google to find your pages, you needed to show the Google crawler pretty much the same pages as everyone who visits your page would see.  There are a lot of ways around this, but since the simplest approach is usually the best approach on the web, the result is that people just didn’t invest as much in dynamic personalization as they otherwise would.  The second casualty of this was client-side interactivity, and Javascript on web pages was relegated to small enhancements, because if you do everything dynamically in Javascript, Google is again not going to see it.

Now, if the enemies of the search-driven web are personalization (user-specific dynamic content) and client-side interactivity (highly programmatic user-driven content), then in the age of social and mobile, we can see that things are going to change in some significant ways.  First of all, search engines are only really good at retrieving content that they have in their own indexes.  The metacrawlers and federated search systems were only ever really good for specialized searches.  For things like Google+ to work the way the user wants and to include Facebook and Twitter, Goggle would need to effectively copy the entire content set and the social graph and access control policies of any social content it wishes to use in it’s results.  I’m sure some sort of escrow agreement could be negotiated between all the parties involved to allow this, and I’m sure such efforts have been made, but I could see it not being very simple, and it’s certainly not scalable and not very “open web”.  Still, it wouldn’t surprise me if Google came up with some sort of standards or service for letting social apps upload their data to Google, so that Google could get the next Facebook or Twitter to already be feeding their social content into Google.  If you think no one sane developer would trust Google with this, I’d counter that plenty of developers who compete with Google already trust Google with even more sensitive information such as their email.

More of a challenge is that applications have leapfrogged in interactivity to now becoming full client applications, most specifically in mobile.  The content that the apps are accessing just isn’t being designed for web page use, it isn’t being rendered in a way that a search engine can use it in any useful way.  Apps are powered by web services and databases full of small snippets of content tied together by relationships and queries that are changing based on any number of dynamic inputs, such as your location.  The semantics are opaque to anyone but the programmer who builds them and the designer who crafts the experience for the user.  Android has, unsurprisingly, a nice search mechanism for apps to be searchable, but this moves most of the search experience to the client, so I’m not sure that it benefits Google to the same degrees that web search does.

Small but relevant digression – I get a lot of people responding with “HTML5 is going to replace native apps, blah, blah, blah” when I talk about this.  This is mostly from non-programmers.  I’ve observed that the only technologists who really think HTML5 for mobile means anything to the “open web” are programmers in their 30’s, who can’t break out of a web page mindset.  Programmers who got started in the late 80’s and early 90’s building client-side apps have a very different perspective that they share with the new programmers in their 20’s, who are now doing client-side apps but for mobile rather than the desktop (but ironically using the exact same tools and languages we did in the 90’s).  There’s nothing “open web” (i.e. semantically rich) about an HTML5 app that contains a single script tag and then 20K lines of Javascript.  There’s nothing there for a search engine to crawl.  So, HTML5 and Javascript will very likely replace native code because they’re easier, but if you think that it’s a return to the SEO and link-driven world, and that app stores are going away, I’m not sure you’re necessarily envisioning HTML5 the way that the guys actually coding apps are going to use it.

But, back to the point at hand, the issue (as always) comes down to vicious and virtuous circles.  Social network optimization appears to deliver better ROI than search engine optimization.  Facebook is optimized for social dynamic content and for apps and it generates proven results (traffic) for those.  Twitter, to some extent, does as well.  So, the effort that you could have put into mirroring all your dynamic content into static search SEO landing pages will be instead put into pushing status updates to Twitter and Facebook and using Twitter and Facebook’s APIs.  For the user, Facebook increasingly becomes where you learn about apps and personally relevant content, rather than Google.  For developers and publishers, the result is that Facebook delivers even more to you and Google delivers even less.  Since Google and Facebook are essentially discovery marketplaces, you end up with both consumers and suppliers moving from one marketplace to the other.

Now, this really has nothing inherently to do with “open” versus “closed,” but it has everything to do with whether we can continue on a model where one or two companies deep mirror the entire web into a database and use it as the mechanism for discovery and whether we expect that the trend towards deeply personalized and social, highly-interactive and primarily client-side executed applications continues. The jury is still out on the former, but I’d certainly bet on the latter.

No Comments

Be the first to start the conversation.

Leave a Reply

Text formatting is available via select HTML. <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*