Communication Architecture

Hi folks,

I guess this is a semi-technical, semi-business question, so I wasn't sure where to post it.

I'm curious, generally, what type of architecture is supporting sites like Expedia.com or moneysupermarket.com.

When a user searches the site they get a list of hotels which have empty rooms, or a comparison of car insurance quotes, basically whatever the user searches for. But how is this information kept up to date? For each search is the information pulled from each vendor, using a web service? Or would vendors push the information to central website at certain intervals and the search is essentially conducted locally?

Now, presumably a vendor would have to integrate this into their own system, especially with a pull model, as they would essentially be providing their own web service that conforms to your own API. But this would require some investment by them, and, from a business perspective makes it difficult to sell.

Basically I'm curious how these sites got off the ground. Did they provide some 'black box' software that was easy for vendors to plug-in an communicate with their own site?

Hi Pleco,

You've asked a pretty complex question. There are really three major questions you're asking: platform architecture, searching, and data synchronization. Let me briefly attack each in turn, but I'll try not to overwelm you with too much detail.

In terms of platform architectures, to support these sites, most use a standard 3-tier design. This does have scalability problems, though, and you can get a hint at eBay's difficulties in this presentation. You'll see that they had to partition their system not by application domain, but by technical aspects. The move to SOA platforms (using a message bus and stateless processing servers), you can split your load by domains (such as travel, entertainment, etc). This provides a better level of abstraction and allows domain-driven design.

Regarding search, there are multiple aspects. For airline bookings, this is done through the GDS systems. These are quite old and mainframe based, so integration is done by automated systems pretending to be agents working directly on terminals. For other domains, such as hotels and dining, I believe the data is loaded into the system, custom search code is written, and reservations are made through a webservice call to a vender. To be honest, though, I'm not entirely sure on this aspect as I work on the core platform of such serviceand only rarely have to jump into the application code.

On the final question, data synchronization, the strategy depends on the situation. Most companies still use ftp drop-offs. This isn't bad, actually, since we're talking about huge amounts of data to cover all restuarants, event tickets, etc neeting to be loaded weekly, if not daily. This is then loaded into a data warehouse using an ETL process (I wrote our company's ETL framework). For smaller amounts of synchronization, webservices are definately used in both push and pull models. This is useful for more granular aspects, such as a customer's profile, where details need to be constantly updated but are quite small in overall data. Other times you might use a webservice to register a user (such as for flight status), and then be pushed updates to the flight. It all depends. Also, whoever owns the data (such as profiles, restuarants, etc) or service (bookings) also owns the format. Everyone else must provide adapters, so integration is everyone's responcibility, depending on the situation.

I'm sorry if I didn't give you a very coherent answer. Its pretty big, and integration is really the next big thing. I can try to answer your questions better if they are more targetted, but my expertise is on the core infrustructure. So I can explain how I designed our ETL process, or how to scale, and can at best ping colleagues who work on the particular stores such as travel or dining. But as you can imagine, the more vague your question the longer the answer!

Thanks for the reply NovaX, you have been most helpful.



Ah, this is assuming that the vendor already offers some kind of service? In the UK the AA(car insurance) website lets the user compare quotes with rival insurance companies so presumably those other insurance vendors already offer some kind of 'request quote' service which the AA can freely hook into?

But what about a third-party comparision web site, would the site would define a web serivce specification, and any vendor who wants have their information on it would simply be expected write their own code to push data into the site?

One of the things I'm trying to understand is how vendors actually get involved, particulary small vendors. In a comparion site naturally it's benefical for the site to have as many vendors signed up as possible, and I imagine a vendor would have a business interest in having their data on the site.

But what if the vendor involved doesn't have the infrastructure or expertise in place to connect to your web service. For instance, say a small hotel wanted to sign-up to a hotel comparsion and booking site, but they only have a simple website with maybe the data loaded from a spreadsheet, or even written directly into the html with frontpage - simply offering a web service specification to these folks wouldn't do them much good. What's typical in this situation, are they simply left out?