(2004-04-15) Vogels Rss Scalability

Werner Vogels [is](http://weblogs.cs.cornell.edu/All Things Distributed/archives/000436.html) investigating the Scalability issues associated with RssAggregator-s as a Model for Distributed Computing. The current situation with the explosive growth of the use of desktop based aggregators is similar to the scalability problems that many other distributed systems and applications have faced. These systems were forced to restructure to be able to maintain scalability from functionality and/or efficiency point of view. It is likely that the publish/subscribe system that the feed/aggregator system will need to address the same type of issues in the near future... Aggregators will need to find ways in which the users can be subscribed to a select set of feeds because they want to read everything that comes from these feeds, but also subscribe to a much larger set of publishers for which the feed abstraction may not be the right metaphor. Aggregation, fusion and selection at the information item level instead of at the feed level seems to be a first abstractions to investigation... XPath and XQuery are not the answer, 3D graphical representations of related information units are more likely to be the ways to approach this. And how the applications can learn and adapt to shifting interests and new information sources will be equally important in the scalable handling of the continuous stream of new feeds becoming available.... The polling for data was an approach that was also used many older distributed systems and has shown to be something that can bring a system to a halt if the number of subscribers increases. There is a range of approaches that can help with this problem ranging from combinations of push & pull, asynchronous notification mechanisms, feed hosting in decentralized clouds, feed and information brokers, intermediary feed aggregators, collaborative delivery of feed content, optimizations in feed transfer, etc. Most of these potential solutions will be the subject of the scenarios once when we are going through the example cases. (Publish And Subscribe)

Upate: [links](http://weblogs.cs.cornell.edu/All Things Distributed/archives/000448.html) to a number of his individual pieces.

Update: Mark Nottingham is less freaked out about it.