Feedster, the news and commentary search engine, currently indexes over 670,000 feeds. Scott Rafer joined the company as President and CEO in October, 2003. Before that, he co-founded WiFinder, a directory of public access Wi-Fi, and Fresher Information, a provider of content indexing software. His personal blog is License-Exempt Soweto. A few weeks ago, we sat down at a cafe in San Francisco and talked about Feedster and the future of RSS. Here's the conclusion of our conversation: 

Harold Check: Of the big companies out there now?Yahoo!, Google, AOL, Microsoft, Amazon, eBay?which do you think really gets RSS, on a corporate level? Which will be the first to make a move as a company?

Scott Rafer: The first group that will make a successful move, as a company, I believe, is Yahoo. It is the quickest way for them to get real revenue around this stuff. Yahoo! News being the biggest news site on the planet has a lot to do with that. I'm not privvy to specific plans, but if you look at the way their business models work, in general, they have a lot more at stake. And with the moves Reuters is making, their business is more Reuters-dependent and they're going to be forced to move much quicker. There are people inside Microsoft that get it, clearly, and people inside Google who get it, but at a corporate level there isn't the prioritization or big-dollar interest. Google can't become more successful as Google by giving a damn. Same with AOL. Same with Microsoft. The Amazon guys, they get it. Amazon Top Products and similar stuff is available through RSS feeds. I think Amazon will start selling books as serials through RSS?that kind of thing will happen.

HC: As far as ecommerce is concerned, how will transactions work through RSS?

SR: RSS is still served off of web servers. It's still H-T-T-P-colon, and that helps. From an ecommerce point-of-view, it helps because all the SSL and HTTPS stuff can be carried over. I would follow the money in terms of RSS and ecommerce?there's greater innovation where's the greater cash. Dog food online isn't going to be the first RSS-ecommerce story. Porn and travel, on the other hand, probably are.

For the mainstream example, let's look at travel. Travel is actually leading a lot on online innovation because the advertising dollars are so high. We've moved from cost-per-impression to cost-per-click, and next is cost-per-transaction. Travel, led by Hotels.com, has already moved strongly in that direction. I suspect that people will figure out how to do good travel deals in RSS before anything else.

In Europe, there are a couple of discount airfare sites that are pulling out nice feeds of this sort. These little metasearch startups are building these really nice maps of current travels deals. If you want to go from Helsinki to Rome via Barcelona, it's four airlines, but it's only about $300. These startups are spitting out search feeds, reminiscent of Feedster, but with a much smaller data set. Here in the states, there's a great little search startup called Mobissimo that's out-Orbitzing Orbitz. And there are a bunch of garage sites behind them that are putting feeds out, the same sort of thing that Mobissimo is doing. Travel will probably be the first place where ecommerce really hits RSS. And auctions will follow soon after. Someone, within 2005, will start trying to complete transactions by RSS. It won't be mainstream, necessarily.

The sleeper organization is all of this is Interactive Corp. Those guys are quiet, which, since they're led by Barry Diller, is shocking. The social network inside of Evite is moving. A social network built out of people you go to parties with works out pretty well. Diller is sitting there, cranking out commerce, and I would be suprised if he didn't do something interesting and innovative in this area pretty quick. They already bought ZeroDegrees, a pre-revenue social network. Much earlier than Tickle, which Monster just bought. They must have needed infrastructure to back up Evite.

The jobs sector is also going to be huge for RSS, as Craigslist has proven. RSS aggregator adoption in San Francisco took off when the jobs part of Craigslist added RSS. Everyone was like "I'm going to miss that next accounting job!" And now everyone just calms down, goes back to work, and watches the RSS reader.

HC: Where do you think RSS reading is going to end up?

SR: As part of the browser and as part of the email client. There's two schools of thought. I happen to be a browser guy. Half of Feedster are email guys and half are browser guys. It's a cognitive thing. I'm one of those tidy little "keep everything out of my email" people. My browser is where I make my mess.

HC: My feed reader, more than my browser, more than my email client, more than my social network, is a powerful reflection of who I am. And right now, that's very rudimentary. I can only subscribe, unsubscribe, read, and prioritize. When will we see tools that really leverage that data?

SR: Scoble would say that the Longhorn desktop, which will certainly include an aggregator, will take your OPML file and do something interesting with that. And with the Google desktop search app that eveyone has been talking about, Evan would probably say the same thing. The rest of us, who are mere mortals, say "If you publish it, we can help you."

There's a bunch of people who do publish their OPML files, and it's standard within TypePad now to publish them. We crawl them, and so, as relevancy builds over time, we'll certainly have to weight things that people are actually bothering to read.

My business during the bust was a hot-spot search engine called WiFinder, which is still going. There's a bunch of people who care when I blog about wireless, and in my first degree of social network there are all these people who also have something to say about wireless. So, there's a certain crowd of people who might say "I would like to see either all the feeds or all the postings that mention the term 'wireless' within one or two degrees of Scott Rafer." In that sense, OPML within the social overlay might be interesting.

You've got people like Tribe.net, who now have feeds coming out each tribe. More interesting than my actual social network are the communities of interest that I belong to. There's a group in D.C. that I work with called Public Knowledge. There's an Orkut Public Knowledge community and so much more interesting that one degree of Rafer is everyone in that Orkut community looking through their OPML files for mentions of "broadcast flag" or whatever. Much more so than pure social networks, the aggregated OPML files across a community of interest is probably important. And you'll opt in or not when you join a community, but enough people will opt in to make it useful.

HC: Which of the social networking sites is making the best use of RSS?

SR: Tribe is the only one that's making use of RSS right now. All this semantic web stuff is derived from a file format called RDF. RSS is a very simple version of that. The next one to gain any popularity is FOAF, which stands for "friend of a friend." Under the hood, several of the social networks, LinkedIn, Tribe, maybe a couple of the others, are FOAF-based, and it's very easy for them to start turning all those relationships into feeds, if they want. And that's what I personally want, given my own habits. I know a lot of people in my "second degree" in LinkedIn. If I could have a feed of my second degree, as it increases, so I could go into my RSS aggregator and just hit links for "Yes, I know that guy" or "No, don't know him," they would end up with a much richer database, knowing more about me and my network. It would be really time efficient, and I'd be even more likely to pay for their service when they start charging.

Feedster, the news and commentary search engine, currently indexes over 670,000 feeds. Scott Rafer joined the company as President and CEO in October, 2003. Before that, he co-founded WiFinder, a directory of public access Wi-Fi, and Fresher Information, a provider of content indexing software. His personal blog is License-Exempt Soweto. A few weeks ago, we sat down at a cafe in San Francisco and talked about Feedster and the future of RSS. Here is the first part of our conversation...

Harold Check: Can you give me a little background on Feedster?

Scott Rafer: I'm not one of the founders. I founded a similar company in 1998?called Fresher. Scott Johnson and François Schiettecatte had much better timing. They were both doing a lot of blogging, and were both real search guys?going fifteen years back apiece. They just started putting together engines, separately?they didn't know each other yet. One of them was really nice to use, and one of them was really nice on the back-end, in terms of scalability. Then Scott and François found each other, and now we have something that looks okay that also works pretty well.

So, we're a new search engine. There's all this meshugaas about blogger tools and all that, and bloggers are our core audience. They're really important to us. They give us all the information that we have. But our job, really, is to bring it into the mainstream.

We've got these 615,000 sources, as of this morning. (It's always at the bottom of the homepage.) News search has about 5,000 sources, today. But news habits are changing. So, when the explosions happened in Madrid, it was okay to go to CNN and it was okay to go to Google News, but if you wanted to also find out what Madrileños on the ground were saying then that's when people came to us. That's becoming a much bigger part of people's news habits.

Depending on people's politics, they don't believe Fox News or they don't believe the New York Times, or both. In aggregate, what RSS covers out there is much closely to accurate, not in the individual posting, but overall, than any single news outlet going to be. So our job is to grow up and go compete with the Google News and Yahoo! News and really become a search engine at that level.

HC: As a search engine, your business is mainly ad-driven. How does that work in the universe you inhabit, especially in regards to feed readers?

SR: Well, you need to separate here. Most of our business is web pages. You have an RSS aggregator and I have an RSS aggregator. A lot of the people we talk to everyday have RSS aggregators. That makes us weird. It's still unusual. We are still early adopters.

HC: But it will be increasingly less weird, right?

SR: Right. And I'm very happy about that because we put out more feeds than anyone else on the planet. We have greater diversity of feeds going out than anyone else. And we'll eventually make a lot of money on that. However, eating in the mean time is really important, and showing month-over-month financial results in addition to dot-com-like traffic results are very important, from a business point of view.

We'll have a real business in being a publisher for one of the big ad engines, and that's just on the HTML side. For a long time, that'll be the primary revenue stream. You'll see that from us very soon. I can't imagine that's going to be very controversial.

Ads in RSS are going to be controversial. They already have been, to a certain exent. There are a couple of ad networks doing a decent job of it right now. We'll end up doing it. How soon is unclear. We certainly won't do it until it makes money. When it help keeps the service free and growing, we'll use it to keep the service free and growing. Servers have gotten cheap, but they're not free.

HC: Years down the road, will everything that changes on the web be in RSS?

SR: There's an old book called Net Gain. It talks about community forming on AOL and other proprietary online services. These guys wrote this book well before the boom and they nailed it. They, and a number of other people, have talked about the progression in which different sorts of content come into new interactive media. There's news and similar flat content at first. Then you work your way up until you're doing full-on transactions, eBay-style. And then, even beyond that, into certain kinds of messaging. The RSS and blog world is not following this perfectly, but it's following it on an 80/20 basis.

I expect all the eBay bid-streams to be available by RSS. I expect weather and traffic and travel deals. I expect Priceline to create a version of their bidding mechanism that's XML syndicated out so it's even more efficient. This may not be in 2005. It may be a 2006, 2007 kind of thing. You can go to Google AdWords today and see everyone who's paying more then ten cents a click for anything, and expect in three to four years their entire marketing and distribution will have moved to XML syndication of some sort. And it won't be RSS 2.0 and it won't be Atom 0.3 or any of the current standards?it will be whatever those grow into to become secure, virus-free ecommerce platforms. But they'll be direct descendants of this kind of XML syndication.

HC: How do we get there? Who's going to have to step up to the plate to make that happen?

SR: Do you use RSS out of Craigslist? It's a perfect outgrowth of that. Craig's guys did a fine job of saying "Give me a three-bedroom apartment in Duboce Triangle in this price range as an RSS feed." Great. Do that for eBay. "Give me Pez dispensers where the bidding ends in less than four days. And give me a cookie-based secure link to make a bid." Or better yet, attach it to the bidding software that everyone's got going. At worst, it will be exploited by the existing eBay clients. Something like 40% of eBay's auction are posted by XML these days. That's a tremendous number using their web services. Amazon is already moving towards web services. 

In the jargon of that whole world, RSS is just the first popular consumer web service. All the Web Services guys cringe when I say that because it's not a true web service?it's not transaction based, it doesn't meet the computer-sciece threshold for web service. But it's still a million people hitting XML feeds over and over again and responding in kind.

HC: Who's your competition out there?

SR: There's no-one that's trying to approach it as a search problem. There are any number of great blogger tools that are getting more and more complex as blogger tools. Technorati is the obvious one to talk about. Sifry's great. He's a helluva lot smarter than I am. But as he moves farther and farther into what he's doing, I see him running into Six Apart before I see him running into Feedster. We're doing our best to avoid having to teach people anything.

The way our interface will evolve over time?you're going to see it going more and more towards a boring white page, box in middle, hit the search button, with fewer obvious bells and whistles. Things we can build in there intuitively, without having to ask the user, those are great. We're trying to make sure that the big, traditional search engines are the ones doing our user education. We don't want to force people to be enthusiasts before they can enjoy Feedster. For us, it's about broadening the thing, not deepening the thing.

PART II: Which big company will make RSS a priority? Ecommerce and RSS. Browsers versus email. Social networking and RSS.

Brent Simmons is the 36-year-old developer behind the highly acclaimed Mac OS X feed reader NetNewsWire. His company, Ranchero Software, started in 1995, was put on hold for several years while Simmons worked for UserLand. In 2002, he and his wife Sheila "rebooted" Ranchero, and shortly thereafter NetNewsWire was born. A public beta of NetNewsWire 2.0 is set to be released in June.

Harold Check: How do you approach developing NetNewsWire? Do you have a vision you're working towards, or is user feedback the main driver behind changes to the software?

Brent Simmons: Both. The upcoming NetNewsWire upgrade will have lots of changes, and Sheila and I both have our hands firmly placed on the rudder. (Ranchero Software is a  mom-and-pop operation.) NetNewsWire is the result of our vision.

At the same time, we rely very heavily on user feedback. For instance, we have a fantastic group of testers, who do much more than just report bugs ? they make great feature requests and provide feedback on user interface design. Some of the testers are accomplished developers themselves, and it's fairly usual to find a couple user interface mock-ups in my Inbox in the morning.

I don't think you can separate the two things ? vision-driven vs. user-feedback-driven. Our vision is partly a result of what people tell us about NetNewsWire. And our overall vision includes lots of room for feedback: it's nimble in the right places.

HC: What do you make of the standards wars? As a developer, does it bother you to spend time addressing multiple formats and the uncertainty of future formats? Are you in touch with any of the keepers of the current standards?

BS: My job is to treat Atom and RSS as peers, and to do a great job supporting both formats. I do not prefer one over the other, and I go out of my way to stay far away from the fighting. (One of the beautiful parts of newsreaders is the Unsubscribe button.)

Most people don't care about the fighting ? they just want good software. So that's what I focus on.

On the bright side, all this fighting does indicate something important, that syndication is a wonderful technology worth getting fired-up about. It is. Even though I work on it at the plumbing level, it still seems a bit like magic to me.

HC: One of the most compelling features of NetNewsWire is the control you offer users of their reading experience. What's the most compelling feature of web-based feed readers, in your opinion? Can you ever see a point where NetNewsWire might have a server-side element?

BS: The best part of web-based feed readers is probably that you can get to your account anywhere there's a web browser. For some people that's utterly crucial; for other people, not so much.

There are people who use web-based email too, some because it's the only option that makes sense for them, others just because they like it for some reason.

I can definitely see a future where NetNewsWire has a server-side element. In fact, my background is in writing server-side code and doing web apps: I did that for many years before writing NetNewsWire.

However, the first thing to do is synching (which will appear in the next release of NetNewsWire). For people who use just a few computers regularly, synching will allow you to copy your feeds and the read/unread status of items to different computers.

But in a future release I can easily imagine integrating with a server-side system. Perhaps Bloglines, perhaps Feedster, perhaps something else.

I would much rather integrate with an existing service than write my own, even
though I have the experience to do it. Here's why:

1. I could continue to concentrate on the desktop application, and leave all the server-side programming to someone else. This means the application would be of higher quality than if I split my time.
2. I wouldn't have to stay up nights worrying about the privacy and security of users' data on my servers. This is best left to people who work on the server all day every day: it's not a job for part-time server-side developers.
3. I like partnerships. An emotional reason, certainly ? but I just find it more fun to work with other people on things like this.

HC: Just as bookmarks (and bookmark managers) became less useful as the number of websites skyrocketed, will the same thing happen when syndication becomes more pervasive?

BS: The proliferation of websites didn't make *browsers* less useful, they just overwhelmend bookmark managers. I don't think the proliferation of feeds will make newsreaders less pervasive. It will mean some user interface additions, though.

Here are a few examples of features that appear in some other newsreaders, and will appear in the next NetNewsWire, that exist in part because of the proliferation of feeds.

1. Search results feeds. Feedster is an example: You know how you can go to feedster.com and run a search? Well, you can also make a feed out of that search. So if, for instance, you're interested in all mentions of "Apache" everywhere, you'd create a feed that does a search on "Apache" at Feedster. You don't have to be subscribed to all feeds everywhere, you have Feedster do it for you.
2. Smart feeds. Maybe you yourself have subscribed to a few hundred feeds, and you want certain things to bubble up. Again, say you're interested in mentions of "Apache" ? so you'd create a smart feed that shows all unread items whose title or description contains the word "Apache." This kind of thing allows you to subscribe to more feeds but also have the more interesting items make themselves more prominent.
3. Plain old searching. Just like in most mail applications, you have a search box, type a search string, and get a list of matching items.

There are lots of other ideas, too ? feed/items ratings, collaborative filtering, Bayesian filtering.

The question is, can newsreaders grow to make it ever easier to deal with yet more data? Answer: they can. Like everything else, it takes time to do the work, but the work is being done. We're still at the very beginning of all this.

HC: Lately, you've been posting a lot about interoperability between feed readers and weblog editors, and about the notion of feed readers as "RSS routers," able to seemlessy move information to email, the web, databases, SMS, or whatever. Why are you choosing to cooperate with other developers rather than just build those features into NetNewsWire?

BC: The main thing for me is that it's *fun.* I enjoy working with other developers, and it just knocks me out when apps written by different developers work together. So my motives are, actually, rather selfish.

I could take another approach ? just build up NetNewsWire until it's some huge Ranchero RSS Office Suite ? but no way. I wouldn't like it, and users wouldn't like it, and other developers wouldn't like it.

There's a tradition among Mac developers of working together on things like this. It's my belief that this tradition isn't self-sacrificing, it's enlightened self-interest.

It's entirely likely that more good ideas like this will come along ? some will come from Ranchero Software, and some will come from other developers, and I'm just as happy to work with them on their ideas as on mine.

But, for me, this is a huge part of why I'm a developer at all. I love this stuff.

Over the course of the next few days, I'll be publishing interviews with a few of the developers of popular feed-reading software. First up is Luke Hutteman, author of SharpReader.

SharpReader is a popular, free, Windows feed reader, based on .NET protocols. Hutteman, a 35-year-old systems architect in Charlotte, North Carolina, released SharpReader a little over a year ago, and since then he's continued to develop the software in his spare time. While he has no plans to turn the project into a full-time business, he is considering selling default feed placements to generate some income to fund future improvements.

Harold Check: I've become increasingly convinced that the future of syndication will be shaped by those who create tools to manage the feed reading process, as opposed to those who create tools to manage the feed producing process.

Luke Hutteman: Well, ideally there will be some level of cooperation between those that produce the specs, those that write the feed-producing tools and those that write the feed-consuming tools. I do agree with you that the creators of feed consumers seem to have a better grasp of what type and form of information is useful?since we're the ones that have to work with it. It's one thing to produce a feed or write some abstract spec with a great theoretical background, it's a whole different ballgame to use this data in ways that benefit end users.

Of course, being an aggregator developer, I could be somewhat biased in my viewpoint here.

HC: How do you approach developing SharpReader? Do you have a vision you're working towards, or is user feedback the main driver behind changes to the software?

LH: For me, it's mainly my own ideas on what would make SharpReader a better aggregator. I initially created SharpReader for personal use only and in some ways still treat it as such. That being said, I'm definitely influenced by user feedback as well. If lots of people ask for certain features, I do try to make those a priority. That is, as long as they won't needlessly complicate the application. I'm a strong believer in keeping things simple, which is why I've tried, for instance, to keep the number of dialogs to an absolute minimum. As an example of this, SharpReader is one of the few aggregators where you can subscribe to a feed without being forced to go though an "add feed" popup wizard.

HC: What do you make of the standards wars? As a developer, does it bother you to spend time addressing multiple formats and the uncertainty of future formats? Are you in touch with any of the keepers of the current standards?

LH: Needless to say, I'm not happy with the standards wars and resulting competing formats. A single, well specified, extensible spec would certainly have been preferable over the current situation of RSS vs. Atom. Since Atom won't replace RSS, it will just be yet another syndication format to support. And while RSS is often blasted over having many incompatible versions, I don't see why the same won't happen to Atom?versions up 'til now certainly have been incompatible, and no guarantees have been made that future specs will remain compatible with the current (already widely implemented) version.

I'm not actively in touch with any of the keepers of the current standards and have been more of a follower than a leader in this aspect. I do keep track of the Atom process but really don't have the time to get too deeply involved in it. I have a lot of respect for people like Sam Ruby and Mark Pilgrim and trust (hope?) that Atom will move in the right direction. That being said, the design-by-committee approach of Atom does lead to a spec that, in my opinion, seems a bit too broad in the base specification. Some Atom elements (like two of the three dates) seem better suited for a namespace extension than as part of the main spec. RSS was "really simple," Atom is not. What Atom is, though, is better specified than RSS, which should make it easier to handle. Unfortunately, like I said earlier, Atom is not replacing RSS any time soon (if ever) so it's just one more format to handle. This means that from an aggregator-writer perspective, Atom just adds extra work for fairly little benefit.

All that said, implementing Atom as if it were nothing more than an RSS namespace took less time than I expected. It still added complexity I'd rather have done without though...

HC: Do you think that standalone readers will ultimately flourish? Or is feed-reading destined to morph back into existing web tools?either on the client or server side?

LH: I think both models will co-exist for the foreseeable future. When I first heard about RSS and aggregators, I tried AmphetaDesk and the Radio Userland aggregator, and did not like either of them. With their web-based UI, I did not really get what was so great about RSS. Then I tried FeedReader and "got it". I prefer desktop aggregators because they tend to give a better user-experience: they're more responsive, can provide a richer UI experience (supporting things like drag-and-drop and taskbar notification of new items), etc. On the other hand, web-based aggregators have come a long way since then. Nowadays they provide services where they can for instance recommend feeds and items based on other users with similar taste; a client-side aggregator cannot do this as it only knows about your own feeds. Also, a server-side aggregator works better across multiple computers since your state is stored online instead of on your local machine.

The ideal aggregator, in my opinion, would be a combination of both models: a rich client application that gets its data from a server-side process. This way a user can have a rich user experience, while still being able to access his data from anywhere (well, as long as he's connected to the net) and data can be mined across users to make intelligent recommendations on feeds and items.

HC: Bookmarks became less useful as the number of sites skyrocketed. What will happen when feeds become much more pervasive?

LH: This is already happening?it's very easy to subscribe to more feeds than you can actively follow and be left with thousands of unread messages. Early versions of SharpReader kept all items until you manually removed them. In retrospect, this was a mistake. RSS is not about history; it's about what's going on now. If you haven't read something after a couple of weeks, chances are, you never will.

Even with an automatic purge of old items (the default in later versions of SharpReader), the flow of new items can potentially be too big to keep up with. This is where things like filters, search folders and Bayesian filters will need to come in, to make the messages that a user is most interested in show up on top. Also, being able to easily follow links, backlinks and comments becomes more important to read an item in context.

The coming convergence of RSS and P2P (i.e. BitTorrent) could become of what some people are calling the poor man's TiVo. The PVR Blog explains the idea:

BitTorrent allows you to share the downloading of large files with thousands of other people that have the file but there's no easy way to search for new torrent files. RSS is a summary file for sites that can let you know when a site or page has updated automatically when using a RSS reader.

If BitTorrent sites used RSS, you could essentially have much of the functionality of TiVo ("Give me all new episodes of Six Feet Under this year") on your personal computer, without the need for television. Of course, the legal issues around this type of technology would kind of make it impossible to do for very long, but it's perhaps a glimpse into the future of where entertainment could be going.

A legal version of this idea could be soon available when BBC starts making its content available through the Internet Media Player.