Feedster, the news and commentary search engine, currently indexes over 670,000 feeds. Scott Rafer joined the company as President and CEO in October, 2003. Before that, he co-founded WiFinder, a directory of public access Wi-Fi, and Fresher Information, a provider of content indexing software. His personal blog is License-Exempt Soweto. A few weeks ago, we sat down at a cafe in San Francisco and talked about Feedster and the future of RSS. Here is the first part of our conversation...

Harold Check: Can you give me a little background on Feedster?

Scott Rafer: I'm not one of the founders. I founded a similar company in 1998?called Fresher. Scott Johnson and François Schiettecatte had much better timing. They were both doing a lot of blogging, and were both real search guys?going fifteen years back apiece. They just started putting together engines, separately?they didn't know each other yet. One of them was really nice to use, and one of them was really nice on the back-end, in terms of scalability. Then Scott and François found each other, and now we have something that looks okay that also works pretty well.

So, we're a new search engine. There's all this meshugaas about blogger tools and all that, and bloggers are our core audience. They're really important to us. They give us all the information that we have. But our job, really, is to bring it into the mainstream.

We've got these 615,000 sources, as of this morning. (It's always at the bottom of the homepage.) News search has about 5,000 sources, today. But news habits are changing. So, when the explosions happened in Madrid, it was okay to go to CNN and it was okay to go to Google News, but if you wanted to also find out what Madrileños on the ground were saying then that's when people came to us. That's becoming a much bigger part of people's news habits.

Depending on people's politics, they don't believe Fox News or they don't believe the New York Times, or both. In aggregate, what RSS covers out there is much closely to accurate, not in the individual posting, but overall, than any single news outlet going to be. So our job is to grow up and go compete with the Google News and Yahoo! News and really become a search engine at that level.

HC: As a search engine, your business is mainly ad-driven. How does that work in the universe you inhabit, especially in regards to feed readers?

SR: Well, you need to separate here. Most of our business is web pages. You have an RSS aggregator and I have an RSS aggregator. A lot of the people we talk to everyday have RSS aggregators. That makes us weird. It's still unusual. We are still early adopters.

HC: But it will be increasingly less weird, right?

SR: Right. And I'm very happy about that because we put out more feeds than anyone else on the planet. We have greater diversity of feeds going out than anyone else. And we'll eventually make a lot of money on that. However, eating in the mean time is really important, and showing month-over-month financial results in addition to dot-com-like traffic results are very important, from a business point of view.

We'll have a real business in being a publisher for one of the big ad engines, and that's just on the HTML side. For a long time, that'll be the primary revenue stream. You'll see that from us very soon. I can't imagine that's going to be very controversial.

Ads in RSS are going to be controversial. They already have been, to a certain exent. There are a couple of ad networks doing a decent job of it right now. We'll end up doing it. How soon is unclear. We certainly won't do it until it makes money. When it help keeps the service free and growing, we'll use it to keep the service free and growing. Servers have gotten cheap, but they're not free.

HC: Years down the road, will everything that changes on the web be in RSS?

SR: There's an old book called Net Gain. It talks about community forming on AOL and other proprietary online services. These guys wrote this book well before the boom and they nailed it. They, and a number of other people, have talked about the progression in which different sorts of content come into new interactive media. There's news and similar flat content at first. Then you work your way up until you're doing full-on transactions, eBay-style. And then, even beyond that, into certain kinds of messaging. The RSS and blog world is not following this perfectly, but it's following it on an 80/20 basis.

I expect all the eBay bid-streams to be available by RSS. I expect weather and traffic and travel deals. I expect Priceline to create a version of their bidding mechanism that's XML syndicated out so it's even more efficient. This may not be in 2005. It may be a 2006, 2007 kind of thing. You can go to Google AdWords today and see everyone who's paying more then ten cents a click for anything, and expect in three to four years their entire marketing and distribution will have moved to XML syndication of some sort. And it won't be RSS 2.0 and it won't be Atom 0.3 or any of the current standards?it will be whatever those grow into to become secure, virus-free ecommerce platforms. But they'll be direct descendants of this kind of XML syndication.

HC: How do we get there? Who's going to have to step up to the plate to make that happen?

SR: Do you use RSS out of Craigslist? It's a perfect outgrowth of that. Craig's guys did a fine job of saying "Give me a three-bedroom apartment in Duboce Triangle in this price range as an RSS feed." Great. Do that for eBay. "Give me Pez dispensers where the bidding ends in less than four days. And give me a cookie-based secure link to make a bid." Or better yet, attach it to the bidding software that everyone's got going. At worst, it will be exploited by the existing eBay clients. Something like 40% of eBay's auction are posted by XML these days. That's a tremendous number using their web services. Amazon is already moving towards web services. 

In the jargon of that whole world, RSS is just the first popular consumer web service. All the Web Services guys cringe when I say that because it's not a true web service?it's not transaction based, it doesn't meet the computer-sciece threshold for web service. But it's still a million people hitting XML feeds over and over again and responding in kind.

HC: Who's your competition out there?

SR: There's no-one that's trying to approach it as a search problem. There are any number of great blogger tools that are getting more and more complex as blogger tools. Technorati is the obvious one to talk about. Sifry's great. He's a helluva lot smarter than I am. But as he moves farther and farther into what he's doing, I see him running into Six Apart before I see him running into Feedster. We're doing our best to avoid having to teach people anything.

The way our interface will evolve over time?you're going to see it going more and more towards a boring white page, box in middle, hit the search button, with fewer obvious bells and whistles. Things we can build in there intuitively, without having to ask the user, those are great. We're trying to make sure that the big, traditional search engines are the ones doing our user education. We don't want to force people to be enthusiasts before they can enjoy Feedster. For us, it's about broadening the thing, not deepening the thing.

PART II: Which big company will make RSS a priority? Ecommerce and RSS. Browsers versus email. Social networking and RSS.