It has become really fashionable lately to talk about the death of e-mail (even VCs do it, see here). Two major issues, at least in my mind, aren't typically being covered, however: the analogy with usenet, and the impact of market-dominant e-mail clients (aka "what Microsoft, Lotus, AOL and Yahoo could do about it"). This post is about these two issues.

The question is: is e-mail becoming an infeasible communications medium given the dramatic growth rates of spam?

I think the easiest answer for those of us who remember being infuriated one morning by the infamous green card legal services advertising post on usenet (first commercial mass posting for many internet users in 1994, see Wired story), is to answer: unless something dramatic occurs, the exact same thing is going to happen to e-mail as happened to usenet since 1994.

If you are only dimly aware of what usenet may be, that is my point exactly: it is still around, it has more newsgroups than ever and probably more traffic than ever, but due to spam, it has become so irrelevant that most internet users today won't even have heard its name. When after that infamous, first spam post to usenet people like myself started to worry about the future of usenet, we were worrying about the exact same thing people are worrying about when worrying about the potential death of e-mail.

It's important to remember that back in spring 1994, the Web browser was really only an interesting fringe application, while usenet was an incredibly useful resource that allowed communication and collaboration like no other. For example, I used it to make contact with some of the later-famous ""object-oriented analysis and design" methodologists who regularly hung out in comp.object (a name that I had to look up for this post because I had forgotten!) There would not have been any other way to create inter-continental relationships like those. And all of us usenet users were terribly worried that this incredibly useful communications medium would soon become unusable.

Fast-forward 9 years, and it's the exact same cry, for e-mail. Except that with hindsight, we really hadn't needed to worry: soon, this thing called the web came around, and people developed private versions of usenet (today known as discussion groups, forums, community sites etc.), and so new technology allowed us to leave the decaying usenet behind with all its spam.

For those who argue that the global e-mail network is far too valuable to leave behind due to its incredible network effect, I'd respond: that was as true for usenet back then, and people still left it behind. Not as an either/or type of thing, of course, but they gradually stopped using it, and that is already beginning to happen with e-mail: many kids don't bother with e-mail any more, for example.

As of today, it is still unclear what that new replacement technology would be (not that it was clear for usenet either for a long time while people hung on to it getting increasingly desparate). Ray Ozzie of Groove Networks, naturally, argues that it will be Groove, and it would be seriously diconcerting if he was going to argue anything else. However, in his argument, how collaborative workspaces suddenly became a "required feature" to solve the e-mail spam problem is unclear to me. (I do understand, and generally agree with the value proposition of collaborative workspaces, but that's a different subject entirely.) The features in Groove that would help against spam are 1) strong identities of all users, and 2) having to obtain permission ("invitations") prior to being able to send messages to anyone.

Which brings be to part two of this post: can anyone fix e-mail without having to abandon it and replace it with something else, as we had to in case of usenet? I think the chances are much better than is generally acknowledged if one looks at the problem from the perspective of market dominance rather than industry standards. Let me explain.

Microsoft Outlook has a fairly large market share in e-mail clients. (I'm just stating this as a fact, not a value statement.) If tomorrow, Microsoft implemented the following algorithm in Outlook, thus making it Outlook++:

• Upon telling Outlook++ about a new e-mail account in the preferences, Outlook++ automatically generates private and public keys for encryption and signing. No user input required. (don't tell me that keys without password are not secure — they are much better than what we have today, which is: no keys at all)
• Upon sending e-mail from alice@alice.com to bob@bob.com, Alice's Outlook++ looks up Bob in the Outlook++ address book. If the record there indicates that Bob is on Outlook++, Alice's Outlook++ contacts Bob's Outlook++ using the SMTP++ protocol (I made this protocol up, but it could be anything. In particular, it could be SMTP, or a protocol on top of SMTP to leverage the existing infrastructure, or whatever.)
• If Bob is not on Outlook++, Alice's Outlook++ sends an auto-generated e-mail message instead through regular SMTP that says "sorry Bob, Alice is taking drastic measures against spam, and you will need to upgrade to Outlook++ if you want timely responses from Alice. Once you have it, respond to this message." (which will automatically updated Alice's record of Bob in her address book because all Outlook++ sent messages contain up-to-date sender information etc.). If Bob does not respond, Alice's Outlook++ will send the original message a few hours later through traditional SMTP, maybe with the attachment that he really should upgrade (I'm thinking something non-intrusive like the Hotmail footer).
• Upen receiving e-mail, Bob's Outlook++ performs this categorization:
  • standard message not sent from Outlook++: put into "may be spam in-box" (today's in-box). It could be filtered like any other regular e-mail message today. If Alice remained the only user ever of Outlook++, we are no worse off than we are today.
  • Outlook++ message from known recipient (i.e. contained in address book): put into "important in-box"
  • Outlook++ message from unknown recipient: put into "not so important in-box"
  • The categorization algorithm would be customizable, and because of the strong identities involved, it would be more powerful/reliable than what we do today with our filters. In particular:
  • Organizations could create organizational identities (of which the individual working for the organization could use in addition to their own personal identity when sending e-mail).
  • Intriguingly, individuals could use their friend's "I'm the friend-of-a-friend certificates" to get higher priorities in in-boxes than without (with permission of the friend, of course). This could be implemented in a variety of ways; which exact one is not relevant to this post as it can clearly be done.
  • As an additional feature, not necessary in version 1.0 of Outlook++, a web service could be accessed by Outlook++ that allowed me to purchase unknown-sender-signatures: by using those, and assuming that Bob has a corresponding categorization rule, Alice could obtain a higher priority in Bob's in-box than the default for an unknown sender. Having such a rule makes sense for Bob, because if Alice pays money to get her message in front of Bob, she clearly has something more valuable to say than someone who does not. (The charged money could go to charity, or to Bob.)

If that product when on the market with these particular default settings, what do you think would happen? (even better, if Microsoft and Lotus and Yahoo and AOL all agreed on the same scheme). First of all, users like me (Apple Mail against sendmail) would be screwed, but we are clearly in the minority. As this is about network effects, I bet that the market for e-mail protocols would flip from SMTP-only (today) to SMTP++ within a fairly short amount of time: While spam would still be sent, it would have no chance to move up the priority ladder, and thus the business case for spam would shrink with the marketshare of regular SMTP.

I realize that it would be quite difficult to get all the major e-mail vendors and ISPs and whatever to standardize, and then implement, a new protocol like my hypothetical SMTP++. However, this is not true for the market-leading e-mail client vendors. Even if Microsoft did it by themselves, without standards support of any type, I'm fairly sure that this time the outcry over "Microsoft is taking an open standard (SMTP here) and closing it" would be much more muted; it clearly would solve a very expensive problem. Note that only a market-dominant vendor could do this, not one with a small market share: and that's ultimately why I post this rather than develop it myself.

Where's the hole in my argument? Is there? Why does no one do it? I'd like to hear from you.