Registry of Public Email Senders – Notes
David MacQuigg,
Governments are understandably reluctant to mandate technical standards for email authentication. Fortunately, there are many things that can be done to facilitate authentication without any mandates.
A Registry of Public Email Senders would be low-cost, low-risk, not interfering with other anti-spam programs, not needing mandatory standards or any other regulations, and could have a huge impact. The Registry could promote some simple, neutral standards, like the format of the records to be kept in the Registry. Establishment of these simple standards does not require that a final winner be chosen among the competing authentication methods. They can all coexist within the standards. They can all make use of the Registry.
What else could be done short of mandatory regulation? A large organization like the FTC could promote a standard format for a Sender's ID declaration, and overcome an issue that has been blocking cooperation for over a year. Same with a standard authentication header. Just keep it simple, and leave room for all the optional parameters each of the methods will want to add.
With a Registry and some simple standards, adoption of authentication should quickly reach the tipping point. If more push is needed, still short of legal mandates, a small amount of funding for upgrades to the most popular MTA programs would help. These could be distributed via a central website with technical support.
Finally, if legal force is really needed, the regulation could be simple. Anyone who wants to operate a Public Mail Server must authenticate, by whatever method they chose, or maybe a list of approved methods. There isn't much opposition to authentication, just a lot of apathy. If they have to do it, they won't be fighting over details of the regulation. They will just pick the easiest method, install the upgrade, and get on with business.
The Email Registry anticipated by Congress in 2003 was a list of recipient addresses, similar to the Do Not Call Registry. There are a number of differences between the email system and the phone system that will require a different kind of registry. The significant differences affecting planning and implementation are:
1) The Do Not Call Registry is an Opt-out system for recipients, depending on the government to enforce the law against unwanted callers. The Registry of Public Mail Senders should be an Opt-in system for senders. Registration is voluntary, so it will require no new laws or regulations. Enforcement will be done by recipients, who will reject mail from any domain that isn't in the Registry or doesn't have a good rating from an acceptable reputation service.
2) The Do Not Call Registry is supported by a small network of computers. The Email Registry must support a much heavier load, perhaps 100 million queries per day. Existing DNS technology is perfectly suited for a fast, efficient, and secure Registry, with servers widely distributed, resistant to denial-of-service attacks, and not loading any DNS servers performing other vital functions on the Internet.
3) The Do Not Call Registry is a simple list of phone numbers. The Email Registry will contain complex information with tricky formats and frequent changes. To avoid a technical support burden, the Registry should simply copy and merge information from records that have already been tested by the domains and the domain-rating services that generate the information.
4) The Do Not Call Registry uses existing standards for web interfaces and file transfers. The Email Registry will use existing DNS standards, perhaps with a few enhancements to improve speed or security.
5) The Do Not Call Registry must be carefully controlled to avoid abuse of a huge list of private numbers of individuals. The Email Registry will be a much shorter list of domains that wish to operate Public Mail Servers. Everything in the Registry is public information, with no need for privacy.
6) The Do Not Call Registry uses methods that are not controversial. To keep any controversy out of the Email Registry, it should not get involved in the technical debates, or any disputes over domain ratings. Domain owners should decide which authentication methods they will offer, and independent rating services should handle all disputes over domain ratings. The Registry simply provides a merger of the records published by the domains and the services.
7) The Do Not Call Registry involves the FTC in processing a large number of consumer complaints. The Email Registry would not have that burden. Spam bounces go to the recipient's ISP, and from there to the rating services. The Registry would only collect statistics.
8) The Do Not Call Registry is limited to the
While this list may not be complete, the remaining differences should have little impact on planning and implementation. In general, deployment and operation of a Registry of Public Mail Senders will be much easier than the Do Not Call Registry.
The Registry will involve a lot of effort and detailed planning to get it right. To guide this effort, we offer some fundamental requirements.
1) Use of the Registry should not limit or interfere with the use of other tools, such as IP blacklists. It must integrate smoothly with existing mailflows, and must not, at any point in the adoption cycle, reduce the effectiveness of current anti-spam tools.
2) The Registry must respond quickly to a query on any Identity[1] of a Public Email Sender[2]. The response packet should be sufficient for a complete authentication and rating check.
3) The Registry must always serve the best interests of email receivers. With regard to the selection of authentication methods and Rating Services[3], it must be neutral in both technology and administration.
4) The Registry must provide a useful service to receivers, even at initial deployment, when most senders are still rated as "unknown".
5) The Registry must have a means for automatically and securely updating its records, based on information provided by Senders and by Rating Services. We need both periodic updates and fast response to change notices.
6) The Registry must avoid burdens wherever possible. This includes technical support, resolving disputes over domain ratings, handling consumer complaints about spam, and efforts to standardize or enforce authentication.
7) The Registry must minimize Internet traffic by providing as much compact and useful information as possible in response to a single query, and by making the lifetime of that information as long as possible.
8) The Registry should avoid creating any new opportunities for DoS (Denial-of-Service) attacks or abuse of the Internet infrastructure. Scalability to the entire Internet is essential. Security should be at least as good as the DNS system that now runs the Internet.
9) The Registry should use existing and proven technology for a large, widely distributed database. Worldwide upgrade costs, including technical support for MTAs, should be minimized.
10) The design of the Registry should avoid anything that would make difficult the replication of its services in other countries, or the transfer of administration to an international organization.
[1] An Identity is a unique name in the Registry designating an organization or individual willing to assume responsibility for email from a particular domain.
[2] A Public Email Sender is any person or organization that authorizes an MTA (Mail Transfer Agent) to send email to unrelated receivers.
[3] A Rating Service is any organization that provides ratings of email senders, whether those ratings are based on accreditation (working with the sender) or reputation (working with receivers).
2) The response need not be a single packet, but efficiency is a concern, and multi-packet protocols will generally be less efficient. The information should be complete, but not necessarily optimal. The initial goal is to get a working system. Later optimizations can be done as senders add more elaborate authentication protocols and data.
3)
Here are some details on one possible implementation meeting the requirements.
1) The Registry will be hosted on a network of fast, widely-distributed, and well-connected DNS servers, with security at least as good as the DNS root servers. A little over-design here will ensure that spammers don't even think about attacking these servers.
2) The authentication record for each domain is located at <Identity>.mail-id.net Initially, all Senders have a Default Identity. This can be extracted from the domain name in the HELO command used by their sending MTA. For example, if the HELO name is mailout17.dallas.example.com, the Default ID would be example.com. If it is mailout17.london.example.co.uk, the Default ID would be example.co.uk.
3) Registered Senders are not restricted to the Default Record setup. They can use any name in any part of the DNS hierarchy that they control. This might be advantageous for a large domain that wants to keep separate records for separate subdomains. Registered Senders can specify their authentication method, limit the range of IP addresses for which they assume responsibility, and suggest Rating Services to be included in the record. They have complete control of their Registry record, except for the ratings, of course.
4) The Default Record for a Sender will include a list of all IP blocks allocated to the organization owning the Sender's domain. If it is not easy to determine these allocations from public records, a Default ID will be assigned based on the entire IP block allocated by a Regional Registry. Legitimate senders will find it to their advantage to use a registrar that makes this information readily available.
5) Rating information will be under the control of selected Rating Services. This information will be merged with the authentication information for each domain, so that the response to a query on that domain will be rapid and complete.
6) The Registry will be neutral with regard to authentication methods and Rating Services. Senders may chose the method for their domain, and what authentication data to include in their record.
3) Changes to a record will be verified with a DNS query or an email to "postmaster" at the domain being changed. Domains can avoid unauthorized changes by making sure their registered servers are secure.
Q1) What if senders authorize only a few IPs, and leave the rest of their IP block uncontrolled? Won't this remove an incentive to clean up their zombies?
A1) We need to deal with the two problems separately - stopping forgery of reputable names, and limiting network abuse. The Registry can support both, and it need not displace any current methods for limiting network abuse - such as IP blacklists and port 25 blocking. For example, an ISP with two authorized senders and hundreds of zombies might have no trouble getting its authorized mail delivered, even while the zombies are spreading worms as fast as they can. An NSP handling traffic from that ISP could use the Registry record to expedite the authorized mail, and an IP blacklist to reject everything else. With a bypass for legitimate mail, IP blacklisting could get much more aggressive.
ISPs and NSPs have plenty of methods to deal with network abuse. Let's not compromise the solution to the forgery problem.
Q2) Tell us more about how to screen out the low-quality Rating Services. How will you avoid attempts to break the system with false spam reports or bogus good mail?
A2) Spam reports from recipients go to their ISP, then a sample of those go to the Registry, where statistics are accumulated. This can provide a raw score for each domain, based on the number of spam reports and the total mail flow. Using standard multipliers to correct for spam that is not reported and spam reports that are invalid, each domain gets a numerical score like 125, meaning that recipients of emails from that domain see one out of 125 as spam.
To rate the Rating Services, just average a large number of scores for domains in a particular category, e.g. domains with an A rating from service S1 average about 113 good messages for every one spam. The Registry will suggest that the services adjust their procedures to get consistency in domain ratings, but the services can do as they wish. If an A rating from one service is a lot better than an A rating from another, that will show up in the numerical scores.
So how will the bad guys break the system? False statistics could result from attempting to damage someone else's reputation with false spam reports, or boost their own reputation with a flood of bogus mail to themselves. In the case of false spam reports, the Registry will provide data to facilitate investigations, but the domain owners and Rating Services must deal with the problem as they do now.
In the case of bogus mail, there will be identifiable patterns – e.g. millions of emails with no complaints to just a few bogus domains, and 100% spam to domains that have no bogus recipients collaborating with the spammers. Normal mail flows will have spam ratios that don't differ much from one recipient domain to another. To hide the spam, bogus flows will have to be really huge, and evenly distributed to each victim domain.
Most of the problems that have held up email authentication can be avoided by a properly designed Registry of Public Email Senders. Here are some of those problems and how the Registry can avoid them.
1) Senders have little incentive to offer authentication until they start seeing a significant amount of rejection by receivers. Receivers can't rely on authentication as long as most senders don't offer it, and a lack of authentication means nothing. Many are calling this the "chicken-and-egg" problem.
Solution: The Registry gets around this problem by not requiring initial participation of senders. The default record is far from optimal, but the error is in the direction of making senders *more* responsible for what comes out of their IP blocks. Senders who cooperate can modify their authentication record, and reduce the size of the IP blocks for which they have to maintain strict control of outgoing spam.
2) No progress can be made until the techies settle their debate over which authentication method is the best.
Solution: The Registry will accommodate all methods, and it will facilitate the use of these methods by allowing method-specific data to be included in the Registry record for each domain. The domain owner will decide which methods he will offer and what to include in the Registry record.
3) The war between the different methods is so intense, that no cooperation is possible, even on the simple things needed for inter-operability – a sender's declaration of identity, and a forwarder's authentication header.
Solution: Once the Registry is widely used, domain owners will find it to their advantage to comply with these simple standards. The method advocates will not be able to block what the domain owners chose to do. For example, a forwarder that uses a standard authentication header will find that it can hand off responsibility for content to the authenticated identity. The forwarders reputation will then depend only on its reliable authentication of the previous sender.
4) Some of the authentication methods pose a significant risk for Denial of Service attacks involving DNS servers. We can't have widespread deployment until these worries are eliminated.
Solution: The Registry will use fast, widely-distributed, and secure servers, completely independent of the normal DNS servers that run the Internet. This will not prevent some of the abuses, but there will be two key advantages of using a Registry. a) The abuse is not likely to involve DNS servers providing other critical Internet functions, and b) The method being abused can be easily replaced by less popular but more secure methods. The Registry will avoid the "lock-in" situation that each of the methods is now pursuing.
5) Some of the authentication methods have a problem with forwarders not distinguishing one source of email from another, and effectively blocking the receiver from identifying the original sender.
Solution: Rating Services will treat each forwarder as just another sender until that forwarder provides authentication of its incoming mail. By authenticating incoming mail, and adding a standard authentication header, the forwarder hands off responsibility for the content of the emails to the prior sender. The forwarders sole responsibility is then accurate authentication of its incoming mail. Forwarders will have the same incentive as regular senders to choose the most reliable authentication methods, and avoid those whose failure could damage their reputation.