Get your Emails read by using Amazon’s Simple Email Service

If you are using Amazon Web Services (AWS) to host a Web Application that sends Emails then you almost certainly need to be using Simple Email Service (SES).

Your IP address might have a dirty history

Public IP addresses (of the IPv4 variety) are a scarce resource and AWS has to recycle them. EC2 instances are allocated a public IP address on start up which will stay the same until the instance “is stopped, terminated or replaced with an Elastic IP address“.

On any hosting platform, cloud or traditional, IP addresses tend to pass through a lot of hands before they reach you and in some cases the IP address that you have been allocated has been used for sending spam emails and is therefore on someone’s blacklist.

If you are not sending emails from your EC2 Instance to the outside world then this isn’t a problem. However if you wish to send emails from your instance then you will need to check your instance’s public IP address on services like Spamhaus, Trend Mirco and the Composite Blocking List (CBL). This isn’t an exhaustive list of spam blacklisting services, in all likelihood you will come across many others.

Getting your instance’s public IP address de-listed from a spam blacklist can be a challenging experience. Trend Micro for instance require that your IP address is marked as a static IP address before they remove it from their blacklist, to do this you will need to allocate an Elastic IP Address to your instance and then configure the reverse DNS for that IP address.

You will need to check the history of every public IP address that you want to send emails from. If the number of instances that you are operating is changing dynamically using Auto Scaling then this could be a challenging task. One solution might be create a single instance as your public SMTP server and then configure all your other instances to use that as a smart host.

Lastly if you are sending a substantial number of emails from your EC2 instance then you will need to apply for your email sending limits to be removed.

It’s possible to send emails from an EC2 instance without fearing that it will be marked as spam, AWS give you all the tools that you need, but it might just be more effort than it’s worth.

Simple Email Service to the Rescue!

Simple Email Service (SES) saves you the effort of creating your own email platform for your applications. You don’t need to worry about keeping IP address off blacklists because AWS takes care of that for you. You also don’t need to worry about availability, SES runs on multiple availability zones meaning that you are very unlikely to ever experience a total failure. SES is scalable, assuming that you are trustworthy, you can send tens of thousands of emails through SES very rapidly. And best of all SES is probably going to cost you next to nothing.

You can connect to SES using an SMTP Interface or through an API that can be used from most languages such as Java, .NET, PHP, Ruby, Etc. It is likely that your existing application uses SMTP so the SMTP Interface is probably going to be easiest to implement to begin with. If you do choose the SMTP route you may want to configure your existing email server as a smart host to SES because you can abstract the process of authentication to SES away from your application which means that you don’t need to change the code of your application.

What’s Missing?

I’m convinced that Platform as a Service offerings like Elastic Beanstalk are the future. So I would really like to see AWS offer the ability to configure the Elastic Beanstalk environment with SES SMTP credentials. This would mean that the local SMTP server on each instance in your Elastic Beanstalk environment would have a Smart Host that was pointing to the SES server. It’s worth pointing out that this can be achieved today manually by creating your own custom AMI for Elastic Beanstalk.

In Conclusion

I’d like to stress again that IP addresses having a dirty history is true of any hosting platform, I just highlight the problem here because AWS provide you with SES as simple solution to do something about it.

Basically it is likely that a team of developers and support engineers can make a better job of creating an email platform than you can.

If you are currently using AWS then SES is probably the logical choice and even if you are not using AWS today you should strongly consider SES for sending Emails from your Web Applications.

Please feel free to share your thoughts with me.

Amazon Web Services – The Missing Features

I like Amazon Web Services (AWS), the rate of change and improvement is staggering but there are still things that they can do to make it even easier for people to create cheap and highly available systems.

HTTP and HTTPS Logs for Elastic Load Balancers

If you have multiple Web Servers behind an Elastic Load Balancer then you have a problem when it comes to Log file analysis. Each server is going to create its own set of logs, in order to analyse them you are going to have to merge your separate logs together using something like logresolvemerge from AWStats. If you are using Auto Scaling you have the additional problem that servers can disappear depending on load.

There are several solutions to this problem such as using a third party logging service like Loggly or by solely relying on page tagging services like Google Analytics but any solution you create is going to add complexity to your architecture which will require maintenance and increase costs.

Load Balancer with HTTP and HTTPS Logging Concept

How the Load Balancer set up page could look with logging

I’d like to see AWS create a new feature which allows you to log all HTTP and HTTPS traffic that passes through a load balancer. It should allow you to specify an S3 Bucket and target prefix just like S3 Logging currently allows. It would make sense if the log format was the same as the S3 logs but in the future I’d like to see IIS and Apache log formats supported. Logging on the load balancer would reduce a large amount of complexity in a load balancing system. It would also allow for 3rd party developers to create analytic tools that could be purchased on the AWS marketplace.

Elastic Beanstalk in the EU region

Elastic Beanstalk and other app engines are the future. One day, very soon, no one will need to be a system architect. But that day is a little while off and part of the reason for that is that Elastic Beanstalk isn’t available in the EU region. This is a problem for companies that have regulatory compliance issues which mean that data can’t leave specific regions. The simple solution would be for AWS to make Elastic Beanstalk available in more regions!

More Regions

AWS opened my eyes to working with one hosting provider across multiple regions. On many projects I’ve worked on I’ve had to solve latency issues or had regulatory issues to comply with. Having one provider for EU, US, etc is fantastic but now I’ve seen how great it is i want more! Australia, New Zealand and Canada would be a great start because they are already developed markets.  I suspect I won’t have to wait to long for new regions.

S3 bucket sizes

S3 is great, so easy to use that most companies seem to be storing an ever increasing amount of stuff on there, log files, backups, static media are all good use cases. However there is a price (all be it small) for all this data it would be great to get a quick over view of the total number of objects and the total size of a bucket from the console.

Route 53 to Support S3 Hosted Sites on the Root Domain

At the moment there is a movement towards (or backwards if you are old enough to remember the early days of the web) creating static websites using tools like Octopress. Static sites often make a lot of sense, there is normally very little on the average Blog that really needs to be dynamic.  By hosting a static site on S3 you can avoid instances costs and create a highly scalable website without any pain.  You could easily create a static site that was viewed by millions of people a day for a running cost of a few dollars a month.

The only catch is that Route 53 doesn’t allow you to host sites on the root domain using S3, this means you are forced to use a sub domain prefix such as www.  You might not agree with no-www.org but you should probably do something with your root domain.

Werner Vogels is using a redirect for his site All Things Distributed so if you enter allthingsdistributed.com it will redirect you to the S3 hosted site at www.allthingsdistributed.com.  This is fine but it doesn’t feel like an elegant solution.

I’d like to see Route 53 allow S3 hosted sites on the root domain.

Auditing for Identity and Access Management (IAM)

IAM allows you to control what level of access to your AWS resources your users have. For instance you can specify that users have the ability to restart instances but deny them the ability to terminate instances.

IAM is great for controlling access but it doesn’t allow you to answer the questions who did what and when did they do it.  Highly available systems are fine but they can still be brought down by employees making mistakes, having an audit trail will often help identify training issues.

Open Source NoSQL Service

Most of Amazons services, such as ElastiCache and Elastic MapReduce, have mature open source equivalents which many companies are already using.  For some reason (I suspect political) AWS chose not to take this route when creating a NoSQL Service and instead created DynamoDB which has no Open Source equivalent. Not having an Open source equivalent is possibly limiting the adoption of DynamoDB because System Architects are wary of tying themselves to a single vendor.  It would be great to see something like MongoDB or CouchDB available as a service from AWS.  Ideally AWS would create a NoSQL Service like their own Relational Database Service which allows for multiple different engines.

More RDS Database Engines

I’d love to see PostgreSQL available as an RDS Database Engine option.  In the last 12 months Oracle and Microsoft SQL Server have been added which is great and indicates that over time we can expect to see more relational databases made available as a service.

A Roadmap

At numerous times I have seen a gap in the AWS offering and I have spent time creating my own solution only to find that AWS launch their own solution a few weeks later! This is great because it shows the fantastic rate of progress that AWS are making and that they are listening to their customers but I have also wasted several weeks on my life creating solutions that are redundant!

I would love a roadmap, maybe giving a preview of the features that are coming in the next month.

In Conclusion

I hope you have found this post interesting, I’d love to hear your thoughts and experiences of using AWS.