AWS ESI 

Creating websites that are fast, cheap, and up-to-date is hard. Not hard like cache invalidation and naming things, but still quite a complex thing to achieve. To fit those 3 requests into an existing architecture can be painful, and at least one normally falls by the wayside in achieving the other two.

To create a fast website you can cache everything. Twice. Your assets? Compress them and cache them. Your generated HTML? Cache it on a CDN's edge server near your users. That dynamic section of the site that absolutely must update every minute? Ajax it in and cache those requests.

This setup should be relatively cheap, depending on the cost of your edge servers, but we're using ajax to keep it up-to-date. This means we're generating additional requests and if we have multiple sections of a page that need to be kept up-to-date, then the number of ajax requests will grow.

We can solve this though, right? Let's not ajax - that's a silly idea. Instead, we'll just generate flat HTML but update it quickly, keeping a short cache time. Great, now we've got rid of the HTML but we're having to generate the HTML whenever a page changes. What if that section that must be kept up-to-date is on the sidebar on every page, and what if we have 40,000 pages on our site? There goes cheap and up-to-date in one go.

A good solution is what are called Edge Side Includes. It's a specification (https://www.w3.org/TR/esi-lang) that allows you to get another program to build your pages from pre-created sections of HTML. The benefit of this is that can have the bulk of our pages pre-created and then just add in the highly-dynamic sections when the page is being served to the user.

There are a number of systems that support this that you may already be using, including Apache, Varnish, Akamai, and Fastly. What there isn't yet is any support in AWS Cloudfront, and this is a shame as AWS's stack is great and cheap. Wouldn't it be nice if you could use ESIs in AWS?

Well, now you can… ish… with caveats, but still, ESIs are cool and AWS is cool so together they're great.

Using the Serverless (http://www.serverless.com) framework, AWS API Gateway, and AWS Lambda, I've created a little open-source project, https://github.com/dittto/serverless-esi that allows you to use S3 to store your flat HTML segments and serve them as a full HTML page.

It currently only supports the basic `` tag which is used for attaching HTML together, but it is quick - much quicker than, say, having Wordpress or Drupal build your page each time it's requested, with an added bonus that as your site gets more popular your response time actually drops!

When using AWS Lambdas, normally they have to be copied to a server once requested which can be a bit slow, but the more frequently they are used then the more likely they are to still be installed on a given server, which in turn means the quicker your site responds, which is also pretty cool.

The other benefit of this is the serverless architecture itself. By removing the load from your servers and into Lambda, you only need a single admin server to generate the ESI HTML files. Oh, and as a bonus, the AWS free tier covers you up to 400,000 seconds of AWS Lambda time, which means you could be serving over a million pages a month and only be paying for admin server.

Have a look and a play with https://github.com/dittto/serverless-esi and tell me what you think below.