Automated mass hosting with AWS Cloudfront, S3 and CDK
This is the third in a short series of articles on using AWS managed services for domain parking. This article cover hosting large numbers of static websites on S3. My current portfolio is only a few hundred domains but I’ve had many more and I wanted something that would support thousands of domains.
I have domains under many TLDs that generate traffic in various niche markets. All need unique content and advertisements to be properly monetized. The first step in being able to provide unique content is providing a unique hosting environment for each. Nuxt will generate pages with all required content but there needs to be something to serve that content.
We’ll use S3 to host the site and Cloudfront in front of that with my SSL certs from the last article.
Costs
The costs for this phase are minimal. There are no fixed costs, simply charges for storage and bandwidth. These are consumption based charges and the costs are the same whether there is one bucket and one Cloudfront distro or hundreds. We want to optimize and silo each site, so we’ll be creating hundreds of Cloudfront distros, each using its own cert and pulling content from its own unique directory. Notice I didn’t say its own bucket? We’ll get to that later.
Storage Costs
Monthly storage pricing is $0.023 per GB. Note that is .023 not .23 cents per month. Most static parked sites will use far less than one gigabyte per month. That is only one or two cents per month per domain.
Request Cost
This is a fee AWS charges for each request of that type, regardless of the amount of data returned.
WRITE Costs
Write operations (PUT, COPY, POST) as well as LIST are $0.005 per one thousand. I don’t update the parked sites often, hence the termed “parked”. I do update content to do A/B testing to see if I can find better converting content. These should never exceed one cent per domain per month, for many domains it will simply be a one time cost of less than one penny.
READ Costs
Read costs are even less expensive. They are only $0.0004 per one thousand, that’s a whole extra zero lower. One visit mat result in multiple requests, choices made later will affect this. Again it’s unlikely we would ever see more than a few thousand requests on a parked domain and that doesn’t even add up to a penny per month.
For budgeting I’ll assume each domain will cost a penny per month for combined read and write requests.
Data Transfer Costs
Data transfer, or bandwidth, costs are obviously zero if the domain doesn’t generate any traffic. Assuming it does the cost is still minimal.
There is no S3 bandwidth charges for traffic between S3 and Cloudfront so as long as all the traffic goes through Cloudfront we only pay its bandwidth fee. That is a measly $0.085 per GB.
I’ll optimize my content for speed, and not just to cut costs. The primary driver is that Google prefers fast websites. With an optimized site I’ll assume less than one GB per month. I’ll budget five cents per domain per month for data transfer.
Total Costs
The total costs for hosting are two cents for storage, one cent for requests and five cents for data transfer. That’s a meager 8 cents per month. (2+1+5)
That is only 96 cents per year per domain. I’ll round that up to an even $1 per year per domain, total. That’s as low as the cheapest virtual hosting provider and this is infinitely scalable with zero effort or changes. This could scale up to millions of domains just fine.
S3 Bucket
I use a single S3 bucket to host all domains. This may confuse people used to thinking that the S3 bucket name needs to match the hostname. In fact many people are used to using at least two S3 buckets, one for the root domain and one to redirect “www” to the root site. Or visa-versa, depending upon your preferences.
AWS has a default soft-limit of 100 buckets in an account. With two buckets required per domain for the site and the redirect that limits us to 50 domains. It’s pretty easy to get AWS to increase that limit, but they might start to balk at making the limit ten’s of thousands. Scaling out and using a new AWS Organization for each group of 50 domains is an option. That would become a management headache no matter how well automated and orchestrated.
The primary reason for using bucket is security. It allows asserting permissions at a single point and have them cascade to all sites. Security is easiest to ensure when there are as few places to check and as few exceptions as possible. This approach leaves the other 99 buckets for additional uses.
Domains, or more correctly hostnames get mapped to sites. Each site has its own aliased directory within the S3 bucket. This allows one site to host multiple domains. This is useful if you have multiple domains in the same niche space and want to use the same content for all.
Cloudfront
There’s no set monthly costs for a Cloudfront and the variable costs for requests and data transfer are the same whether there is one Cloudfront distribution or many. We’ll use one Cloudfront distribution for each site. All the sites’ domains and wildcard hostnames will be in the SSL certificate, and they will all server content from the same subdirectory of the single S3 hosting bucket.
Implementation
I’ll create a set of files and commands at a later step to automate the processes around this. At this point it is important to understand the architecture as it affects how content is deployed to the sites. Now that I know how I want to arrange my content within the S3 bucket I can think about how to get the content there.
But first things first, I need to have content. While I may not know where the content will come from I do know I want to store it in a Git-based CMS. Implementing that will come next. Knowing how the data needs to be arranged in the S3 bucket will make sorting through the various options for the CMS easier.