Barney Parker

Organization Terraform State Buckets

Thu, 23 Apr 2026 11:56:39 +0000

Organization Terraform State Buckets

Terraform requires a backend to store state files. These are crucial to allow Terraform to know how to map real-world resources to code definition.

Over the years i’ve seen it done an awful lot of ways, and most of them tend to be less than ideal at some time or another. In this post i’ll share my preferred way of doing this and the rationale behind it.

The Problem

When you initialize a Terraform project, it needs to create the state file. When you;re first learning and haven’t provided a backend config, Terraform will just create a local file called terraform.tfstate in the same directory as your code. This is fine for learning, but it definitely wont be long until something goes wrong. either you accidentally delete the file and terraform is trying to depoy resources that conflict with things that already exist.

Often the next problem is multiple people start working on the code. Its just a text file and so teams sometimes think storing the statefile in Git might be a good idea, except that it isn’t. Terraform state files can contain sensitive information, and they also change frequently, which can lead to merge conflicts and other issues when stored in version control.

Thats when teams learn about remote back ends.

You can have a few different backend types, but i work almost exclusively with AWS, so the natural choice is S3. Create a bucket in your account, and then tell terraform to use it as a backend. For a lot of cases, this is fine, but you have to remember to configure everything correctly, for example make sure its not public, and make sure versioning is enabled. Its not a huge issue, but its one more thing to remember, and its something sat right in the account that should only be hosting the application.

When things get a little bigger, you create an AWS Organization. now you have multiple accounts, each with a state bucket, and you have to remember to configure each one correctly. Its not hard to create a simple CloudFormation StackSet that will create this in an account by default.

But I try my absolute best to stay away from CloudFormation unless i have no other choice. There is also the posibility an account wont need a state bucket, so now i need exclusion logic in my stack set and now we’re going down another messy, unmaintainable road….

The Solution

In a well built organization you essentially divide all accounts into a small number top-level Organization Units (OUs). You might have Core for centralized components, Shared for shared services, Workloads for applications and so on.

I tend to create an account that contains deployment related components, so i imaginatively call it Deployment. In here I create the One True State Bucket. This bucket is used by all accounts in the organization to store their Terraform state files. This way, I only have to configure one bucket, and I can ensure that it is configured correctly with versioning and encryption.

But we hit on a small problem. If all accounts are using the same bucket, how do we prevent conflicts? The answer is to use a bucket policy that controls how the bucket can be accessed.

The Rules

Org Only Access

The first rule is that only Principals from our organization can access the bucket. This is done by starting with a deny statement that denies all actions where the Principal (the requestor) has the Org ID we expect. For this we use a Global Condition Context Key: aws:PrincipalOrgID

This gives us:

{
"Sid": "DenyNonOrgPrincipals",
"Effect": "Deny",
"Principal": "*",
"Action": "*",
"Resource": ["arn:aws:s3:::YOUR-BUCKET-NAME", "arn:aws:s3:::YOUR-BUCKET-NAME/*"],
"Condition": {
"StringNotEquals": {
"aws:PrincipalOrgID": "o-1234567890"
}
}
}

Deployment Role Access Only

Next, we dont want just anyone to access this. You only want your deployment roles to be able to access this bucket. You are only allowing CI/CD to deploy aren’t you????

{
"Sid": "DenyNonDeploymentRoles",
"Effect": "Deny",
"Principal": "*",
"Action": ["s3:*"],
"Resource": ["arn:aws:s3:::YOUR-BUCKET-NAME/*"],
"Condition": {
"StringNotLike": {
"aws:PrincipalARN": "arn:aws:iam::*:role/Deploy-*"
}
}
}

This policy says that only roles that start with Deploy- can access the bucket. This way, we can ensure that only our deployment roles can access the state files, and we can prevent any accidental access from other roles. You can change that role name to whatever you want, but the point is that you should have a naming convention for your deployment roles, and then use that convention to control access to the bucket. Deployment roles are one of the few good use-cases for a CloudFormation StackSet in my opinion, so you can use that to create the roles in each account, and then use this policy to control access to the bucket.

Deployment Roles can Read Anything

Terraform has a concent of the remote-state block. This lets you read output from one stack and use it in another and its incredibly useful. For example, you might have a network stack that creates your VPC and subnets, and then you want to use the output of that stack in your application stack to create your EC2 instances. With the remote-state block, you can read the output of the network stack and use it in your application stack without having to hardcode any values.

Because of this, we want to be able to read from any state file, but again, only if its a depoyment role, so we can add this statement:

{
"Sid": "AllowDeploymentRolesRead",
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:GetObject"],
"Resource": ["arn:aws:s3:::YOUR-BUCKET-NAME/*"],
"Condition": {
"StringLike": {
"aws:PrincipalARN": "arn:aws:iam::*:role/Deploy-*"
}
}
}

This just ensures GetObject is allowed for deployment roles, but no other actions are allowed. This way, we can ensure that our deployment roles can read from any state file, but they cant write to any state file that they dont own.

Formatted Keys for Writing State

This final statement lets our roles write to the bucket, but to ensure one team cant accidentally over-write another teams state, we add some restrictions.

The Key must identify the account that is trying to write to the state file. We can get this from the IAM variable aws:PrincipalAccount.
The key must end with .tfstate to try to enforce teams to use the standard naming convention.
The key must contain the project name - sometimes there are multiple projects or stacks deployed to one account
It must identify the environment, e.g. Dev, Test, Prod etc.

So our key must look like:

/<account id>/<project name>/<dev|test|prod>/<stack name>.tfstate

This leaves us with this policy:

{
"Sid": "AllowDeploymentRolesWrite",
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:PutObject"],
"Resource": [
"arn:aws:s3:::YOUR-BUCKET-NAME/${aws:PrincipalAccount}/*/dev/*.tfstate",
"arn:aws:s3:::YOUR-BUCKET-NAME/${aws:PrincipalAccount}/*/test/*.tfstate",
"arn:aws:s3:::YOUR-BUCKET-NAME/${aws:PrincipalAccount}/*/prod/*.tfstate"
],
"Condition": {
"StringLike": {
"aws:PrincipalARN": "arn:aws:iam::*:role/Deploy-*"
}
}
}

The Full Policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EnforceOrgOwnership",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": ["arn:aws:s3:::YOUR-BUCKET-NAME", "arn:aws:s3:::YOUR-BUCKET-NAME/*"],
"Condition": {
"StringNotEquals": {
"aws:PrincipalOrgID": "o-xxxxxxxxxx"
}
}
},
{
"Sid": "RestrictToDeployRoles",
"Effect": "Deny",
"Principal": "*",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*",
"Condition": {
"StringNotLike": {
"aws:PrincipalArn": "arn:aws:iam::*:role/Deploy-*"
}
}
},
{
"Sid": "AllowScopedWriteAccess",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": [
"arn:aws:s3:::YOUR-BUCKET-NAME/${aws:PrincipalAccount}/*/dev/*.tfstate",
"arn:aws:s3:::YOUR-BUCKET-NAME/${aws:PrincipalAccount}/*/test/*.tfstate",
"arn:aws:s3:::YOUR-BUCKET-NAME/${aws:PrincipalAccount}/*/prod/*.tfstate"
],
"Condition": {
"StringLike": {
"aws:PrincipalArn": "arn:aws:iam::*:role/Deploy-*"
}
}
},
{
"Sid": "AllowGlobalReadAccessForDeployRoles",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*",
"Condition": {
"StringLike": {
"aws:PrincipalArn": "arn:aws:iam::*:role/Deploy-*"
}
}
}
]
}

Conclusion

In this post, we have discussed how to organize Terraform state buckets in a way that is secure and maintainable. By using a single bucket for all accounts in the organization and controlling access with bucket policies, we can ensure that our state files are protected and that only authorized roles can access them. This approach simplifies management and reduces the risk of misconfiguration while still allowing for the necessary flexibility in accessing state files across different projects and environments.

This bucket can be backed up, and managed by a limited group of people, and you can be confident that your state files are safe and secure.

WiFi QR Codes

Mon, 20 Apr 2026 08:00:00 +0000

Backoff Simulator

Thu, 16 Apr 2026 08:00:00 +0000

Token Bucket Rate Limiter

Fri, 10 Apr 2026 08:00:00 +0000

Token Bucket Rate Limiter

Standard Rate Limiting

A common requirement for many APIs is to limit the number of requests that can be made per unit of time. This can be to prevent abuse, to ensure fair usage, to differentiate between different tiers of users, or to simply prevent overloading the system.

Rate limiting in itself is a simple approach - - you set a limit of X requests per Y time period, and if a user exceeds that limit, you block them until the time period has passed. This works great when all your users make requests at a consitent frequency, however in most cases users tend to need to make requests in short bursts. Setting a fixed limit in this way means they have a limit when they need o get work done, but most of the time they’re wasting available transactions that they could be using.

Token Buckets

Token buckets are a more flexible approach. They still set an average rate limit, but they allow for limits to be exceeded for a time, as long as the average rate is maintained. This is done by using a “bucket” that fills with “tokens” at a certain rate. Each time a request is made, a token is removed from the bucket. If there are no tokens left, the request is blocked until a token becomes available.

For example, if you have a token bucket that fills at a rate of 1 token per second, and has a capacity of 10 tokens, then you can make up to 10 requests in quick succession, but after that, you will need to wait for the bucket to refill before you can make more requests. This allows for bursts of activity while still enforcing an average rate limit.

Combination Token Bucket Rate Limiting

The problem with a pure token bucket approach is that users can burst, but once their tokens are exceeded, they are limited to an extremely slow rate, e.g. 1 request per second.

A middle ground is to always ensure an absolute minimum number of tokens are available, e.g. 10 tokens, but then also have a token bucket that fills at a certain rate, e.g. 1 token per second with a maximum of e.g. 100 tokens. This allows for bursts of up to 100 requests, but ensures that even if the bucket is empty, users can still make 10 requests per second.

Implementation

To implement this we dont need to keep track of every request, we just need to keep track of the number of tokens in the bucket and the last time it was updated. We’re going to need a database record.

We have to consider that a new user wont have a record, and we dont want a table full of records relating to users that either never made a request, or made a request a long time ago and are no longer active. To solve this, we can use a “lazy” approach to creating records. When a user makes a request, we check if they have a record. If they dont, we create one with the default number of tokens and the current time as the last updated time. If they do have a record, we calculate how many tokens should have been added since the last update, add those to the bucket (up to the maximum), and then check if there are enough tokens to allow the request.

I am going to show this implementation using AWS DynamoDB which has a TTL feature that allows us to automatically delete records after a period we specify. This doesnt cost anything and happens auotmatically.

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, UpdateCommand } from '@aws-sdk/lib-dynamodb';
const client = new DynamoDBClient({});
const doc = DynamoDBDocumentClient.from(client);
async function checkRateLimit(userId: string) {
const nowSec = Date.now() / 1000;
const ONE_DAY_IN_SEC = 24 * 60 * 60;
const ttlValue = Math.floor(nowSec + ONE_DAY_IN_SEC);
const REFILL_RATE = 10; // 10 tokens per second
 const MAX_BURST = 100; // Maximum tokens in the bucket

try {
const result = await doc.send(
new UpdateCommand({
TableName: 'UserRateLimits',
Key: { userId },
// We add 'ttl' to the SET clause
 UpdateExpression: `
 SET lastRefill = :now,
 ttl = :ttl,
 tokens = (
 if_not_exists(tokens, :max) +
 ((:now - if_not_exists(lastRefill, :now)) * :rate)
 ) - :cost
 `,
ConditionExpression: `(if_not_exists(tokens, :max) + ((:now - if_not_exists(lastRefill, :now)) * :rate)) >= 1`,
ExpressionAttributeValues: {
':now': nowSec,
':rate': REFILL_RATE,
':max': MAX_BURST,
':ttl': ttlValue,
},
ReturnValues: 'UPDATED_NEW',
}),
);
return { allowed: true, remaining: result.Attributes?.tokens };
} catch (err: any) {
if (err.name === 'ConditionalCheckFailedException') {
return { allowed: false, message: 'Rate limit exceeded' };
}
throw err;
}
}

When a request comes in from a user we can call checkRateLimit(userId) to check if they are allowed to make the request. If they are allowed, we can proceed with processing the request. If they are not allowed, we can return a response indicating that they have exceeded their rate limit.

Using an Update statement with a ConditionExpression allows us to atomically check the number of tokens and update the record in a single operation, which is important for ensuring that we don’t have race conditions where multiple requests from the same user could potentially allow them to exceed their rate limit.

We use if_not_exists to handle the case where a user does not have a record yet. This allows us to create a new record with the default number of tokens and the current time as the last updated time when they make their first request.

Active User Example

An active user will already have a record in the database. When they make a request, we will calculate how many tokens to add based on the time since the last update, add those to the bucket (up to the maximum), and then check if there are enough tokens to allow the request. If there are enough tokens, we will extract one token and update the record with the new number of tokens, the current time as the last updated time, and set the TTL to be 24 hours from now.

The record will look something like this:

{
"userId": "user123",
"tokens": 95,
"lastRefill": 1620000000,
"ttl": 1620086400
}

If a new request comes in for user123, and the timestamp is 1620000005, we would calculate the number of tokens to add as (nowSec - lastRefill) * REFILL_RATE which with values is:(1620000005 - 1620000000) * 10 = 50. Given that we already had 95 tokens we now have 145 tokens, however, we have a maximum burst of 100 tokens. We then extract one token giving us 99 tokens remaining. We also update the lastRefill to the current time and set the ttl to be 24 hours from now.

We will essentially attempt to create a new record that looks like this:

{
"userId": "user123",
"tokens": 99,
"lastRefill": 1620000005,
"ttl": 1620086405
}

New or Inactive User Example

An Inactive or New user will not have a record in the database. When they make a request, we will use if_not_exists to create a new record with the default number of tokens (which is the maximum burst) and the current time as the last updated time. We will then extract one token for the request and set the TTL to be 24 hours from now.

We will default them to the maximum burst of tokens, which in this case is 100 tokens. We will then extract one token for the request, giving them 99 tokens remaining. We will also set the lastRefill to the current time and set the ttl to be 24 hours from now.

The record will look something like this:

{
"userId": "User123",
"tokens": 99,
"lastRefill": 1620000005,
"ttl": 1620086405
}

Alternative Options

In our example we automatically give a user the maximum burstable amount of tokens. This may not be desirable in all cases. It may be that we have an initial balance. We can solve this with this change:

.
.
.
const INIITAL_BALANCE = 50; // Initial tokens for new users
 .
.
.
ConditionExpression: `(if_not_exists(tokens, :initial) + ((:now - if_not_exists(lastRefill, :now)) * :rate)) >= 1`,
ExpressionAttributeValues: {
.
.
.
':initial': INIITAL_BALANCE,
.
.
.
},

This way a new record gets created with an initial balance of 50 tokens, and then refills at the same rate as before. This allows us to control the initial burst that a new user can have, while still allowing them to refill their tokens over time.

Real World Examples

This algorithm is used in the AWS EC2 service for burstable instances. These instances have a baseline performance level, but can burst above that level for short periods of time. The token bucket algorithm is used to determine how much burst capacity an instance has available at any given time, and to ensure that it does not exceed its limits. If you launch a T series instance, you get a certain number of CPU credits that allow you to burst above the baseline performance level. As you use those credits, they are consumed, and if you run out of credits, your instance will be limited to the baseline performance level until you earn more credits over time. This can be seen in a metric called CPUCreditBalance which shows how many credits you have remaining. If you have a high CPU usage and a low credit balance, you may see a drop in performance as your instance is limited to the baseline level until you earn more credits.

Sunlight Lamp

Wed, 01 Apr 2026 08:00:00 +0000

Introduction

Twice a year it happens. Daylight savings puts the time either one hour ahead, or one hour back. I have a stressfull next week trying to remember how to set the time on my sunlight alarm!

I find it really hard to wake up in the dark. Many years ago my wife bought me a sunlight alarm and it changed my life! The problem is, i’ve long lost the instructions and it doesnt have anything on it to say what it is.

To make matters worse, this is one of those things that for some reason i just cannot seem to commit to memory! I finally decided that i needed to spend some time working out exactly how to do everything and write myself some more permanent instructions and commit them to Git so i can modify them if i am wrong, and witha bit of luck never actually lose them!

The Clock

The clock does a few different things. The most important part is the clock. Its a fairly basic digital clock in a dim orange. On the right-hand edge there is a button marked Dimmer | Time Set which cycles between bright, dim, off

The clock cannot be set while the alarm light is on.

The clock can be set when the alarm light is off by holding Dimmer | Time Set for a few seconds. It then works through the following settings which can be adjusted by pressing the -, OK, + buttons on the edge at the 11:00 position:

24h | 12h
Hour
Minute

The Alarms

The clock supports two alarms which can be turned on and off individually by pressing the 🔔1 or 🔔2 buttons on the edge at the 9:00 an 10:00 positions. An alarm is active (i.e. not disabled) when their icon shows to the left of the clock

The alarms have a start time. The light will begin at the lowest brightness level and will increase to the Max Brightness setting over the course of the Light Time setting.

Alarms can be snoozed by pressing the 💤 button on the edge at the 00:00 position.

The alrams can be set by holding the chosen alarm button for a few seconds. As with the clock, the following settings can be adjusted by pressing the -, OK, + buttons on the edge at the 11:00 position:

Hour
Minute
Sound ♪:
- Off
- S-01 - Bird Song
- S-02 - Waves
- S-03 - Beeping Alarm
- S-04 - High Piano & Birds
- S-05 - Low Piano
- S-06 - Synth music
- S-07 - Wind Chimes
- FM Radio - Last chosen station (See Radio)[## Radio]
Volume (If Sound not set to Off):
- U-1 - Very Quiet
- U-20 - Very Load
Light Time - Time the light should be on for from the start time set previously
Max Brightness
- L-1 barely visible
- L-20 is extremely bright

Note: In testing, the sound didnt seem to happen? really not sure how that works, but i dont use it, so don’t really care too much

Sleep

There is a Sleep setting which plays sounds that gradually get quieter, and can

S-01 - gently “aaah” sounds
S-02 - birds
S-03 - Waves
Radio - last station set

You can set a volume, time and colour, but again, I dont use this so not really sure.

Radio

Ah, who cares. It only does FM and there are only a few FM stations left. If there is a zombie apocalypse, i’ll work it out then, assuming there is power……

Storage First Design

Mon, 30 Mar 2026 17:00:00 +0000

What is Storage First Design?

Storage-first is a method of ensuring resilience and scalability in a system. Its a way to ensure a system never loses data, and can always recover from failure. It’s a way to make sure that your system can handle any amount of data, and give itself breathing room to scale up when needed.

It sounds like a pretty basic idea, but its actually a really powerful way to design systems. By making sure that your system is designed around storage, you can ensure that it can handle any amount of data, and that it can recover from failure without losing anything. It changes a few common approaches to the way a system works, but in such a small way its often overlooked. It can be the difference between a system that can handle a few hundred users, and one that can handle millions.

Storage vs Process First

Lets think about a theoretical system with and without storage first.

The system is a payment step in an ecommerce system. At the point this service is invoked, the user has already added items to their cart, and is ready to check out. The system is responsible for taking the payment details, processing the payment, and then returning a response to the user.

Process First Design

In a Process-First approach, we make one call to our back-end and attempt to get a reponse to our user.

The browser makes an HTTP POST request to /process-payment. The service takes the payment details, processes the payment, and then returns a response to the user. If the payment is successful, it returns a success response. If the payment fails, it returns a failure response.

The problem is we dont own the actual card processing step. We have to call out to a third party service to process the payment. If that service is down, or if there is a network issue, then our service will fail. The user will get an error message, and they will have to try again later. We might even lose the payment details if we dont have a way to store them.

Storage-First Design

With storage first design, we would do this in two steps. The browser makes an HTTP POST request to /process-payment. The service takes the payment details, and stores them in a database. It then returns a 202 (Accepted) response with a unique ID for the payment. Ideally the unique ID is generated as some kind of hash of the transaction details. This makes idempotency much easier. If an additional request comes in and has the same hash, we can just return the same response without having to worry about duplicate payments.

Then we have a separate process that runs in the background, and takes the payment details from the database, and processes the payment. If the payment is successful, it updates the database with the success status. If the payment fails, it updates the database with the failure status.

The user can then check the status of their payment by making a GET request to /payment-status/{id}. We can have any number of statuses depending on the actual process, but the user will see essentially “Pending”, “Success” or “Failed”. If the user gets a “Pending” response, they can check back later to see if the payment was successful or not. If the user gets a “Failed” response, they can try again without having to worry about duplicate payments.

On the back end we can have a scalable process that takes not-started payments from the database, and processes them. If the payment processor is down, or if there is a network issue, then the payment will just stay in the database until it can be processed. We can also have a retry mechanism in place to handle transient failures. If our whole system crashes, we can just restart it, and it will pick up where it left off without losing any data.

The only additional complexity is that we now have a separate process that runs in the background to process the payments, and the front end needs to poll for the status of the payment. Neither of these are complicated or novel so not something any team should have any concerns implementing.

Conclusion

Storage-First design is a powerful way to design systems that can handle any amount of process requests, and can recover from failure without losing any data. By making sure that your system is designed around storage, you can ensure that it can recover from failure, both ours and someone elses. Its easy to implement, and can be the difference between a system that can handle a few hundred users, and one that can handle millions. Its a simple change to the way we design our systems, but it can have a huge impact on the resilience and scalability of our systems.

Gamification to the Rescue

Mon, 30 Mar 2026 16:30:00 +0000

What is Gamification?

Gamification is a method of improving by adding game-like elements to non-game contexts. It’s a way to make work more fun and engaging, and it can be a powerful tool for improving motivation and productivity.

It sounds compeltely crazy, but it’s actually a really effective way to get people to do things they might not otherwise want to do. It’s all about tapping into our natural desire for competition, achievement, and recognition.

A Story of Gamification

A few years ago I was working at an organization that was cutting edge in its field. So much so that it had to invent much of the technology it used.

If you know something can be done, its really easy to do it, and do it well. If you’re having to feel your way in the dark, then its hard to see the problems until they’re deeply embedded in the system. In our case we were so busy on the “can it even be done” that we weren’t looking at whether we were doing the basics well.

One major area of concern was vulnerabiltiies. We had a lot of them, but we had such a backlog of features, tech debt and bugs, we just never seemed to find time to get to them. We had a lot of tools, but they were all just sitting there, not being used.

We tried running “Vulnerability Sprints” - a week where we would focus on fixing vulnerabilities. It was a good idea, but it just didn’t work. We would get a few done, but then we would get distracted by other things, and the momentum would be lost.

I was running a small SRE team trying to work on making our systems work better with each other. Vulnerabilities were a problem, but they weren’t my problem. I had my own backlog of work to do, and I just didn’t have the time to focus on them.

Then we had a minor situation occur. It was nothing earth shattering, and nothing was at risk, but it was the first time something had gained the visibility that vulnerabilities were a problem we couldnt ignore. We had to do something that not only got on top of the problem, but something that lasted.

My team didnt have much capacity, but I was asked to look into it. I wrote some scripts to pull some stats, and I was able to match them up to various tags on our cloud infrastructure. It was by no means perfect, but it was good enough to get a rough idea of where the vulnerabilities were coming from.

I wrote a simple scheduled script that would pull the stats and push the data into our company reporting tools. I set up a dashboard that would show the number of vulnerabilities, and how they were trending over time. I also set up some alerts that would notify us when we had a certain number of vulnerabilities.

And the tumbleweeds kept on tumbling. No one cared. We had the data, we had the tools, but no one was using them.

Instead I madified the script to generate a set of simple HTML charts. I used that to create a set of (worst case first) leaderboards. Essentially the same data sliced and diced in different ways. I didn’t know what was going to work, so I made a few that seemed like they might be useful. I had the script send the report the the company-wide email distribution list.

On day one we had:

Critical: 6237
High: 64598

The email went out at 3am, and by the time i logged in at 8:30 I had a few Slack messages from people asking about the report.

By the end of the day I had more people talking, and some genuinely angry that I was attempting to make them look bad.

My response to everyone was the same - “I’m not trying to make anyone look bad, i’m just making sure we all know where we stand”

over the next few days I had engineers reaching out asking where i got the data from, what they could do to fix problems etc. It was a little overwhelming. By the end of the week i was regretting it, but the CTO reached out and couldn’t believe what a huge reaction there had been. Despite various attempts in the past, this was the biggest noise it had ever created.

Among the various conversation my team identified a few key areas that were causing the most problems. The biggest being Container repositories. Teams created images, but never cleaned up old ones. The vulnerability scanners however were still scanning them, and they were still showing up in the reports. We had a lot of old images that were just sitting there, and they were causing a lot of noise. Engineers complained that they weren’t a risk so shouldnt be counted, but we had to draw the line somewhere so i pushed back.

I did however keep telling people that I still wasn’t telling them to do anything they didnt want to do.

I spent the first couple of days the next week writing a tool that scanned a repository, looked at all the places the images could be used, and then deleted images that weren’t needed any more. It could be run in dry-run mode, and it could be set on a schedule. By the time the third email went out, a few teams had implemented it in dry-run mode, and by the time the fourth email went out, a few teams had implemented it in production.

Eventully we replaced my nieve script with a better tool that captured more data and was more accurate, but by the time I stopped sending my report emails we had:

Critical: 298
High: 12417

Those numbers weren’t perfect, but to put it into context we never told anyone to deal with vulnerabilities, we just published the top 10 worst offenders each week. We reduced cricital vulnerabilities to just 4.7% of the original Criticals, and 19.2% of the original Highs.

The Power of Gamification

The key was that we never told anyone to do anything, and we didnt ask anyone to de-prioritize work they were already planning.

We, as a central team provided tools to do a lot of the heavy lifting, but we didnt force anyone to use them.

We just made sure everyone knew where they stood, and let them decide what to do about it. We let them decide how to prioritize it, and we let them decide how to fix it.

After the Game

Once things were at a manageable level we introduced a new CI/CD module that blocked deployments if they have any critical vulnerabilities at all, and if they have more than 10 high vulnerabilities. We also helped teams set up Dependabot to keep their dependencies up to date. This stopped the flow of dependencies into production, but if we had done this on day one the whole company would have crumbled trying to deal with it. Instead we got on top of the problem, and then put in guard-rails to stop it from happening again.

It may have taken time, but it was a sustainable solution that lasted. We didnt just fix the problem, we fixed the problem and then stopped it from happening again.

Laziness Engineering

Mon, 30 Mar 2026 15:00:00 +0000

What is Laziness Engineering?

Lets face it - if you can avoid doing it, you would! Do what I hear you ask? well, anything really.

More specifically I’m talking about building and running large-scale software systems. There are a lot of things that we do in software development that are just plain hard work, and if we can avoid doing them, then we should!

The good news is that you dont have to work hard, but you do have to work hard to be lazy.

The Benefits of Laziness Engineering

There are lots of Laziness Engineering approaches, but we call them “productivity” tools and processes, because you dont get a pay rise for being lazy - but you should!

The less you have to do, the more you can do. It all comes down to being better value for money as an engineer - after all, we’re not cheap resources.

I love CI/CD pipelines. When I started out (before we even called it “engineering”), we passed zip files around on floppy disks. You had to remember to run tests, build, and upload everything by hand. Inevitably, someone forgot a step, and production broke. It was chaos.

Someone invented Make, which meant I only had to figure it out once, put it in a Makefile, and then just run make build. Suddenly, the build was one command—and I didn’t have to think about it.

Makefiles are awesome, and definitely still have a big place, but things got more complicated (as they always do).

Enter CI/CD pipelines: now you can hand off everything—linting, static analysis, SBOMs, docs, security scans, dependency updates, code formatting, coverage, test reporting, and more. All automated, all the time, on every commit or pull request.

Something broken? Let the pipeline tell you. Need to keep dependencies fresh? Let Dependabot handle it, and let your pipeline do the rest.

Of course, you still have to write good tests. If you don’t, the pipeline will happily deploy your broken code. But that’s still better than manually checking every build. And with the cloud, you can spin up ephemeral environments for every branch—test everything, auto-merge if it passes, and if not, the pipeline tells you exactly what failed.

If something slips through and gets deployed, that’s on you. But now you know what broke—so write a test for it! Over time, your test suite gets stronger, catches more edge cases, and you can trust what you ship.

CI/CD is great, but it can be a bit slow. That’s where Git hooks come in. Pre-commit hooks let you run tools before you even make a commit, catching dumb mistakes before they hit the pipeline.

Take it to the extreme: if your tests and pipeline are solid, why wait for someone to approve your code? Merge it yourself! If it breaks, fix it. If it works, ship it faster.

And now, with AI agents, you can automate even more!

It sounds too good to be true, but if you repeat the process enough thats exactly where you end up.

CI/CD is just one example, but the principle applies to everything. You can automate, delegate or avoid pretty much your whole job. Whats more, your team, your boss, your family, and you bank balance will love you for it!

Be lazy and make the tools do the work for you. Focus on the things that really matter. Things like Coffee, podcasts, music (you could even play that guitar you have to look cool when on video calls….). You can even go outside and touch grass.

Telemetry, not Logs & Metrics

Mon, 30 Mar 2026 10:00:00 +0000

Whats wrong with Logs and Metrics?

For decades we’ve worked iwht logs and metrics.

Our software developers have spewed out random pieces of text - some useful, some not so useful, and we’ve collected that information. They handily categorized it into levels, i.e. informational, warnings, errors etc.

At the same time our infra has spewed out numbers that ean something. usually things like how much is the cpu doing, how much disk space do we have left. how many requests are we getting.

Over time we got al ittle clever and started to include the magical Traces! These let us see a bit more detail into an operation, and how long function calls took.

They’ve served us well. So well that we gave them a grand title: The Three Pillars.

The thing is though, if we’re honest, they’re a bit shit really…..

If you know where to look, and what to look at, and you can pattern match across millions of log lines, and you squnit a little bit and tilt your head, sometimes you can see…… something….

Metrics make some pretty charts though - and thats very satisfying….

Except, they don’t tell you why something is happening. They dont tell you anything in themselves, they just show you that something is happening. its on you to decide what any of that means. In some ways its a little like Astrology.

Not very 21st century tech…..

What’s better

Charity Majors has written, spoken and (correctly) ranted about observaility. And she has a point.

What we really need is Telemetry.

When I think about Telemetry i think about Formula-1 cars, Rockets.

I dont remember the last time i saw people in the NASA control room grep’ing logs (although i’m sure it does happen!). The same with the track-side team at a Formula-1 race.

Telemetry is about collecting information that lets us understand whats happening, even when we’re not there. It lets us understand the state of the system, and how it got there. It lets us understand why something is happening, not just that something is happening.

The funny thing is, its actually not that hard to do, but it does require a slight change of mind-set.

The Properties of Telemetry

Telemetry becomes useful when its consistent, and when it has context. We call this High Cardinality.

Logs have a problem in that a single system may be concurrently processing millions of requests, and if we want to understand the state of a single request we need to be able to correlate all the logs that relate to that request.

Logs also have the problem that they only contain what the developer thought would be useful - and thats generally based upon their mental model of the system, which may not be accurate.

Logs are also often essentially the debugging method of last resort where we output lots of “We got here” log messages in order to trace whats going on.

Untangling all this can be a nightmare.

Traces make this a bit better, but they still rely on the developer to have instrumented the code in the right way, and to have included the right information in the spans.

Structured Logging

The first step in untangling all of this is to structure our logs into a consistent, machine-readable format.

There are a few formats we could use, but the most common is JSON. This is both human and machine readable.

Canonical Logs

The second step is to output one single log line per “request”. Any job should create a single line of all the information, end-to-end for that job. This stops us having to pull otgether lots of different log lines to understand whats going on.

Dimensionality

The more fields we have in our structured, canonical logs, the more we can slice and dice our data to understand whats going on. Knowledge is power - and this is where the power of telemetry comes from.

Cardinality

For each field of dimensionality, we want to have it as highly cardinal as possible. HTTP verbs are a simple example: they can be GET, POST, PUT, DELETE etc. If we just have a field called “method” with the value “GET”, then we can easily filter our logs to see all the GET requests. UserId is highly cardinal because there are potentially millions of users, and each user has a unique ID.

Correlation Ids

Correlation Ids are a special type of field that we include in our logs to allow us to correlate logs across different systems. For example, if we have a request that goes through multiple microservices, we can include a correlation ID in each log line that relates to that request. This allows us to pull together all the logs that relate to that request, even if they are spread across multiple systems.

What This Gives Us

This allows us to filter our logs to see all the requests from a particular user.

The more cardinal our fields are, the more dimensionality we tend to start adding - one feeds into another. For example we start by including UserID, but a user is part of a specific customer, so we add CustomerID. A customer is part of a specific region, so we add Region. A region is part of a specific data centre, so we add DataCentre. Our customer is on a specific payment plan, so we include PlanID.And so on….

Now we can query our logs to see all the requests from a particular user, or all the requests from a particular customer, or all the requests from a particular region, or all the requests from a particular payment plan. We can also start to correlate this with other information, such as metrics and traces, to get a more complete picture of whats going on.

Dimensions can also be metrics - for example startTime, endTime, duration. Now we can query how may user in the US region took more than 1 second to compelte their request on the Enterprise plan.

Try that with the old school pillars of logs and metrics!

Isnt This Open Telemetry?

Yes and no. OpenTelemetry definitely advocates this approach, but really it just a transport mechanism for getting telemetry data from our systems to our observability platform. It doesn’t tell us how to structure our logs, or what fields to include, or how to correlate them. That’s up to us to figure out.

As an example, a siple http request to an ecommerce system may show:

{
"span_name": "GET /products/8365",
"duration": 836,
"status": 404,
"message": "Product not found"
}

On the other hand, we could also show:

{
"correlation_id": "019d3e53-4f79-715e-b4e3-182a400092ca",
"method": "GET",
"path": "/products/8365",
"timestamp": "2026-03-30T10:03:51.387Z",
"duration": 836,
"status": 404,
"message": "Product not found",
"user": {
"id": "12345",
"customer_id": "67890",
"region": "us-east-1",
"plan_id": "enterprise"
},
"cart": {
"item_count": 3,
"total_value": 123.45
},
"flags": {
"black_friday": true,
"new_ui": false
},
"error": {
"code": "PRODUCT_NOT_FOUND",
"message": "Product not found"
}
}

The same request, but the second example gives us far more information. Did we need all that? maybe not this time, but if we had a problem with our product search, then having all that information would be invaluable in helping us understand what was going on, and how to fix it.

Wont this create a lot of data?

Why yes, yes it will!

but thats the point. The more data we have, the more we can understand whats going on. The more we can understand whats going on, the better we can fix it when things go wrong, and the better we can improve it when things are working well.

Tail Sampling

Tail sampling is a technique for reducing the amount of data we store. By making the storage decision after the request compeltes, based on its outcome we an choose whther to keep it, how long to keep it, and where to keep it:

Keep 100% of requests that result in an error
Keep 75% of requests that take longer than 1 second
Keep 25% of requests that take longer than 500ms
Keep 5% of all other requests

Importantly, because we have a correlation ID, we can keep all the telemetry when a failure occurs, not just the telemetry for one system. This lets us understand not only that a failure occured, but why it occurred and where it occurred, even if that operation in itself wasnt a failure.

The ROI

Implemented properly, the knowledge and understanding of a system that we can gain from telemetry is invaluable.

We can use it for debugging, on-call, understanding features, user behaviour, cost analysis, and so much more. The ROI is huge, and the cost of implementing it is relatively low, especially when compared to the cost of not having it when things go wrong.

Whats even better is we dont have to get it right on day one. We can start small, with just a few fields, and then iterate and improve over time.

Each time there is a problem and there was data missing - add it so we capture it for next time
When a product manager asks a question about user behaviour, add the fields to answer that question for next time

Summary

We’ve got a long way on the three pillars, but its time to move on. Telemetry is the future, and we should be embracing it. It gives us a much richer understanding of our systems, and allows us to fix problems faster, and improve our systems more effectively.

Whats more it helps us get to be lazy. Information is power, and giving it to the people who need it means engineers are freed from trying to wrangle insights from logs and metrics, and can instead focus on building new features and improving the system.

Every engineer should aspire to be lazy, and telemetry is a great way to achieve that.

Priorities in System Design

Mon, 30 Mar 2026 09:00:00 +0000

Introduction

Is any system there are competing proprities that need to be balanced, however not all proirities are created equal!

As an example, you may need to nesure you;re using Brand colors in your UI, but that is clearly less important than ensuring you;re not leaking credit card data.

The Priorities of any System

Like the Asimov’s Three Laws of Robotics, there are certain priorities that should guide the design of any system. Each priority builds upon the previous one, but previous ones always take precedence over the next ones. In other words, you should never sacrifice a higher priority for the sake of a lower priority.

They are also sub-grouped into three categories. Non-negotiable, Negotiable, and Evolutional:

Non-Negotiable Priorities

1. Security

Beyond anything else, a system must be secure. We need to protect our users data, but we also need to ensure nobody can get root access to our servers. This is Priority One for the simple reason that the most secure system is the one that is turned off. Presumably thats not an option, so we need to work backwards from there and ensure we have the smallest possible attack surface, and that the surface we do have is as well designed for security as we are able to make it.

2. Durability

Durability is an interesting word. Its often confused with Availability or Reliability, but its actually a different concept.

Durability is about ensuring that data is not lost, even when something goes wrong. If we still have the data, at least we can try to do the thing again. If we lose it, it’s game over.

This makes it an interesting prioity because it directly influences the design of the system. It affects the order of operations, the way we handle errors, and the way we design our data storage.

3. Correctness

It seems obvious that a system needs to be Correct, but its surpising how often we seem to accept “bugs”. For some reason we seem to have accepted that software is inherently buggy, and that we just need to live with it. This is a failure of our design, not an inevitability of software development.

Correctness is about ensuring that our system does what it is supposed to do, and that it does it correctly. Its obvious that we dont want our Bank to make a mistake and lose our money, but we also dont want our Social Media platform to accidentally post something we didnt intend to post. We need to ensure our customers receive the products they purchased and that they are not overcharged or undercharged. We need to ensure our users can trust our system to do what it is supposed to do, and to do it correctly.

Negotiable Priorities

The following 3 Priorities are negotiable. You can’t sacrifice the non-negotiable priorities for the sake of these, but you can make trade-offs between them. Your system sits at a point somewhere in the triange between these three priorities, and you need to decide where that point is based on your users needs and your business goals.

An obvious example is adding an extra 9 to your availability may reduce you latency and increase you costs. If you dont have the money, or if your users dont care about that extra 9, then you shouldnt add it. If you do have the money, and your users do care about that extra 9, then you should add it. But you shouldnt add it just because it sounds good, or because your boss wants it, or because your competitors have it. You should add it because it is the right thing to do for your users and your business.

4. Availability

At last, we get to Availability! This is often the first thing that gets thought about when designing a system - mostly because its a solved problem. Modern technology means we can achieve very high levels of availability with relative ease, and its common-place to over-spec a system to ensure it is always available.

Using the SRE model though, we shouldn’t just throw money at the problem (more on this in the Cost section), we should actively design our system to “Just available enough”. We shouldn’t be unreasonable and aim for 100% availability just because it feels right, or makes our boss feel warm and fuzzy inside. This is the first in a subsection of priorities

5. Latency / Performance

I’ve given this Priority two names because they’re kind of interchangeable. Essentially we need answers, and we need them ~now!~ as soon as is pratically possible within the constraints of our system.

Its technically a lower priority that availablity because i’d rather the process compeltes late than not even being able to start it in the first place.

6. Cost

Cost is the most easy to quanitify, and the most likely to be lied about! Companies often say thhey want the best rexperience for their users, and hey, look, it’s priority 6 - it barely matters, right?

The reality is that cost is often the most important priority, we just dont like to talk about that. Really, this is the one that underlines all of the rest:

We want the safest, most durable, most correct, most available, and best performing system that we can afford.

The interesting thing is, if 1-5 are done really well, costs tend to lower than if they are done poorly. It also tends to be the case that if 1-5 are done really well, users are more likely to pay for the system, which means we have more money to spend on the system, which means we can do 1-5 even better. Its a virtuous cycle.

Evolutional Priorities

These are priorities that aren’t necessary for the system to function, but they are important for the long-term success of the system. They are about ensuring the system can evolve over time, and that it can adapt to changing requirements and user needs.

7. Self-Sustaining

There are other evolutional priorities we could add here, but I think this one is the most important. A system should be designed to be self-sustaining, meaning it can operate with minimal human intervention. In the event of a problem, the system should be able to recover itself, or at least provide enough information for a human to intervene and fix the problem.

This used to be extremely hard, but with modern infrastructure and automation tools, it’s becoming increasingly possible to design systems that are self-sustaining.

Cloud systems emit events when all sorts of things happen (and we should emit custom events also). We can use those to trigger corrective actions. In the worst case we can alert a human who can A) fix, and then B) automate the fix for next time.

Conclusion

When designing a system, it’s important to consider the priorities that should guide our design decisions.

I’ve spent a lot of time on-call, and whether the problem is tiny or huge, waking up at 3am and trying to deal with a broken system isn’t fun. If we design our systems with these priorities in mind, we can reduce the likelihood of waking up at 3am, and we can make it easier to deal with the problems when they do happen.

Random Notes on System Design

Mon, 22 Dec 2025 11:05:00 +0000

Introduction

Over the years I have picked up various insights and best practices related to system design. This post is a collection of random notes that I find useful when thinking about designing scalable and maintainable systems.

They’re not particularly organized, however they come up time and again in my mind when considering different aspects of system design. Working with teams can sometimes be difficult becuase different people have different mental models of how systems should be designed. I hope that by sharing these notes, I can help align some of those mental models.

One-Liners

Simple is generally the most resilient

The point of a System

The point of any system, whether it is digital or physical, is to fulfil a users need. A system is not a piece of art, it is not a monument to human ingenuity, it is not a testament to the skill of its creators. A system exists solely to serve its users.

A system that does not serve its users is a failure, regardless of how well it is designed or how impressive its architecture may be.

Evolution as an Agile process

Many companies say they are “Agile”, but the reality is they often are not. Using daily stand-ups, two-week sprints, and other Agile rituals does not make a company Agile. Agile coaches or Scrum Masters are not required, and often appear to sustain process over evolution. Somehow Agile got coopted into a set of rituals and ceremonies that have little to do with the original intent of Agile.

Agile is the embodyment of continuous evolution. A system should be designed to evolve over time, to adapt to changing requirements, and to improve based on user feedback.

In Darwins Theory of Evolution, the most important factor for survival is the ability to adapt. A System only needs to be “good enough” for its current environment, and to excel in that environment it needs to be “a little bit better than the competition”.

By contrast I often see companies trying to design systems that are “perfect” from the start, or that are “future-proof”. This is a flawed approach, as it often leads to over-engineering, increased complexity, and a lack of adaptability. Essentially this is Waterfall thinking implemented using Agile rituals to tell ourselves we are Agile.

A perfect process doesnt exist (after all, there is no perfect animal or plant in nature), however Agile gives us an evolutionary cycle:

Build -> Measure -> Learn -> Adapt -> Repeat

Every iteration should end with a working system that we hope is better than the previous iteration. By learning from this system, we can adapt our design and improve it over time.

Internally we should be using Metrics and Observability to measure how well our system is performing, and to identify areas for improvement. This data should inform our decisions about how to adapt our system in the next iteration.

Externally we should be talking to users to gatehr feedback on how well our system is meeting their needs. This feedback should also inform our decisions about how to adapt our system in the next iteration.

From this we can plan the next iteration. Sometimes that will be to add or extend functionality, but often it should be to wind back functionality because it was an evolutionary mis-step. Unfortunately thats often viewed as failure, when in reality its just part of the evolutionary process.

To Quote Einstein:

“A man who has never failed has never tried”

Failure should be more common that success in our evolutionary process, yet we often treat it as a taboo subject. We should embrace failure as a learning opportunity, and use it to inform our future decisions.

Build for 3am, when it’s all gone to hell….

No matter what we do, something will always go wrong. Servers will crash, networks will go down, databases will become corrupted.

When designing a system, we should always be thinking about how it will behave in the worst-case scenario.

Computers dont care about data. People do.

Many AWS services for example can have randomly generated names. For example if you create an S3 bucket but do not provide a name, the S3 service will generate one for you - and it’s not pretty. Thats fine because our IaC got that name and passed it to our Lambda functions and an env var, but what happens when we need to debug something at 3am because the system is down? We have no idea which bucket is which, and we have no idea which Lambda function is which. We are going to be in a world of pain. Its not impossible to debug, but its going to take a lot longer than if we had just used sensible names in the first place.

When designing a system, we should always be thinking about how it will behave when things go wrong. We should be thinking about how we will debug it, how we will recover from failures, and how we will ensure that our users are not impacted.

Incidents

Analysis

Triage is fast, but shallow. You’re analyzing to pin-point the problem so you can mitigate it quickly. You are not trying to find the root cause.
Post-mortems are slow, but deep. You’re analyzing to find the root cause so you can prevent it from happening again. You are not trying to mitigate the problem quickly.

Log Files

Log files are the API your operators use to debug an issue
Log files were not intended for this specific issue
OTel is making this better, but we still have a long way to go and they are often sampled to save money
Design your log files with the assumption that they will be used to debug issues in the future
Ensure your log files have sufficient context to understand what was happening at the time of the incident (Otel helps a lot here with tracing)

AI can be relly good at finding patterns and sequence of event in large, incoherent log files

Consider using AI to help analyze log files during incidents
AI might not be the smartest operator, but it doesnt get tired, and it doesn’t miss tiny changes between two nearly identical lines

A Product Hierarchy of Need

Tue, 19 Dec 2023 17:56:39 +0000

Introduction

Developing and running software systems is a complicated balancing act between keeping the lights on, satisfying various competing demands from the business and its customers, and staying ahead of the competition. Attempting to do incorporate all of these needs while also working out the “right” thing to do next is near impossible.

The SRE approach prioritises operational running of a system above all else, but even that is too general to help us. Instead we need to define a language of work types, and agree how we will split our time based on a worst case situation. As situations improve and we clear our backlogs of higher priority work, we can spread the remaining time in to higher-level work types.

Classifying work with a hierarchical “type” and defining the priority of each work type can help us decide where to focus our energy, and also to help justify why some work was chosen over others.

TL;DR

If we classify work in our backlogs according to its “type”, and we prioritise work types based on importance to running stable, reliable, compliant systems we can free ourselves to do “higher level” work without concern for the behaviour of our products because our foundations will be solid enough to support us without the need for constant attention.

While we draw the hierarchy of important as a triangle, with foundational work being the largest and lowest part of the triangle, we intend to “invert” the triangle by ensuring the foundations are so well taken care of that we spend the least time. We can only get there by consistently prioritising the foundations beyond all other work.

Human Hierarchy of Needs

Maslow’s hierarchy of needs is a theory from psychology that gives a five-tier model of human needs, depicted as a pyramid. The segments location in the pyramid shows priority; lower levels have a higher priority overriding lower priority needs at higher layer.

The idea is that base needs such as food and water are more important than higher level needs such as education (self-improvement). If you have no food or water a person is unlikely to be concerned with whether they are going to pass an exam.

Product Hierarchy of needs

Technical products are very similar to a person. While they don’t need food and water, they do need basic working hardware to run on. If a system isn’t even able to stay online, its owners are unlikely to care about the outcome from the latest spike end engineer has spent investigating a potential feature.

While the hierarchy of human needs is relatively simple to understand, its not immediately clear what the hierarchy should be for a product, yet its not particularly hard to find an obvious home for the majority of things product teams regularly work on.

Product Self-Actualisation

For a human, self-actualisation refers to becoming better than ones current self. For a product this translates fairly simply in to discovery work or in Agile terms, a Spike.

A Spike is used to try out some ideas. They don’t make it to production in themselves but they do help us learn how an idea might work, what the constraints are, think about the things we might have otherwise not have fully understood. In this sense a Spike is the betterment of the product and should sit at the top level. If there are other problems with our product we don’t want to spend time on blue-sky thinking when customers are leaving and not paying our bills.

Product Esteem

Esteem for a human is where, in the developed world at least, we spend a lot of our time. Doing our jobs to make our products better, implementing new features that we know our customers want.

It is enjoyable to add new features, or to improve old ones, but again its not particularly important if our service wont stay online. We can only afford to do this if our lower level needs are already fulfilled and we have spare capacity to do this work.

Product Love and Belonging

Our products don’t work in a vacuum. They have customers (whether that it a human or another system doesn’t matter) who we need to love our product. They need to be able to rely on our product to work as expected.

A prime candidate for this category is bug fixing. The product isn’t “broken” as such, but it may have unexpected behaviour in certain circumstances, or may perform poorly. These are the thing that drive customers away towards our competitors because they get irritated by the problems the product causes them.

Fixing bugs is more important than adding new features because we need to keep current customers happy, but as with the other higher categories, its not as important as a system that can’t even stay up to serve our customers in the first place.

Product Safety Needs

There are some things related to a product that draw a line between prototype systems that we wouldn’t trust, and professional systems that we would.

Product Safety covers things such as data security, authentication and authorisation. We need to be sure that when a user accesses our product they only see data to which they have been given access. We also need to be sure that only the authorised users are allowed access.

While bugs are annoying, its even worse to find out someone else has access to our banking information for example. Personally Identifiable Information has legal requirements to ensure it is handled properly. If we aren’t sure this is being handled well its almost guaranteed that all higher-level work will be stopped until this is resolved.

Product Physiological Needs

Products, like people have physiological requirements. They need to run on infrastructure, and that infrastructure needs to be available for customers to access it.

At this layer we need to include not just the infrastructure itself, but connectivity between components, the ability to detect failures and the ability to resolve problems before a customer notices. Here we include on-call rotations, run-books, monitoring, alerting, dashboards etc.

This is an area we would, in an ideal world not like to think about. a huge reason why we use AWS is because it provides an awful lot out-of-the-box to help us with this layer.

How does this relate to Tech Products?

Agile prescribes working in short sprints, with each sprint delivering a working product.

What Agile doesn’t do however is tell us how to decide what to work on.

Its common to spend time defining work, refining the work to manageable chunks, and estimating the complexity to help us break things down in to chunks we can deliver in a single sprint. We then load up a sprint with enough complexity that we can deliver what we said we would.

In principal that’s fine, except for one small problem - who decide how important a chunk of work is?

The hierarchy of human needs, when drawn as the pyramid above shows priority in therms of vertical location, and importance as width. What it doesn’t show however is how much time is spent on each section.

A caveman might spend most of their time hunting for food and water. They will also spend a reasonable amount of time on ensuring their own personal safety. They want to spend some time with friends and family, ensuring their place in their community. They spend a little time on providing for their community in general to maintain their place in society and if there is any time left over, they might practice their skills to become better.

In the developed world however, the pyramid is turned on its head. I spend very little time on food and water - i turn on a tap and i can drink, and I open the fridge to get food. I own a home with locks on them although I do spend a little time occasionally making sure i repair things, and i mow my garden. I have a family which i spend my spare time with as much as i can. I work every day in a job I enjoy and get paid for doing. It takes up a lot of my time, but i get respect for doing my job. I spend a a good deal of time on reading about systems engineering, AWS services, and gaining certifications etc. Suddenly the base layer things are small, working towards more and more time spent at higher layers.

The hierarchy of needs of a product are identical in many respects. If we have tickets relating to the lowest layer we should prioritise those over all else. The fundamental foundations of our product are the most important thing to work on, because without them everything else crumbles.

If we have less work at the bottom layer to fill our sprint, we can select tasks from the next layer up. As we work harder and harder to remove work at the bottom of the pyramid, we need to spend less and less time on it and can afford to spend our available time on progressively higher layers. Over time, lower layers become less cumbersome, and we can spend more and more of our time working on features, and investigating ideas on how the future might look.

When a problem occurs at a lower layer however, the hierarchy steps in and says it must be dealt with before anything else. We then maintain a solid, healthy, reliable system that our users are happy with, and continue paying for.

Hierarchy of Needs and Incident Management

Given the hierarchy of needs, it becomes relatively clear how we should infer “importance” of a type of work. This extends further in to the territory of incident management!

Incidents are measured as P1/2/3. P1 is necessarily the most important, and potentially damaging to the product. It is therefore reasonable to say that any incident in the Physiology layer should be a P1. Similarly incidents in the Safety layer aren’t normally quite as damaging - incorrect security configuration or a DB authentication problem aren’t a complete outage, but they definitely deserve to be a P2.

Bugs however are pretty annoying. They annoy users, and they might cause some support work to occur. we probably don’t need to get anyone out of bed at 3am though. In that sense we can say P3 incidents live within the Love/Belonging layer.

Beyond P1/2/3 we don’t alert, but given that this is now squarely in to product development and future investigations, that makes sense.

Any work that is needed in the lower three levels has the potential to become an incident if we don’t deal with it.

HoN and OKRs

In [X Company] we use OKRs to steer our products over the medium to long term. OKRs tend to focus on the future - where we want to be in 3 or 6 months.

OKRs contain two specific types of work - work we have thought about, refined, and planned far enough to put them in to a fairly specific OKR. This is where a squad gets to show off to the rest of the business and to its customers. This is the Esteem layer.

Beyond the work we have already planned there is work that the business or our customers have asked for but we can;t necessarily do just yet. We need a spike to evaluate it or to define it more clearly. As we decided earlier, this sits in the Self-Actualisation layer.

We can’t stop developing features, our customers wont be happy!

That is partially true. Its is important to ensure our product is seen to be improving, but stability and reliability is an improvement in itself. There is a good argument that reliability is actually the only feature that all products have, and is therefore the only feature that truly matters.

Self Actualisation: 0%

Esteem: 5%

Belonging: 16.5%

Safety: 33%

Physiological: 45%

This might seem a little extreme, but if there are 45% of complexity tasks available at the lowest level, it should be clear that you really should be prioritising those and not being too concerned about future product development work. You might still need to do some feature work, but its decidedly less important than fixing bugs or ensuring data integrity.

Over time, assuming you’re doing the right work, the number of available tasks at the lower levels should decrease. this frees up more time further up the stack allowing you to spend additional time on more interesting work. Eventually the stack turns upside down, and almost no time will be spent at the base layer, and almost all of your time can be spent on features and really thinking about the future.

Closing Thoughts

Sprints can be organised in many different ways, but consistently prioritising the more fundamental requirements of a product will set things up for success and free up more time for the interesting work we all came here to do. Ignoring it will lead to problems, take time away from delivering what customers want, and will eventually lead to failure of the product

This is true even for a potentially great product.

Appendix A - specific work categorisation

All work should exist here. If a type of work doesn’t appear to be listed, please leave a comment to allow discussion on best placement

Product Self-Actualisation

Spikes

Product Esteem

OKR specified Work

Product Love and Belonging

Bug fixes

Product Safety Needs

Alerting
Dashboards
Vulnerability Management
Cost Awareness

Product Physiology Needs

CI/CD
Metrics
No reliance on other environments
Leverage Managed Services

React Build-less MVP

Tue, 12 Dec 2023 09:48:43 +0000

Going Build-less

I don’t do a huge amount of front end work, but from time to time I do need to put together a small front-end.

Front-end work is some of the most complicated tech work there is. Not only do you need to know HTML, CSS and JavaScript, but you need to know how to configure and use build tools, npm, frameworks, and handle a bunch of security things such as CORS, CSP, and a whole bunch of other odds and ends!

As you might expect, unless you;re doing this stuff daily, its hard to keep it all in your head, so removing any piece makes life much easier.

One area I have found particularly useful is being able to use a framework such as React, but without needing a build step. Doing this is easier than it might appear, and only takes one or two little tricks to make it work.

Removing JSX

The first piece of the puzzle is removing JSX. JSX looks like HTML, but actually gets compiled to a bunch of calls to Reacts createElement function. In turn this builds out the Virtual-DOM, and React then does the diff/update steps etc. Without a build step we need some way to do that in the browser.

Htm is a small library by Jason Miller that takes tagged template literals that look pretty close to JSX, and turns it in to the createElement structure used by React!

There is one minor different in that you need to refer to a component as <${myComponent}> instead of plain <MyComponent>, but otherwise everything behaves the same.

Htm needs to be bound to createElement to make it work so its good to do that in its own little module and export the html function in a file called html.js:

import htm from 'https://esm.sh/htm@3.1.1';
import { createElement } from 'react';
export const html = htm.bind(createElement);

Import Maps

Normally without a build you would have to import React like this:

import { useState } from 'https://esm.sh/react@18.2.0';

This is a bit clunky and cumbersome, so instead we can leverage import maps to make that look a bit nicer.

In the head section of your html document include the following:

<script type="importmap">
{
"imports": {
"react":"https://esm.sh/react@18.2.0"
}
}
</script>

In your JS files you can now do:

import { useState } from 'react';

A Simple App Example

To test all this out we need a simple hello-world example. Create a file called App.js containing the following:

// @ts-ignore
import { html } from 'html';
import { useState } from 'react';
const Counter = () => {
const [count, setCount] = useState(0);
return html`
 <button onClick=${() => setCount(count + 1)}>Click Me [${count}]</button>
 `;
};
export const App = () => html`
 <div>
 <h1>Hello (Build-less) World!</h1>
 <${Counter} />
 </div>
`;

This contains two components, the App component which we will call to start our app, and a local Counter component. You can see on line 18 that we needed to reference the Counter component as a variable rather than an element name, but otherwise it looks exactly like a normal React component.

Tying it all together

As normal, we bring this all together with an index.html document which contains the import map, the root div that will contain the application, and a small inline bootstrap script to kick it all off:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<script type="importmap">
{
"imports": {
"react": "https://esm.sh/react@18.2.0",
"html": "/html.js"
}
}
</script>
<title>Build-less React MVP</title>
</head>
<body>
<div id="root">This will be replaced</div>
<script type="module">
import { html } from 'html';
import { createRoot } from 'https://esm.sh/react-dom@18.2.0';
import { App } from './App.js';
const root = createRoot(document.getElementById('root'));
root.render(html`<${App} />`);
</script>
</body>
</html>

Note: we import react-dom with a full URL since this is the only place we’ll use it, so it isnt necessary to add it to the import map.

As we go though app development, we can add other external scripts to the import map. for example we can import react-router-dom by adding:

{
"imports": {
...
"react-reouter-dom": "https://esm.sh/react-router-dom@6.4.1",
...
}
}

and then :something like

import { html } from 'html'
import { BrowserRouter, Routes, Route } from 'react-router-dom'
import { MyComponent } from '/Components/MyComponent'
import { NotFound } from '/Components/NotFound'
export const App = {} html`
 <${BrowserRouter}>
 <${Routes}>
 <${Route} path="/" element=${html`<${MyComponent} />`} />
 <${Route} path="*" element=${html`<${NotFound} />`} />
 <//>
 <//>
`

Running Locally

As a final touch, all this can be run locally using something like the http-server utility from npm. You’ll need to run it in SSL mode which requires an SSL certificate which can be created with:

openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem

and then start the server with the command:

http-server -S -C cert.pem

Import Maps to simplify Browser Based ESM

Tue, 12 Dec 2023 08:34:47 +0000

Import maps let you use short names when importing packages in the browser rather than using script tags.

In order to mark a script as an import map, the script tag needs a type attribute with a value of importmap.

The contents should be an object with a single attribute of imports which is an object where the keys are the name you will use to refer to a package, and the value is the URL to the package. e.g:

<script type="importmap">
{
"imports": {
"React": "https://esm.sh/react@18.2.0",
"react-dom": "https://esm.sh/react-dom@18.2.0",
"htm": "https://esm.sh/htm@3.1.1"
}
}
</script>

This should be placed in the head section. The parser reads the import map, but doesn’t preload the imports. instead it waits until a package gets imported, and then translates it to its real url for loading:

import htm from 'htm';

Uploading File Trees to S3 with Terraform

Fri, 02 Jul 2021 14:50:30 +0000

Uploading a single file to S3 using Terraform is pretty simple, but sometimes you need to upload a whole folder. If you’re serving the files using S3 as a website, or through CloudFront you also need to make sure you set the correct mime types and eTags. Its actually a whole lot simpler than you might think!

Single File Uploads

A single file is easy enough to upload:

resource "aws_s3_bucket_object" "single_file" {
 bucket = aws_s3_bucket.web.id
 key = "index.hmtl"
 source = "${path.module}/content/index.html"
}

While that will upload a file, it will get a mime type of application/octet-stream which means if you’re trying to serve it through CloudFront it will download rather than loading as the right document type. To set the right mime type, all thats needed is:

resource "aws_s3_bucket_object" "content" {
...
 content_type = "text/html"
}

Often a browser will perform a HEAD request to see if a file has changed since the last time it downloaded it. The easiest way to check is via the etag header which contains a hash of the contents. We can also add that automatically:

resource "aws_s3_bucket_object" "content" {
...
 etag = filemd5("${path.module}/content/index.html")
}

Extending to Multiple Files

Modern versions of terraform let is create multiple resources using the for_each iteration operator. To get a list of the files we want to upload we can use the fileset function which collects a list of files at a path based on a pattern.

fileset takes two parameters, a root path and a pattern to match files against. The pattern field is able to allow for recursively searching a folder tree using the ** globbing operator, so we could for example do something like:

resource "aws_s3_bucket_object" "content" {
 for_each = fileset("../web/build", "**/*.hmtl")
 bucket = aws_s3_bucket.web.id
 key = each.key
 source = "${path.module}/content/${each.key}"
 content_type = "text/html"
 etag = filemd5("${path.module}/content/${each.key}")
}

Which is a pretty simple way of finding all the files ending in .html and adding the to the bucket with the right mime type. Its a bit cumbersome working this way though because now we need to add a new block for JS, another for CSS, one for JPGs and so on…..

Mime-Type Lookups

Using some clever indexing we can save ourselves an awful lot of effort by creating a map of file extensions and the mime type they map to. Inside the for_each loop we can dissect the filename to get the file extension, and then look up what mime type it maps to:

Our map of mime types should look like:

locals {
 mime_types = {
 "css" = "text/css"
 "html" = "text/html"
 "ico" = "image/vnd.microsoft.icon"
 "js" = "application/javascript"
 "json" = "application/json"
 "map" = "application/json"
 "png" = "image/png"
 "svg" = "image/svg+xml"
 "txt" = "text/plain"
}
}

we need to cover every file type we have, and Terraform will fail to plan if a file extension is added that we don’t have a type for, but at least we don’t end up with the wrong mime type!

To do the lookup we need to split the file name and path by the . character, and then get the last element and use that in a lookup against the map. Using a local actually gives us an object rather than a map, but you can convert using the toset function.

resource "aws_s3_bucket_object" "content" {
 for_each = fileset("${path.module}/content", "**/*.*")
 bucket = aws_s3_bucket.web.id
 key = each.key
 source = "${path.module}/content/${each.key}"
 content_type = lookup(tomap(local.mime_types), element(split(".", each.key), length(split(".", each.key)) - 1))
 etag = filemd5("${path.module}/content/${each.key}")
}

Now every file will be found, mapped to a valid mime-type, an etag calculated and then be individually uploaded to S3! Since the etag is set, subsequent runs will only upload files that have changed!

Authenticate Users with Azure OIDC on AWS Application Load Balancers

Tue, 22 Sep 2020 18:36:55 +0000

Securing legacy applications which were deigned for internal use only has become a hot topic in these times of sudden remote working. There isn’t the time or money available to re-architect authentication, and its not always easy to provide remote accessibility to systems when the company doesn’t have the resources to implement it safely.

Thankfully AWS Application Load Balancers provide a really simple method of adding an authentication layer to an application which doesn’t require any application code changes, and can be linked to an auth provider that you already have via OIDC, in this case the extremely common Office 365!

OpenID Connect and AWS Application Load Balancers

OpenID Connect is an identity layer built on top of OAuth2 which can be used natively with AWS ALBs. An ALB requires a listener which has rules on how to forward traffic based on the incomming connection. In the vast majority of cases just a simple default listener rule to “forward” to the application target group is all that is used.

In order to make use of OIDC, all we need to do is add a condition which checks for an OIDC auth cookie, and if its not present it starts the OIDC OAuth2 authentication flow!

Creating an Azure App Registration

We need to create an application registration in Azure Active Directory. This will give us the various URLs and magic values we will need to be able to configure OIDC on our ALB.

Create the App Registration

Under Azure Services click Azure Active Directory

In the left had menu, under Manage click App Registrations

On the top bar, click New registration

Start by entering a name for your application. This helps identify the purpose of this configuration

Under Supported Account Types select the option Accounts in this organizational directory only ( - Single tenant). This will ensure only your companies’ Office 365 users will be able to use our application.

Finally, click the Register button.

Next we’re going to need to collect some information that we need to supply to the ALB.

Create the Client Secret

In the left side menu, click Certificates & secrets

Under Client Secrets click New client secret

Add a description to remind you what this secret is for. The application registration could be used for multiple applications, but each application should have its own secret, so the description will be helpful if the secret ever needs to be removed.

Select an expiry. Most likely Never is the best choice here.

Finally click the Add button

The Client Secrets list will now contain your new secret. Copy the Value field and add it to the following Terraform code:

variable "oidc_client_secret" {
 default = "<paste secret here>"
}

Collect Our Other Information

On the left hand menu again, click Overview

In the Essentials pane, copy the Application (client) ID and paste it in to the following Terraform code:

variable "oidc_client_id" {
 default = "<paste client id here>"
}

Next copy the Directory (tenant) ID and paste in to the following Terraform code:

variable "oidc_tenant_id" {
 default = "<paste tenant id here>"
}

And we have all the information we need to go ahead and implement OIDC in our listener.

Redirect URIs

Part of the OAuth flow requires the load balancer to redirect to the Azure authorization endpoint. We need to configure this in Azure

In the Overview / Essentials block, click the Redirect URIs link

Click the Add Platform button

Click Web

Enter the url to the root of the load balancer and append /oauth2/idpresponse. e.g. if our app is at https://app.exmaple.com this should show https://app.exmaple.com/oauth2/idpresponse

A little further down the page, under Implicit Grants tick the option Access Tokens, and untick ID Tokens

Create our OIDC ALB Listener(s)

For OIDC to work we need to be using HTTPS. It’s unwise to expose an application over HTTP anyway, so lets enforce this by adding a listener with a default action that redirects http traffic to https:

resource "aws_lb_listener" "http" {
 load_balancer_arn = aws_lb.application.arn
 port = "80"
 protocol = "HTTP"
default_action {
 type = "redirect"
redirect {
 port = "443"
 protocol = "HTTPS"
 status_code = "HTTP_301"
}
}
}

Now we need an HTTPS listener

resource "aws_lb_listener" "https" {
 load_balancer_arn = aws_lb.application.arn
 port = "443"
 protocol = "HTTPS"
 ssl_policy = "ELBSecurityPolicy-2016-08"
 certificate_arn = "arn:aws:iam::012345678999:server-certificate/test_cert_abc"
default_action {
 type = "authenticate-oidc"
authenticate_oidc {
 authorization_endpoint = "https://login.microsoftonline.com/${var.oidc_tenant_id}/oauth2/v2.0/authorize"
 client_id = var.oidc_client_id
 client_secret = var.oidc_client_secret
 issuer = "https://login.microsoftonline.com/${var.oidc_tenant_id}/v2.0"
 token_endpoint = "https://login.microsoftonline.com/${var.oidc_tenant_id}/oauth2/v2.0/token"
 user_info_endpoint = "https://graph.microsoft.com/oidc/userinfo"
}
}
default_action {
 type = "forward"
 target_group_arn = aws_lb_target_group.application.arn
}
}

Outbound Traffic ALB

An essential part to add is an Egress rule to the security group attached to the ALB for HTTPS traffic to allow the ALB to contact Azure to confirm tokens:

resource "aws_security_group_rule" "applications_alb_egress_https" {
 from_port = 443
 to_port = 443
 protocol = "tcp"
 cidr_blocks = ["0.0.0.0/0"]
 security_group_id = aws_security_group.app_alb.id
}

Notes

For brevity I have not included the Terraform code for the load balancer or target group, however this is relatively trivial to add.

Once added you should find a couple of extra headers in the requests forwarded to our application:

x-amzn-oidc-accesstoken holds the full JSON Web Token (JWT) x-amzn-oidc-data is the user claims in JWT format x-amzn-oidc-identity contains the subject field (sub) from the user info endpoint

Installing NPM Dependencies with Terraform

Tue, 23 Jun 2020 14:29:42 +0000

Terraform has the capability to create a zip file containing Function code, and for simpler functions that’s fine. For more complex functions it quickly becomes necessary to install dependencies to make avoid the need to write every line of code ourselves.

The problem with Terraform is that it only wants to do things that have a Provider resource. npm installing dependencies needs to be done outside of this, and while we could manually there is no way to guarantee we will remember to do it every time we apply.

In order to keep this inside out Terraform workflow we need to abuse some resources a little.

Triggering Dependency Installation

To convince Terraform to install our dependencies, we need to know

Has package.json changed?
Has package-lock.json changed?
Do we have all the files we expected?

We also need to store something in the state file that we can compare to in the future.

resource "null_resource" "lambda_dependencies" {
provisioner "local-exec" {
 command = "cd ${path.module}/src && npm install"
}
 triggers = {
 index = sha256(file("${path.module}/src/index.js"))
 package = sha256(file("${path.module}/src/package.json"))
 lock = sha256(file("${path.module}/src/package-lock.json"))
 node = sha256(join("",fileset(path.module, "src/**/*.js")))
}
}

a null_resource doesn’t do anything in itself, but it acts as a container that we can use to create one resource based on a set of trigger values.

local-exec allows us to run a system command, which in our case is to move in to the Lambda Function source directory and execute npm install.

We can set triggers based on our requirement, i.e. has index.js, package.json or package-lock.json changed by checking their sha256 signatures. We can also get a list of every .js file in the source directory, join them in to a single string and then get the sha256 of that string.

These trigger values will be stored in the state file, so even if we run this on another machine we’re going to be able to tell if anything doesn’t match between the previous deployment and this one. If something doesn’t match, then the local-exec command will be executed, and our dependencies installed!

Building the Bundle

So far so good, but we need to enforce Terraform ordering in the dependency graph. If we just add an archive_file data resource there’s no guarantee it will wait for the dependencies to be installed before creating the bundle.

To fix this we add an intermediate resource called a null_data_source. These resources take a bunch of inputs which can be computed form the outputs of other resources or variables. Until all of the inputs are satisfied Terraform wont allow anything depending on the null_data_source to be created or modified.

data "null_data_source" "wait_for_lambda_exporter" {
 inputs = {
 lambda_dependency_id = "${null_resource.lambda_dependencies.id}"
 source_dir = "${path.module}/src/"
}
}

The lambda_dependency_id input wont be known to Terraform until the previous null_resource has completed - which means our npm install has either been completed, or nothing had changed and it wasn’t required.

The source_dir input can be pre-computed by Terraform, and we will use this value to tell the archive_file what to add to the bundle.

Finally we create our archive_file:

data "archive_file" "lambda" {
 output_path = "${path.module}/lambda-bundle.zip"
 source_dir = "${data.null_data_source.wait_for_lambda_exporter.outputs["source_dir"]}"
 type = "zip"
}

So now we’re creating our archive_file.lambda from the path given to us by null_data_source.wait_for_lambda_exporter, which wont give us an answer until null_resource.lambda_dependencies is satisfied, which ensures the correct dependency ordering is maintained!

Less than ideal

There is one minor issue with this method. On the first run there is no node modules, so the null_resource.lambda_dependencies.trigger.node value gets an initial sha256 value, and that causes npm install to be executed. This will then change the sha256 that would be computed to the src directory, and a second run would also create a new bundle. subsequent runs wont do this, but in an ideal world a single run should be all that is needed to converge our required to actual states.

Its certainly not perfect, but if you can tolerate this one caveat, its a relatively simple way to include node dependencies without having to do anything more elaborate than a normal deployment.

HTTP APIs with Simple Lambda Functions

Fri, 19 Jun 2020 07:56:34 +0000

API Gateway has always been somewhat of a beast to configure, and while its incredibly flexible its also far more hassle than most people want or need. Terraform was also completely the wrong tool for deploying these because of the sheer number of resources required and the need to deploy APIs to a stage just didn’t quite fit with the way Terraform works.

The new HTTP APIs, whilst almost completely un-google-able, are an amazingly simple was to expose Lambda Functions to the outside world. They’re easy to deploy, simple to define, have great 404 handling, built in JWT authentication, and an easy CORS config. Apart from the (currently!) missing service integrations, there’s literally nothing to dislike about them!

About The $default Route

HTTP APIs allow you to add a special route called $default which will essentially capture anything that doesn’t already have a route handler attached to it. It sounds like an ideal fallback route, but there is one caveat in that it will also capture CORS OPTIONS requests in some situations, which is definitely not what what I was expecting to happen with it.

The documentation doesn’t state this particularly clearly, but instead of using a $default route, instead you should use a /{proxy+} route to capture anything without a handler, which still allows the CORS requests to hit the HTTP APIs built-in handler without you needing to write any code.

Defining the API

Creating an API is pretty simple and allows you to create the CORS config at the same time:

resource "aws_apigatewayv2_api" "api" {
 name = "v2-http-api"
 protocol_type = "HTTP"
cors_configuration {
 allow_credentials = false
 allow_headers = ["*"]
 allow_methods = ["*"]
 allow_origins = ["*"]
 expose_headers = ["*"]
 max_age = 3600
}
}

This resource creates an HTTP type API in API Gateway, and adds an extremely open CORS policy. You would likely want to tie this down a little more, but this policy makes the API open everywhere which is easy for testing.

The Stage (With Logging)

Its good to be able to get request log files, which in AWS terms means a CloudWatch Log Group. In out case that needs to be attached to the API Stage. Incidentally HTTP APIs can be configured to “auto-deploy”, i.e. you don’t have to tell it to update the stage, just updating the API routes is enough.

resource "aws_cloudwatch_log_group" "api_logs" {
 name = "/api/logs"
 retention_in_days = 30
}
resource "aws_apigatewayv2_stage" "stage" {
 api_id = aws_apigatewayv2_api.api.id
 name = "$default"
 auto_deploy = true
access_log_settings {
 destination_arn = aws_cloudwatch_log_group.api_logs.arn
 format = jsonencode(
{
 httpMethod = "$context.httpMethod"
 ip = "$context.identity.sourceIp"
 protocol = "$context.protocol"
 requestId = "$context.requestId"
 requestTime = "$context.requestTime"
 responseLength = "$context.responseLength"
 routeKey = "$context.routeKey"
 status = "$context.status"
}
)
}
lifecycle {
 ignore_changes = [
deployment_id,
default_route_settings
]
}
}

The lifecycle block at the bottom is necessary at the time of writing due to some slight oddities between the way HTTP APIs work and the way Terraform works. Basically it tells Terraform to ignore the deployment_id and default_route_settings fields if AWS says they don’t match whats in the state file.

The access_log_settings.format field specifies the log format. You can find more about that in the AWS Docs

There are some other resources available at this level to allow you to add a domain name and map this stage on to it, and to create a JWT Authorizer config, but for the sake of simplicity I will explore those in a separate article.

Adding a Route

Adding Routes in V1 of API Gateway required a lot of work. Multiple path segments, a resource to handle the incoming request, one for the Integration, another for the integration response and multiple resources for potential return codes. There is a lot to love about all that including VTL templates doing a lot of work for you, and being able to avoid calling Lambda Functions altogether, but it really was an awful lot of resources, and Terraform is very verbose. There were even some timing issues with AWS eventual consistency that made deployment of infra code changes quite painful.

HTTP APIs have done away with all of that in favour of a single resource for the route, and another one for the integration of the lambda function (with the option to add the JWT Authorizer)

resource "aws_apigatewayv2_route" "route" {
 api_id = aws_apigatewayv2_api.api.id
 route_key = "GET /{proxy+}"
 target = "integrations/${aws_apigatewayv2_integration.integration.id}"
}
resource "aws_apigatewayv2_integration" "integration" {
 api_id = aws_apigatewayv2_api.api.id
 integration_type = "AWS_PROXY"
 connection_type = "INTERNET"
 description = "This is our {proxy+} integration"
 integration_method = "POST"
 integration_uri = aws_lambda_function.lambda.invoke_arn
 passthrough_behavior = "WHEN_NO_MATCH"
lifecycle {
 ignore_changes = [
passthrough_behavior
]
}
}

The fields are fairly self-explanatory, the aws_apigatewayv2_route attached a specific route to an API, and directs it to an integration.

The aws_apigatewayv2_integration tells API Gateway to forward the request to a Lambda function as an AWS_PROXY integration type, using a POST request to the Lambda service.

At some point there will be more integrations (you can already pass through to another API via a VPC link for example) however Lambda is the only service currently integrated with HTTP APIs.

Again, due to time of writing oddities in Terraform we need to add a lifecycle block to ignore the passthrough_behaviour field if it changes in AWS.

The Lambda Function Handler

The Lambda Function is as simple as providing the ARN (shown above). Given this is a {proxy+} route it could capture multiple route requests, and in our case we will handle just the / route and 404’s for anything not defined:

module.exports.handler = async (event) => {
console.log('Event: ', event)
if (event.path === '/') {
return {
statusCode: 200,
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
message: 'Welcome',
}),
}
} else {
return {
statusCode: 404,
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
message: 'Not Found',
}),
}
}
}

You can drop this code in to the resources created in the article Deploying Lambda Functions with Terraform and drop it straight in here.

Accessing the API

We’ve not added a domain name to map the API to, but we can still use the default url provided by API Gateway. The easiest way to get this is by adding an output to the code to tell Terraform to tell us what URL has been assigned to it:

output "api_url" {
 value = aws_apigatewayv2_stage.stage.invoke_url
}

You should now be able to curl the api_url that Terraform gives you and see that for a GET request on the / route you see a welcome response, a 404 response on all other requests, and an OPTIONS request should return the CORS response to allow browsers to access the API!

Deploying Lambda Functions with Terraform

Tue, 16 Jun 2020 20:05:28 +0000

Serverless architectures are built from a collection of Managed service wired together to create a specific service. This could be anything from a shopping cart to a full ETL solution. The advantage of building applications this way is that each of the component parts are fully managed by AWS. That means you (either the developer or company) don’t have to worry about them functioning, you just need to make sure they’re wired together correctly.

Managed services are great, but these service only offer basic functionality such as queues or databases. What they can’t do is know how your business actually adds value to the world.

Lambda functions are the Serverless super-hero because they let you run your own code without ever having to worry about how the server operates (yes, there are servers!) or even how how to install the software that runs your code.

There are a lot of ways to deploy AWS infrastructure, and even more for Serverless infrastructure, but my personal favourite is Terraform. Its designed to allow you to write your infrastructure as code (IaC) and is surprisingly flexible. It also allows you to interact with non-AWS services which can really help tie everything together.

Terraform can be a little verbose, but its very easy to understand, and very simple to work with. On the face of it Terraform doesn’t obviously lend itself to Serverless, but that’s because it designed to give you extremely low-level access to APIs in the form of Resources.

In this article I will show you the minimal set of code required to deploy a lambda function. Its not going to be production-ready, or even do anything very impressive, but its going to introduce at an entry level the minimal components required.

About Terraform

Terraform works by creating a direct-graph of resources. The resource dependency order can be determined by using outputs from one resource as the inputs to another. Terraform can then deploy resources which have no outstanding dependencies recursively until all resources have been deployed with their dependencies satisfied.

Terraform then creates a state file which it uses on further deployments to identify which specific resources in AWS it should be adding, deleting or modifying. State files can be stored in local files or remotely, but for now we will just keep them locally.

Terraform Meta-Resources

Terraform is incredibly configurable, and to do that it happily eats it own dog-food by configuring with Terraform resources! This is the content of a file I like to call meta.tf:

terraform {
 required_version = "> 0.12"
}
provider "aws" {
 region = "eu-west-1"
 version = ">= 2.61.0"
}

The terraform block tells Terraform to use a version no less that 0.12.0. While older versions are available, this version added some functionality that makes it a little easier to code and so is a good base.

The provider block tells Terraform that we want to use AWS resources, we want to place them in the eu-west-1 region and we want to use a provider version no less than 2.61.0

Creating the Lambda Function Bundle

A Lambda Function needs some code to run. The code can be in a variety of languages but in this case we’re going to use Javascript. Here is some simple code that we’re going to deploy:

To supply the code to the Lambda Function it needs to be in a Zip file. Terraform can create these zip files for us, and for simplicity we’re going to define the Function code inline:

data "archive_file" "lambda" {
 type = "zip"
 output_path = "${path.module}/.terraform/lambda.zip"
source {
 filename = "index.js"
 content = <<-EOF
 module.exports.handler = async (event, context) => {
console.log("EVENT: \n" + JSON.stringify(event, null, 2))
console.log("CONTEXT: \n" + JSON.stringify(context, null, 2))
return Promise.resolve()
}
EOF
}
}

Lines 7-10 will be written to a file called index.js which will be added to a zip file called lambda.zip. ${path.module} will be interpolated as the local path during deployment. The code exports a function called handler. Keep these in mind as they will be useful when we actually deploy our function.

The code is extremely simple: it’s going to print out the Event and Context objects which will be passed to it upon execution. This is how the function will get information related to its execution.

The Log File

Since the Lambda Function is running inside of AWS, we need some way to capture its output. The Lambda service automatically captures anything which would normally be sent to stdin and stderr and store that in a CloudWatch Log Stream. Although the Lambda service will create the Log Group, its best practice to create this yourself and set an expiry period so that the logs are automatically cleared out after a period of time:

resource "aws_cloudwatch_log_group" "lambda_log_group" {
 name = "/aws/lambda/${aws_lambda_function.lambda.function_name}"
 retention_in_days = 30
}

All Lambda functions log to a log group called /aws/lambda/<lambda name> and so we’re creating that and capturing the name from the Lambda Function resource (created very soon!)

We also set the logs to expire after 30 days. Logs are pretty cheap, but if the function is called a lot or it generates a lot of log output the cost can quickly mount up so after a short period we want AWS to delete them on our behalf.

Permissions

In order to be able to interact with any other service, AWS resources need an IAM Role which can be “assumed” by the calling service. In our case that’s the Lambda service:

resource "aws_iam_role" "lambda" {
 name = "simplest_lambda_role"
 assume_role_policy = <<-EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
}
}
]
}
EOF
}

The role needs to have policies associated with it in order to define what its allowed to do. In our case the only thing we need permission to do is to create Log Streams within the Log Group, and write to them. Policies can be stand-alone, re-usable policies, or in-line policies specific to this function. For simplicity in this case we’re going to create an in-line policy:

resource "aws_iam_role_policy" "logging" {
 name = "Cloudwatch-Logs"
 role = aws_iam_role.lambda.name
 policy = <<-EOF
{
"Statement": [
{
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": "arn:aws:logs:*:*:*"
}
]
}
EOF
}

The Lambda Function

At last we have all the pieces in place to be able to create our Lambda Function resource. Here we will give it a name, point it to our code bundle, tell it what runtime to use and what exported function to execute and what Role it should be assuming to give itself the correct permissions:

resource "aws_lambda_function" "lambda" {
 function_name = "simplest_lambda_function"
 filename = data.archive_file.lambda.output_path
 source_code_hash = data.archive_file.lambda.output_base64sha256
 runtime = "nodejs12.x"
 handler = "index.handler"
 role = aws_iam_role.lambda.arn
}

Terraform hides some nasty internal values such as ARNs and file hashes by taking the outputs from some of the resources we created earlier as inputs to this resource. In doing so we have also implied the dependencies of resources and helped Terraform work out what order things should be deployed.

Deploying the Function

To start the deployment process, Terraform needs to be initialised with the command:

terraform init

This will ensure we’re using a valid version of Terraform and download the providers specified in meta.tf (or a default if we didn’t specify a version). All of its internal “stuff” will be stored in a directory called .terraform

To see what Terraform is going to do execute the following command:

terraform plan

You should get an output something like:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.archive_file.lambda: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:

 # aws_cloudwatch_log_group.lambda_log_group will be created
 + resource "aws_cloudwatch_log_group" "lambda_log_group" {
 + arn = (known after apply)
 + id = (known after apply)
 + name = "/aws/lambda/simplest_lambda_function"
 + retention_in_days = 30
}

 # aws_iam_role.lambda will be created
 + resource "aws_iam_role" "lambda" {
 + arn = (known after apply)
 + assume_role_policy = jsonencode(
{
 + Statement = [
+ {
 + Action = "sts:AssumeRole"
 + Effect = "Allow"
 + Principal = {
 + Service = "lambda.amazonaws.com"
}
},
]
 + Version = "2012-10-17"
}
)
 + create_date = (known after apply)
 + force_detach_policies = false
 + id = (known after apply)
 + max_session_duration = 3600
 + name = "simplest_lambda_role"
 + path = "/"
 + unique_id = (known after apply)
}

 # aws_iam_role_policy.logging will be created
 + resource "aws_iam_role_policy" "logging" {
 + id = (known after apply)
 + name = "Cloudwatch-Logs"
 + policy = jsonencode(
{
 + Statement = [
+ {
 + Action = [
+ "logs:CreateLogGroup",
+ "logs:CreateLogStream",
+ "logs:PutLogEvents",
]
 + Effect = "Allow"
 + Resource = "arn:aws:logs:*:*:*"
},
]
}
)
 + role = "simplest_lambda_role"
}

 # aws_lambda_function.lambda will be created
 + resource "aws_lambda_function" "lambda" {
 + arn = (known after apply)
 + filename = "./.terraform/lambda.zip"
 + function_name = "simplest_lambda_function"
 + handler = "index.handler"
 + id = (known after apply)
 + invoke_arn = (known after apply)
 + last_modified = (known after apply)
 + memory_size = 128
 + publish = false
 + qualified_arn = (known after apply)
 + reserved_concurrent_executions = -1
 + role = (known after apply)
 + runtime = "nodejs12.x"
 + source_code_hash = "TFoUgCw/pW7Sy3/gY/TB3AtGzXoqRjN1L6YT066iPGg="
 + source_code_size = (known after apply)
 + timeout = 3
 + version = (known after apply)
+ tracing_config {
 + mode = (known after apply)
}
}
Plan: 4 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

This shows that Terraform will create 4 new resources:

The Log Group
An IAM Role
An Inline IAM Policy attached to the Role
Our Lambda Function

To actually create resources in AWS, execute:

terraform apply

When prompted, enter yes

Testing our Function

To invoke our function from the command line, execute:

`aws lambda invoke --function-name simplest_lambda_function /dev/null --log-type Tail --query 'LogResult' --output text | base64 -d`

which should respond with something similar to:

START RequestId: ab934699-7094-4829-8cee-aecd2c193583 Version: $LATEST
2020-06-16T16:58:46.098Z ab934699-7094-4829-8cee-aecd2c193583 INFO EVENT:
{}
2020-06-16T16:58:46.098Z ab934699-7094-4829-8cee-aecd2c193583 INFO CONTEXT:
{
"callbackWaitsForEmptyEventLoop": true,
"functionVersion": "$LATEST",
"functionName": "simplest_lambda_function",
"memoryLimitInMB": "128",
"logGroupName": "/aws/lambda/simplest_lambda_function",
"logStreamName": "2020/06/16/[$LATEST]759235165de84d3bbe7d3a87a7bd2db4",
"invokedFunctionArn": "arn:aws:lambda:eu-west-1:522711524578:function:simplest_lambda_function",
"awsRequestId": "ab934699-7094-4829-8cee-aecd2c193583"
}
END RequestId: ab934699-7094-4829-8cee-aecd2c193583
REPORT RequestId: ab934699-7094-4829-8cee-aecd2c193583 Duration: 2.86 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 63 MB Init Duration: 129.60 ms

While the command was a little long-winded, this is the log data you will also find in the CloudWatch Log Group. We can see the invocation id ab934699-7094-4829-8cee-aecd2c193583 and the content of the event and context objects. Finally we can see the REPORT line with tells us

The function took 2.86ms to run
It will be billed at 100ms (always rounded up to the nearest 100ms)
It was allocated 128MB of memory
At its peak, it actually used a total of 63MB of memory
It “Cold-Start” initialisation time was 129.6ms

You can use these values to tune the function which I will address in a future post.

Cleaning Up

Since this was just a simple demo, you probably want to remove all these resources at some point. To do so execute:

terraform destroy

Final Remarks

This function doesn’t do a great deal, and right now we can only invoke it from either the command line or from the AWS Lambda console, but it covers the basics of using Terraform to get it, and all its supporting resources deployed.

For more information see the official Terraform site.

About Barney Parker

Mon, 15 Jun 2020 00:00:00 +0000

Barney is pretty special :)

World Simplest HTML5 WYSIWYG Inline Editor

Mon, 05 Aug 2013 00:00:00 +0000

I am pretty new to web/JavaScript development, and often quickly reach for Google in order to find out how to do things.

Recently I needed to add the ability to do some pretty basic in-page editing of content. I didn’t want to use iFrame’s, i didn’t want a hugely featured editor, literally some pretty basic tools to be able to edit content, bold, italic, h1, h2 and so on.

After a fair bit of Googling, it seemed that there wasn’t a hope! all the plug in’s i found just seems unnecessarily complicated so, in a fit of self-importance, i decided i could do it myself. i’d end up with the editor i wanted, and code that i fully understood. Granted, it wasn’t likely to be an easy job, but hey, why would i code anything if there was an easier option?

To my surprise, as it turns out, it’s so simple, it’s almost a joke! all the tools are already available, all you need to do it tie them together with some really simple JavaScript, and it works!

There are two things you need, the first is

{% highlight javascript %}contenteditable{% endhighlight %}

contenteditable tells the browser that this item (a div for example) can be edited inline. You can do all the normal text editing here with ctrl-b for bold, ctrl-i for italics etc, but it’s fairly limited.

The second thing you need is a special function called

{% highlight javascript %}execCommand{% endhighlight %}

execCommand actually allows us to do some fairly complex stuff with the contents of the **contenteditable **area, and all in a really compliant way!

If you want to read what it can do in detail, check out The Mozilla Docs

Essentially, execCommand takes three parameters:

Command Name – the name of a command

ShowDefaultUI – not implemented, and best set to false

ValueArgument – optional for some commands so for example, using the command ‘bold’ will make the currently selected text turn bold with a tag!

ValueArgument can be set to null in all but one case. When you want to set a tag for a line of text (h1, h2 for example) you need to use the formatBlock command, and put the tag you want in the ValueArgument (without angle-braces).

Now this is simple enough, but to make life even easier, a single click handler and a data-attribute on each link allow us to have one very simple function to handle everything.

The demo uses bootstrap and jQuery, but only to make the code a little more brief, and could all easily be done in plain old JavaScript.

[CodePen height=300 show=result href=idjCG user=barney-parker ]

EDIT: CompuSam suggested using {% highlight javascript %}href='javascript:void(0)'{% endhighlight %} instead of {% highlight javascript %}href’#'{% endhighlight %} to stop and unwanted page movements etc….

Mon, 01 Jan 0001 00:00:00 +0000