I recently came across this thought-provoking discussion on Erik Wilde’s excellent YouTube channel “Getting APIs to Work”. It presents a small nugget of insight for your REST API Design. In my mind, this RESTful API Idea certainly deserves a broader audience to learn about. So, here’s what I took away from it.
API Idempotency is one of these topics you know are important, but somehow not important enough to end up in your Minimum Viable Product. And guess what? It seldom makes the cut in any subsequent sprint planning. The itch simply never gets urgent enough to scratch.
For many architects, exactly once processing is a reason to promote event-driven architectures. In here, the broker will make sure any message gets delivered. And once you implement a transactional client, you’re pretty close to exactly-once processing already. You just have to make good at an RPO=0 on your target system, or make it replay proof. In the architect’s book, that counts as easy. Fortunately, there is an even easier way. And this is how.
What’s the core idea?
The core idea is tantalizingly simple. If you only use idempotent REST API operations, you can keep repeating them until they succeed. And in case you keep repeating them even after they’ve succeeded, perhaps because the confirmation got dropped somewhere on the line, it has simply no effect on the state of the resource (the ‘thing’ behind the URL – a Uniform Resource Locator).
When you design a REST API, you can choose from four main operations: POST, PUT, GET, and DELETE. Of these methods, POST is the only one that’s not idempotent in nature. When you POST the same representation of an object twice, you’ll add two resources to the collection in your datastore. That’s how it behaves (assuming you POST to a collection, as is the recommended practice in REST). Sometimes, that’s exactly what you want, but more often, that’s an unfortunate side-effect of poor error handling, or poor user interface design. And that’s why you might consider a ban on using POST altogether.
Too radical an idea? Not feasible? Well, it is by far the easiest implementation of delivering full idempotency that I’ve come across. Please read on to learn why.
What problem are we trying to solve?
On the internet, things go wrong all the time. It happens every now and then that you POST a request, but never get a response back. Now, this might mean the resource server never processed your request, in which case you want to resend it. But it is also possible that, somehow, the request got processed all right, but the response got lost. Now you don’t want to resend your request, because that would introduce a duplicate. And we all know how painful it can be to safely deduplicate those.
But how to find out what actually happened? You could try a HEAD method, but for that to work, you need the Identifier of the resource that may or may not be created. With a POST, the resource server is typically in charge of issuing that identifier. You can find it in the respons message you never received. So, you’re pretty much stuck, and you’d have to take a guess.
Helping you to select and design your Enterprise API Management platform
Download nowWhat’s the alternative exactly?
Obviously, you can use a PUT instead of a POST. You can PUT an object as often as you want, it will create at most one resource. You just take a random GUID as identifier of the resource you PUT. That’s a good practice anyway. You shouldn’t use easily guessable identifiers like a sequence, an email address, or an SSN, remember? And whether the client or the server issues the GUID doesn’t really matter, right?
But this also comes with some downsides. One edge case is regression. If you think about it, a POST equals an INSERT in the datastore, whereas a PUT equals an UPSERT. Say, you’ve PUT a new Order in, but the response got lost. While waiting for a timeout, one or more updates were processed. The inventory was updated, and a payment request was issued. Now, you PUT the state of the resource back to its original. Bad things will surely happen. But it turns out you can totally prevent that from happening.
The magic of this pattern is in its use of conditional requests (check RFC 7232 for details). Using this API Design trick, the HTTP method gets processed only when a condition is met. It’s often used in conjunction with an E-tag, or Entity tag, as a signature of a response. Since you have PUT the object once before, but didn’t get the confirmation, you now want to PUT the object if and only if the resource doesn’t exist yet. It turns out there is an easy way to do just that, without knowing any E-tag value issued by the server. Just add the following header to your PUT, and you’re safe to retry as often as you want.
If-None-Match “*”
Effectively, this condition turns a PUT request from an UPSERT into an idempotent INSERT. It signifies to only PUT the resource if there’s nothing to match. In other words, if the resource does not exist yet. An empty E-tag, if you want. Otherwise, it will return a status code HTTP:304 (Not Modified). It will never update anything. Brilliant.
Is there a catch?
Not really, not as far I can think of. Of course, there is an ever so slight chance that another resource with the same GUID exists already in your user store. But that can be easily circumvented using the same trick. If you add the same header to the initial PUT request, getting a 304 on the first try would indicate the GUID you generated is, in fact, not unique. Presto!
That only leaves the infinitely tiny chance that the 304 on the initial request gets lost, and you misinterpret the 304 on the retry as a success. If that worries you, you can always insert a HEAD request before your first PUT. If you receive an HTTP:200, it’s your lucky day. Consider buying a lottery ticket, but don’t forget to generate another GUID before PUTting your resource once again.
Is there yet another way?
On the web, there’s always another way. In this case, if you’re really beholden to using POST instead of PUT when adding a resource, consider this. There’s nothing in the HTTP specification that stops you from POSTing to an identified resource, rather than a collection. POSTing to a collection is just a practice in RESTful API Design. And this identifier could still be a client-generated GUID, for that matter. You would get an HTTP:409 in case of conflict, and handle that exactly in the way you would handle the 304. But it would never, ever overwrite anything.
The event-driven approach I mentioned in the introduction also could work. You could consider the application of CQRS. In here, there is a broker between the client application and the back-end. Once the create request has been confirmed, the broker will guarantee its delivery. In other words, the client doesn’t have to wait, but gets its confirmation almost instantly. You would probably use an AsyncAPI to publish a create event to the broker, using a CloudEvents description. The Resource Identifier could equally be generated in the client or the back-end. The back-end would publish its response either to the reply-channel your client is subscribed to, or as a direct callback if the client supplied a webhook. Sounds complicated? Not so much, at least if you’re already invested in the eventing capability. But the Idempotent REST API definitely wins the simplicity contest.
What to choose?
There are multiple options with different pros and cons. As an Integration Architect, analyzing the trade-offs between multiple architectural alternatives, both large and small, is my daily business. I can help you make a deliberate choice. Moreover, I will make sure that the chosen option will get a buy-in from its stakeholders. Feel free to contact us.
Now, are you ready to PUT your POST behind you? Do you prefer the idempotent usage of the POST? Or do you lazily accept the brittleness of your POST?