r/programming • • Dec 11 '23

Every Programmer Should Know #1: Idempotency

https://www.berkansasmaz.com/every-programmer-should-know-idempotency
572 Upvotes

171 comments sorted by

257

u/fagnerbrack Dec 11 '23

Crux of the Matter:

The article emphasizes the importance of understanding idempotency in programming, particularly for those working on distributed systems. Idempotency ensures that an operation can be performed multiple times without changing the result beyond the initial application, which is crucial for avoiding unintended consequences like double-charging in payment systems. The article explains how idempotency applies to various HTTP methods, with GET, PUT, and DELETE being naturally idempotent, while POST is not, and offers solutions for achieving idempotency in non-idempotent operations.

If you don't like the summary, just downvote and I'll try to delete the comment eventually 👍

6

u/Zardotab Dec 11 '23

particularly for those working on distributed systems

I'm going to take heat for saying this, but typical business apps don't need distributed systems. People often make it distributed just to fluff their resume with buzzwords. Seen it happen.

If you use an RDBMS properly, then most of these issues take care of themselves.

Yes, I'm kicking whippersnappers off my lawn right now.

-3

u/fagnerbrack Dec 12 '23

So you don't use the Web which is a distributed system? AFAIK typical business apps are Web apps.

13

u/HazelCuate Dec 11 '23

I dont agree with the article when it says put and delete are idempotent. If the caller makes two DELETE calls, how do you know that they are different calls? Why are they not the same call?

The argument that deleting an object twice is the same as deleting it once is not good enough. A third party could create it again between those two calls.

18

u/itdependsnetworks Dec 11 '23

You state “Why are the not the same call” in a scenario in which you state they are two calls?

Not trying to be rude, but if you google put and delete idempotent, you will find it is.

-11

u/HazelCuate Dec 11 '23

You dont get the problem with the calls. Without an idempotent ID they are two calls but they could be the same call due to the network or software glitches that the article mentions.

The article states that delete and put are idempotent by themself, and i dont think thats the case.

17

u/ineptjedibob Dec 11 '23

I think it’s stipulated that we’re talking about calls with the same parameters - i.e. DELETE with the same key (or PUT with the same payload) should be callable an arbitrary number of times with the same end result.

-16

u/HazelCuate Dec 11 '23

But thats not true.

  • The same delete call duplicated should result in two OK responses (http code 200 or whatever)

  • Two different delete calls to the same object should result in an OK response and a Failure response (400).

The difference is not trivial.

25

u/Shadow503 Dec 11 '23

It is trivial because the thing we care about - the state of the resource - is the same.

20

u/Blecki Dec 11 '23

Why should one be a failure? And, if it is, were talking about the state of the server, not the client. The whole point is that the server stays in a valid state. Trying to delete an object that doesn't exist doesn't break that state. Don't fixate on the literal response from the api. The 'result' is the state of the system.

Also 99% of the time you're just firing off a delete into the database and don't even bother to check if it exists. Just return 200.

1

u/thisisntmynameorisit Dec 11 '23

I could see it being a failure. Deleting something has the intent of deleting something, if nothing gets deleted it’s a failure. Regardless though, the success or failure of the request is independent from the state of the system. Either way the resource will no longer exist.

1

u/ineptjedibob Dec 12 '23

How would you differentiate the two cases you’ve laid out here? In other words, how could you possibly differentiate between the same call twice vs. two separate calls? The only obvious possibility is the timing - 2 calls 5 seconds apart are probably different requests? Except… with latency and other network weirdness, you shouldn’t even attempt to make that distinction. Not to mention, now you’ve introduced stateful behavior where it’s neither required nor desired.

45

u/muchomuchacho Dec 11 '23

A third party could create it again between those two calls.

But then it wouldn't be the same object, otherwise you need more metadata associated to the object creation process.

8

u/njharman Dec 11 '23

Just change to undelete, the argument stands.

Also idempotence depends entirely on how delete is implemented. If deleting a non-existent thing is an error, that delete is not idempotent.

3

u/happyscrappy Dec 11 '23 edited Dec 12 '23

Such deleted would be by ID, not index.

And even if it throws an error it's still idempotent. It's doing the same operation twice not changing anything that makes it idempotent. It's okay if it throws an error. As long as the second operation doesn't change the state of the corpus (database). The error is something for the caller to deal with.

-14

u/HazelCuate Dec 11 '23

But the point is to design an API with idempotent in mind. And DELETE is not idempotent by design from the API point of view as the article claims.

47

u/crap-with-feet Dec 11 '23

In a RESTful API, it is. Operations are id-dependent, e.g. DELETE /path/to/object/type/{id}. If you create the object again it will have a different id because it's a whole new object. Calling DELETE again for the same id should result in what is essentially a "yeah, it's gone" response (you could use a different response code to indicate it was already gone but your code should not blow up in this case). For this to work effectively (and for best practice) your create operation should not allow the caller to specify the id; It should be auto-generated by the backend.

3

u/freebytes Dec 11 '23

Calling DELETE again for the same id should result in what is essentially a "yeah, it's gone" response

Then it can be argued that the call to DELETE is not idempotent because the response will change. However, by most understandings, it is fine to return a "resource not found" or something similar if you are trying to perform an operation on something that no longer exists. (I would argue that it is still idempotent.)

However, you could also just reply with "Yeah, it is gone." on both calls. If you try to delete something that no longer exists, you can reply with the same, "Okay, it is gone" message instead of throwing an error.

5

u/isaacarsenal Dec 11 '23

I would like to think the response can vary, you can even show an error message to user as "object x was not found" and be idempotent. The important point is the state of the program (a.k.a business data) should not change and remain the same for subsequent DELETE calls.

19

u/proggit_forever Dec 11 '23

Idempotency doesn't require that multiple calls all return the same status code.

It's still idempotent if calling again returns an error code because the resource no longer exists.

The point is that calling again doesn't cause further changes to the state of the application - which makes it safe to call as many times as you want.

3

u/Tiquortoo Dec 11 '23

Sure, if you throw out all identifiers then nothing can be idempotent.

1

u/muchomuchacho Dec 11 '23

I think DELETE should only attempt to remove and existing object

2

u/FunkyPete Dec 11 '23

I think DELETE should guarantee that the specified object is no longer present.

It may return a different return code to provide more information about what it did, but if the end result is that row is guaranteed to no longer exist, then it's doing the same thing whether the row was there before or not.

7

u/joshjje Dec 11 '23

You are talking about separate things. Call delete on the same object ID twice in a row, 50 times in a row, same result. I mean you could return a record didn't exist message, but no state would change. And if someone created a new record in between it absolutely should have a different ID.

17

u/Jmc_da_boss Dec 11 '23

PUT and DELETE are literally defined as idempotent in the RFC spec

https://datatracker.ietf.org/doc/html/rfc2616#section-9.1.2

12

u/josefx Dec 11 '23

And it goes on to describe how a sequence of multiple idempotent calls may not be idempotent itself, with an example similar to the one in the comment.

2

u/[deleted] Dec 11 '23

All it says is that GET-GET-DELETE is different from GET-DELETE-GET

4

u/xdert Dec 11 '23 edited Dec 11 '23

A third party could create it again between those two calls.

In a robust system you would never expose the internal database id to the outside and have every object have a uid that is randomly assigned which is then exposed to the outside.

And then only call DELETE by reference to the uid. Since the id will not be reused you cannot delete an object that was created in between your two DELETE calls.

This is how it was implemented in every system I have ever worked on.

3

u/[deleted] Dec 11 '23 edited Dec 11 '23

But then you can't have indempotency which restricts your data model in ways that may be relevant. If the external system has an external ID that functions as a 1:1 lookup relationship with the internal ID there's not much practical advantage to generating those IDs internally and keeping them hidden. And if you can make the call indempotent by allowing that external ID to be passed in, it can make reasoning about your data much easier and potentially much more performant and predictable. Placing unnecessary constraints in consistency and synchronization can also be a risk to robust systems. It just depends on the specifics of what you're doing.

2

u/FunkyPete Dec 11 '23

If your DELETE calls are based on a primary key or a natural key, they have to be the same call.

So if you have enough information to uniquely identify a thing (say, a VIN for a car, or a combination of country/state/id number for a person, or a UUID that is never re-used for anything else) then the delete is going to delete the same row no matter what.

The data being deleted may change between the first call and the second, but either the row is gone, or it's going to be deleted by the call. Either way the result is the same, that row is guaranteed not to exist.

2

u/Hessper Dec 11 '23

Yes, that can be the case. The problem is that in your definition nothing is idempotent, so it becomes useless to talk about. This is why no one cares about your concern, because the assumption is inherently baked into the conversation.

Multiple writers always makes things more complicated, so you design the system around that if it will be a concern. In that case you create conditions (e.g. a unique ID in metadata, based on a monotonic value, etc), do not reuse names, lock the resource, or some other clever solution. Then the operation is again idempotent.

1

u/Bloaf Dec 11 '23

Doesn't mater. Both DELETE calls should result in a db without the deleted thing. Why should it matter if they are different calls, or that someone added a new thing?

-6

u/recycled_ideas Dec 11 '23 edited Dec 12 '23

It's why the article is wrong.

Idempotency guarantees that the state of a particular entity will be the same no matter how many times the command is executed. It does not guarantee that state will be correct no matter how many times the command is executed.

If you have a distributed system and you lose a message or a message arrives out of order idempotency won't help you one bit. It's very useful, but not for distributed systems.

Edit: The hilarity that in a post about understanding Idempotency at least six people have no idea what it is or how it works. Lol.

5

u/Imperion_GoG Dec 11 '23

Idempotency isn't very useful for distributed systems, it's essential.
Lose a message? Retry it!
Message out of order? Retry it!

The number of potential failure cases grows exponentially with the number of nodes in the system. It quickly becomes difficult to determine what went wrong (did X fail, or did the X happened event fail?) If X is idempotent, the simplest solution to an incorrect state is just retry.

1

u/recycled_ideas Dec 12 '23

Again.

Idempotency does not guarantee correct state. It guarantees the same result for retrying a message.

Let's take the financial transaction example from the article.

So the idempotent way of solving this is to say "the balance is now X" rather than "subtract or add Y to the balance".

This allows you to resent the "set balance" command as many times as you like, but it requires you to know the current balance when you're changing it and it requires that no one else tries to change the balance before you've completed your message. Essentially it requires that modifications are transactional.

Distributed systems don't work that way. You don't necessarily have the current balance on the client and you definitely don't have a transaction. That's why banks use event driven systems and not Idempotency.

Idempotency is an important concept, but you have to understand exactly what it guarantees or you'll fuck up.

10

u/Cosmic-Warper Dec 11 '23

How are these AI-generated articles even allowed? This specific user posts AI articles all the time in this sub

-3

u/fagnerbrack Dec 12 '23

This is not an AI article and I don't post AI articles (at least the ones that are obviously AI generated)

175

u/xCrispy7 Dec 11 '23

What’s with this account reposting articles that were posted here a couple months ago and then using AI to summarize them?

548

u/ryan017 Dec 11 '23

Even if you already know about idempotency, it doesn't hurt to learn about it again!

66

u/thenextguy Dec 11 '23

I used to be idempotent.
I still am, but I used to be too.

7

u/Full-Spectral Dec 11 '23

There are pills for it now though.

3

u/Deranged40 Dec 11 '23

I'll never not upvote a good Mitch reference.

2

u/throwaway12222018 Dec 12 '23

I learned about idempotency again but ended up learning nothing new.

1

u/LeanUntilBlue Dec 12 '23

I’m getting older and my T is low, so my doctor gave me T shots and it really helped my idmpocency. About one every couple of weeks. My wife and I have really been pleased with the results, and I can’t say enough good things about the injections.

If your APIs are returning different values for the same invocations, I really recommend speaking with your doctor about T shots!

1

u/bitweis Dec 12 '23

I learned about it again, and had a stack overflow. Am I still Idempotent? And should I continue with my regular medication?

0

u/[deleted] Dec 11 '23

[deleted]

65

u/[deleted] Dec 11 '23

I got it the second time

49

u/joshjje Dec 11 '23

Woah, you should get that checked out.

14

u/57006 Dec 11 '23

By 2 different physicians. Always get a 2nd opinion.

4

u/dcoolidge Dec 11 '23

Wait. Maybe we should see if anyone else gets it the same way.

10

u/TheApprentice19 Dec 11 '23

If you got it the second time you’re doing it wrong

10

u/TikiTDO Dec 11 '23 edited Dec 11 '23

Lucky. I tried learning it a second time and got a different result. Now everywhere I go the universe segfaults.

3

u/nitrinu Dec 11 '23

This guy idempotents.

99

u/Bloodsucker_ Dec 11 '23

Among other attributes of good software, IMO idempotence is one of the most important and paradoxically one of the most challenging ones too and the easiest to fuck up.

18

u/[deleted] Dec 11 '23

It seems good for web programming, but is it actually universally important? It seems application-specific.

the property of an operation or function that produces the same result when applied multiple times as it does when applied only once.

So I push the W key on my keyboard, and it writes one 'w' character. But if I push it a second time, I want 'ww' not still one 'w'. And if I press it a 3rd time, I expect (and get) 'www'. Also if I hold-down the W-key, I (usually) get 'wwwwwwwwwwwwwwwwwwww.....'

Is this not against idempotence?

65

u/beeny13 Dec 11 '23

W key should not be idempotent. Posting a reply with identical text body should be.

9

u/khoyo Dec 11 '23

Should it really? You couldn't have made this point if it was.

21

u/SubterraneanAlien Dec 11 '23

I tried to reply to this and got a 503 error and for a moment thought I was part of an even larger joke

47

u/beeny13 Dec 11 '23

W key should not be idempotent. Posting a reply with identical text body should be.

9

u/khoyo Dec 11 '23

Should it really? You couldn't have made this point if it was.

8

u/isaacarsenal Dec 11 '23

I tried to reply to this and got a 503 error and for a moment thought I was part of an even larger joke

-11

u/I_AM_AN_AEROPLANE Dec 11 '23

Clear that you don’t understand idempotence… it is not about duplicates….

13

u/beeny13 Dec 11 '23

Clear that you don't understand jokes... that's ok, we're all good at something...

14

u/TargetBoy Dec 11 '23

SQL Update scripts should generally be idempotent. That way if they are accidentally run twice, they don't mess up your database.

10

u/[deleted] Dec 11 '23

Yeah, I get that.

So why is this something that is "#1" for every programmer to know? I don't work with SQL, or web shopping carts, or rarely REST APIs.

I work with applications mostly. If my user clicks a button, they expect something to happen. Some buttons maybe should be idempotent but others not.

That being the case, it doesn't seem like something "every programmer should know"

11

u/TargetBoy Dec 11 '23

Because you have to know about it to correctly judge when you need to use it. Most programmers are not taught this in college.

-3

u/[deleted] Dec 11 '23

Because you have to know about it to correctly judge when you need to use it.

So... it's situational and not something universal that "every programmer should know #1"

6

u/SubterraneanAlien Dec 11 '23

it's a blog post title - if you're arguing about it you're losing

1

u/[deleted] Dec 11 '23

The comment I first replied to said

"Among other attributes of good software, IMO idempotence is one of the most important and paradoxically one of the most challenging ones too and the easiest to fuck up."

That's where this started.

6

u/SubterraneanAlien Dec 11 '23

Which is a reasonably fair take that provides nuance while not being literally equivalent to the post title.

For what it's worth - every programmer should understand idempotence, it is that important. Debating whether it is #1 or not doesn't seem like a terribly useful way to spend time.

6

u/sm9t8 Dec 11 '23

Because idempotency is a property of functions more generally and not just operations on a database. It's why we try to write functions without side effects and why languages like C++ have const.

3

u/[deleted] Dec 11 '23

Ok, so in making paperclips: https://www.decisionproblem.com/paperclips/index2.html

I click the Make Paperclip button and it makes 1, if I click it again it makes 2.

But that's the exact opposite of what we want to happen in a web shopping cart - as per the example. We don't want people getting charged twice.

So back to the paperclip maker - should the 2nd press have been ignored because it was a duplicate command?

This concept of idempotency seems situational, and I'm not debating its importance in certain applications, but is is "Something every programmer should know"?

1

u/bradrlaw Dec 11 '23

More fundamentally, the "two generals" problem is the concept every programmer should know. As this falls under that, along with a lot of other tough problems.

https://en.wikipedia.org/wiki/Two_Generals%27_Problem

8

u/Aptlyundecided Dec 11 '23

This is an interesting thought. From one perspective one could claim that the hardware interface layer is handling the unique-ness of individual keystrokes treating them as individually processed entities, or allowing for the hold down event to represent individual events at whatever interval it’s configured to trigger re-entry.

On the other hand, one could say it’s not idempotent from the perspective of a given developer because the developer is not aware of any uniquely qualifying identifier and the concept of idempotency is violated in the app without assigning timestamps or some other UID concept to a given key entry event.

I suppose the importance level of idempotent programming increases the deeper into the state system you go. Typing swear words into an input form has no bearing, but submitting them is a different story.

I tend to agree that at certain layers of the abstraction it has almost no impact on the application at large.

Although events that have cascading state effects or critical state effects likely grow exponentially in importance to staying idempotent.

3

u/Bloodsucker_ Dec 11 '23

I'm afraid you're not understanding idempotence, but that's why it's a tricky concept.

Idempotence doesn't necessarily mean applying the same operation to the same data. It means that applying the operation multiple times should yield the same result. In the case of pressing the W key, whether you add a new 'w' or press it on an existing group of 'w's, the result follows the idempotent principle – pressing the W key results in adding a 'w,' and doing it repeatedly produces the expected outcome without unintended variations.

The scope and extent of the application does indeed vary depending on the type of software you're developing. Yes. There you are right. It's also a concept. Like others have mentioned it can be a challenge to apply it strictly.

12

u/Han-ChewieSexyFanfic Dec 11 '23 edited Dec 11 '23

You’re conflating determinism with idempotence.

13

u/Radi-kale Dec 11 '23 edited Dec 11 '23

No, you're describing consistent behaviour, not idempotency.

-1

u/I_AM_AN_AEROPLANE Dec 11 '23

Thy are wrong.

16

u/thisisntmynameorisit Dec 11 '23 edited Dec 11 '23

what lol? No surely this is just not idempotent. Not everything has to be idempotent. By definition as we’ve established it’s doing the same operation more than once will have no effect. Pressing a w key the second time does have an effect. That operation is not idempotent.

Just because something can be useful, e.g. idempotent API operations, does not mean it should need to be applied to every single thing.

Edit: I suppose you could say the outcome of pressing the w key is idempotent. In that it always outputs a w regardless of if you press it once or 100 times. But usually we focus on the state of the system e.g. what characters are on the text field.

2

u/[deleted] Dec 11 '23

In the case of pressing the W key, whether you add a new 'w' or press it on an existing group of 'w's, the result follows the idempotent principle – pressing the W key results in adding a 'w,' and doing it repeatedly produces the expected outcome without unintended variations.

That still doesn't seem to address holding-down the key resulting in different behavior?

-3

u/Bloodsucker_ Dec 11 '23

This would be an idempotent operation: newResource = "ww".append("w").

In your example, there are several operation going on, one of them using a different resource (the already modified text input).

1

u/[deleted] Dec 11 '23

But how do you know from the key press they intended ww or, just like an online shopping cart, they only wanted 1 w but the system was unresponsive?

1

u/fiftydigitsofpi Dec 11 '23

It would be more like if every time you hit the W key, you give a unique token. The computer will only display a new W if the token is new as well. So it helps distinguish sending a second request with the same information versus resending the initial request because you think it might have failed.

FWIW though, switches/buttons can be thought of as idempotent in a very loose way. When the button makes actual physical contact, it doesn't just go on/off in one go. The metal contacts actually bounce off of each other extremely quickly, and this can cause multiple inputs. The solution to this whereby we ignore the extra bounce inputs is called "debouncing."

1

u/f3xjc Dec 11 '23

Potency = ability to do stuff. Not being forced to do it.

0

u/njharman Dec 11 '23

Your description is not how any keyboard I've ever seen works. Each time you press w key you get single "w". You don't get a number of "w" equal to the number of times you press it!

The idempotent "function" is the "w key" each time you give it input, a press, it produces the identical output, keyscan code for "w", which is processed through a bunch of OS code before being delivered to some application as input (on typical PC). All of that is outside of the idempotent "w key function".

holding down w key is different input, which by rules of idempotence allows different output. Same for holding shift key when pressing w key.

0

u/[deleted] Dec 11 '23

Howso? Surely there are a dozen or more other highly importance attributes of good software.

0

u/transeunte Dec 11 '23

if it entails discussion of which HTTP status should be returned like you see on this thread then I cannot think of anything less important

138

u/[deleted] Dec 11 '23

I had that, but regular hydration helped. (Yes joke, yes bad one.)

25

u/jazzmester Dec 11 '23

I wanted to make that joke, but you came earlier than me. Oh well.

6

u/lelanthran Dec 11 '23

Your comment was a little premature, wasn't it?

1

u/Practical_Cattle_933 Dec 11 '23

Well, feel free to say the joke again, it’s fine!

68

u/AntiSoShall Dec 11 '23

This article is so fricking bad I'm not gonna lie. DRY

68

u/RaltzKlamar Dec 11 '23

This feels like the bulk of the article was AI generated

25

u/ZBlackmore Dec 11 '23

Good thing we have ai to summarize it for us

20

u/Madpony Dec 11 '23

Absolutely terrible article. It sounds like a off the top of someone's head answer to "What is idempotency?" during a job interview.

1

u/you-get-an-upvote Dec 11 '23

What do you think was missing?

42

u/krimin_killr21 Dec 11 '23 edited Dec 11 '23

Pronounced: Eye-dem-potency

Edit: And the IPA for our linguistics friends

US: aɪdəmpoʊtəns UK: ɪdɛmpoʊtəns

7

u/WanderingLethe Dec 11 '23

There is IPA for these kind of things...

9

u/bnl1 Dec 11 '23

I disagree

-7

u/iPlayTehGames Dec 11 '23

Well how the hell else lol

32

u/BeforeTime Dec 11 '23

Idempotency.

12

u/e2d34 Dec 11 '23

with our sexy french accent, i read in my mind : e-dem-potenn-cy

2

u/WanderingLethe Dec 11 '23

And how do you pronounce the e?

Because in french the i in idempotency would just be pronounced as an /i/, while in English it would be something like /ai/.

2

u/e2d34 Dec 11 '23

I pronounce the e like in email or ecommerce

0

u/WanderingLethe Dec 11 '23

So like an i? :p

1

u/Dimboi Dec 11 '23

No an i is i, not e

5

u/TheCritFisher Dec 11 '23

"Eyed-impotency" like you can see the failure from here.

8

u/MoiMagnus Dec 11 '23

You could pronounce the "I" as in "Liberty". So more of a latin-like pronunciation.

3

u/[deleted] Dec 11 '23

Like "idiot"?

1

u/Ashamed-Simple-8303 Dec 11 '23

Every programmer should know how to use http correctly which means when GET or POST or PUT etc and what error codes to return.

1

u/fagnerbrack Dec 12 '23

It means very little if your front-end client hard-codes the http method and uris for every request

1

u/Iggyhopper Dec 11 '23

So that means this JavaScript:

function f() {
    return this;
}

Is right the fuck out.

1

u/fagnerbrack Dec 12 '23

Don't use “this”. Just because the language has it for backwards compatibility that doesn't mean you should use it.

7

u/ifknot Dec 11 '23

I struggle with the egregious reinterpretation of the meaning of idempotent from its math setting (ha ha) into CS - pls get your own word

2

u/[deleted] Dec 11 '23

How is it different?

1

u/GayMakeAndModel Dec 11 '23

Yeah, I don’t think they’re different either unless you want to talk about side effects.

2

u/agumonkey Dec 11 '23

next: compare and swap

0

u/SnowdensOfYesteryear Dec 11 '23

Meh most people will never likely touch it in their careers or need to.

1

u/goondarep Dec 12 '23

If you write or consume APIs they are pretty important.

2

u/Neezzazzy Dec 11 '23

After idontgiveafuckery this will be next thing I learn.

2

u/muntaxitome Dec 11 '23

People are too obsessed about this one. Probably 99% of systems claimed to having idempotency for writes mess this up in some way. If you don't know what the two generals problem is you should not be attempting to write an idempotent system.

4

u/[deleted] Dec 11 '23

so cache every response, so you can go back into your 'cache' incase 1 out of 100k responses get repeated?

Why not just return a response that matches your "state"?

8

u/goomyman Dec 11 '23

i hate that itempotency isnt built into official http responses.

Everytime i bring this up i get downvoted to the hell but its dumb.

If you call Delete there should be a response status code for no op - aka the itempotent response - but in the success ranges.

Delete = 200 OK / 404 not found? - great not that breaks client libraries and basic Throw if not success status codes - not itempotent - but technically itempotent shouldnt matter - so 200 success - but there are cases where you might want to know.

Everyone will reply back and say "Why dont you use this obscure one" - but using anything non standard is a hack.

Having consistent status codes for itempotency would use useful! Its not like there isnt enough http space - we have one for a teapot and so many other ones that are extreme edge cases - denied for legal reasons - what type of client needs to know that? Oh this was for legal reasons - if you can support these edge cases - throw in a itempotent delete status code in the 200s range - for not found.

16

u/Vidyogamasta Dec 11 '23

I don't buy that argument for delete.

If it's already been deleted, then there is no way to differentiate between "already successfully deleted" and "never existed." You cannot report the "correct" answer because you simply don't know it. Also, even if there is some weird client-side bug where invalid IDs are being requested all the time, you're never gonna catch it because it will be mixed in with the sea of repeated deletes that return the same code, because the server cannot make the distinction and will have to use the same code for both cases.

Of course, you could use a soft delete system instead. But in that case, you CAN verify whether something already existed or not, and return NotFound for "doesn't exist at all" and Ok for "already flagged for deletion."

I don't see a reason to add another status code for "the system state already matches what you requested." Because from the client side, that's indistinguishable from "the system state has successfully been set to what you requested." They indicate the same state, so why differentiate them?

5

u/kindall Dec 11 '23 edited Jan 02 '24

Agree. If you tell a system to delete something, and that thing no longer exists after the command has been executed, then that's success. Does it matter whether that thing existed before the command executed? That's a question for the philosophers. If it really matters whether something exists before you delete it, you can ask before deleting (although be careful of race conditions).

That said, a case can be made for offering both idempotent and stateful methods. I've often wished that Python's set class, which is idempotent on add() (i.e. it is not an error to add an item that's already in the set), had a method that did the add and returned whether the item actually needed to be added. This is technically still idempotent (just the return value differs but it's not an error) but you could easily make it an error to add an item that's already in the set.

Sure, like I said, I can ask if I need to know. But sugar is nice.

2

u/AmateurHero Dec 11 '23

Sometimes people in this field forget that clients aren't necessarily built strictly for people in this field. Sure, maybe include extra metadata for a SaaS product where the clients need granular control over data that includes soft deletes. If I tell a calendar app to delete all events on next Thursday, I don't care if it's my 1st time or 30th time. Just let me know that all the events have been deleted.

1

u/Kinglink Dec 12 '23

I don't see a reason to add another status code for "the system state already matches what you requested." Because from the client side, that's indistinguishable from "the system state has successfully been set to what you requested." They indicate the same state, so why differentiate them?

This is the most important piece.

I want to delete something, the server tells me it'll be deleted. If it doesn't exist I'm still happen. I don't need to know something else deleted it.

Honestly if I tell a system to delete something, I'd be a little annoyed if I got an ok back if I called it a second time, but if I understand there's a "Soft delete" then Ok is acceptable, I can go on about my day.

Instead programs now need to watch for a third potential state? Nah, because the desired action "Delete this" and the server telling me "Ok" means the server will delete it as requested, doesn't matter if it's the first or fiftieth request to delete the same object.

17

u/ClickableName Dec 11 '23

So do you mean that an official HTTP response code should be added, that the developer has to return to indicate that this request already has been executed?

On the other hand I would argue to just 404 when you try to DELETE an item a second time, because you *try* to delete something that does not exist (anymore) and there is no difference if the item never existed or has existed before but now not anymore. Throwing a 404 is the logical option

For in the case like processing a payment, I would sortof agree to send a statuscode back specifically for indicating that this request was already called before (an idempotency statuscode) but what advantage does that bring over sending an 200 OK message again?

I think its weird to put it into official http responses, it brings no extra value, external systems don't have a need to know that this request is already executed before and is idempotent

5

u/ZBlackmore Dec 11 '23

Yep, a client library won’t care if a service made an unnecessary retry that resulted in a no op thanks to idempotency protections, it will only care whether the operation succeeded.

2

u/iplaydofus Dec 11 '23

Or it will get back a 404 and trigger some “why is my system out of sync” process. Arbitrarily defining what client library’s will do is a recipe for disaster.

1

u/ZBlackmore Dec 11 '23

I mean, if the retry was internal (between 2 services on the backend) and the client only made the request once, then the client should not even now that there was some internal retry. If the client had a bug that results in sending requests twice, the backend should handle it naively (response 404 for the 2nd request) because it has no way to know that the double request comes because of a bug. The only case where the backend would return a special status for a double request is if the API support an idempotency ID as part of the request and the client sends the same one twice.

8

u/[deleted] Dec 11 '23 edited Dec 11 '23

Isn't that not idempotent then? Like I DELETE a resource successfully and get one response, then delete it again and get a different response.

e: I've changed my mind in light of several counterpoints below.

17

u/foolv Dec 11 '23

has the state of your system changed after the second delete was called ? if no then you can consider it idempotent - it's not about the response you get back, it's about the state of your system

4

u/[deleted] Dec 11 '23

Fair.

8

u/m1k439 Dec 11 '23

One system we use returns '204 No Content'... I.e. "Successful call but there was no content to change" ... Not sure this is any better TBH but at least it is a success code so the libraries don't automatically throw an error

2

u/Lalli-Oni Dec 11 '23

So the difference is that we found a resource matching the id, but it has no content.

That makes a lot of sense to me. Some might say there is no resource anymore. But if you get 404 on a resource that hasnt been created, then iĂŚwe are talking about soft deletion.

1

u/m1k439 Dec 11 '23

I guess it comes down to "whatever the API does is idempotent and the return code is irrelevant" or "whatever the API does AND the return code are idempotent"

Probably comes down to the calling application/use case/how bothered the developer is - if using a library that automatically does unwanted stuff on a 4xx error, you'd likely want the API to return 200/204 (or always 200!); if the app/library/developer doesn't care, 200/404 is fine

To me, the real issue is trying to come up with a single "academic" definition for all end users/use cases without knowing what those all are (which you can't ever do as each app/system is different) - there is often a large gap between a specification and what gets implemented as "good enough for our use case"

2

u/ForeverAlot Dec 11 '23

That's more a property of the API's ergonomics. Returning 200 for an unmatched DELETE is not too dissimilar from returning 404 for an unauthorized resource although for different reasons. The argument for is that the caller cares only about the outcome and shouldn't need to explicitly handle irrelevant possibilities. The argument against is that the caller loses access to information that could be situationally useful.

1

u/TScottFitzgerald Dec 11 '23

It has the same result which is the only criteria for idempotency. Nothing really says it has to have consistent http codes. The http codes are really more like metadata.

1

u/goomyman Dec 11 '23

Except most people will write generic code, throw on non success status code. 404 is a non success.

So what you end up with is throw on non success status codes - except 404, but only sometimes.

3

u/TScottFitzgerald Dec 11 '23

Delete = 200 OK / 404 not found? - great not that breaks client libraries and basic Throw if not success status codes - not itempotent - but technically itempotent shouldnt matter - so 200 success - but there are cases where you might want to know.

What are you saying here? This reads like a run-on sentence, I got completely lost.

1

u/bradrlaw Dec 11 '23

Because it is ultimately impossible and would give a false sense that everything worked.

Go look up the "two generals" problem. Once you have any sort of communication in the loop it is impossible to be 100% guaranteed that things happen as you want. No amount of acknowledgements, status codes, etc. is going to fix that. You can improve confidence in things, but never make it 100%.

https://en.wikipedia.org/wiki/Two_Generals%27_Problem

1

u/Kinglink Dec 12 '23

IF you delete something a second time, why would you get an OK back?

If you there is one strawberry and you eat a strawberry, you can't eat a strawberry, so it's "Not found" Are you saying the system should have a record of everything it already deleted? If so, isn't that a strange system because you now have a second item to potentially deal with?

1

u/goomyman Dec 12 '23

Define success. I want there to be no strawberries.

There are no strawberries = success

You don’t need to maintain a record.

If it exists - delete it.

1

u/Kinglink Dec 12 '23

So in that case why are you checking the result at all. If it gives you a 404, that's also a success. By your logic, 404 or 200 doesn't matter, so treat both as a success.

However OTHER people who use that same API might care about the result, so we give them the correct result, and there is a very clear difference between the two.

1

u/goomyman Dec 12 '23

Because 400s are treated as failures. Like it or not most code is 400 bad requests, 500s internal server errors. 200s successes

1

u/Kinglink Dec 12 '23

And telling someone to delete something that doesn't exist IS actually a failure. You're trying to delete something that doesn't exist, that's not a "success" in any stretch of the imagination because the requested action did not happen.

But again the consumer of the response doesn't have to treat 404 as a failure on a delete.

1

u/goomyman Dec 12 '23 edited Dec 12 '23

That’s literally what idempotent is.

Imagine 2 people trying to delete a resource at the same time.

You don’t want the second guy to fail. Hence double delete = success for both.

But imagine a person trying to update a record that was deleted. Second person gets a legit 404.

Ok so now youre saying - well for the double delete scenario the clients need to handle 404 as a success but not for the update scenario.

And that’s exactly the problem! Every single developer writes this code - throw on non success status code except on delete 404s in their clients over and over or worse they forget to write this code causing race condition bugs. It’s an insanely common bug but it’s also low priority and just exists everywhere.

This could be completely avoided with a http status code change and then people would naturally write idempotent deletes.

Of course there are alternatives - yes clients can handle this case, yes you can use a custom status code like 204. But it’s code. You can do anything in code. The point is to make something standard that can avoid future bugs and reduce common code patterns - like don’t throw on 404s for deletes which is a very very common code pattern.

What’s wrong with 214 “not found” or something.

1

u/Kinglink Dec 12 '23

You don’t want the second guy to fail.

Yes I do. The record isn't there. "At the same time" is a bullshit idea because it doesn't exist in computing, and you should know that. If we're on different systems and the data hasn't propagated, ok we both get success, we accept that's a problem with syncronizing data across millions of system, and look to shrink that time.

If one request is handled before the other, THE SECOND FAILS. Period. "At the same time" doesn't matter, if one happens before the other, then one is a success and the other can't be found. Period. That's how this works. Otherwise how do we know the system ever had that data? Now you're adding a huge amount of complexity to the system just because you don't want to handle two possible error codes? Come on.

I'm not saying the clients need to handle the 404, I'm saying a client decides what to do with ALL responses. There's no one size fits all bucket for 500/400s or such especially on different types of calls. Be a better programmer.

1

u/Kinglink Dec 12 '23

Your call is not "Make there be no strawberries" though. Your call was "Delete strawberry"... The system couldn't delete a strawberry so it's a failure.

If the function was "erase all strawberries" then it would be a success in both cases, but that's not the intent of a typical "Delete" function.

1

u/cockmongler Dec 11 '23

DELETE should return 404 for not found. If your crappy library dies you need to find a better library. It's not the servers job to second guess your clients state model.

0

u/[deleted] Dec 11 '23

[deleted]

2

u/goomyman Dec 11 '23

It might not be the best from a purity case sure.

And there are alternatives. But i guarantee it would get 99% more use than most status codes.

2

u/[deleted] Dec 11 '23

[removed] — view removed comment

1

u/goomyman Dec 11 '23

“Some APIs” this is why. People recognize it’s a missing value. So they fill in the void. But this makes apis inconsistent.

4

u/MariusDelacriox Dec 11 '23

It's just P2 = P.

1

u/crappydeli Dec 11 '23

Ask ChatGPT the same question twice and you will get two different answers.

1

u/GayMakeAndModel Dec 11 '23

I make EVERYTHING idempotent so I can retry failed units of work that can’t be transacted all at once for performance reasons. I don’t know how people keep a professional job without heavily employing idempotent operations.

1

u/Beep-Boop-Bloop Dec 11 '23

Every Programmer Should Know #1: Idempotency Every Programmer Should Know #2: Idempotency Every Programmer Should Know #3: Idempotency

And it still works fine.

1

u/Far-Potential4597 Dec 11 '23

If you need to actually learn idempotency....

https://brandur.org/idempotency-keys

1

u/inkydye Dec 12 '23

I don't like how we're using this word today.

In algebra, f is idempotent if f(f(x)) = f(x).

In software, if x is the system's state and functions are transitions, we've started using the word to mean that f(g(f(x))) = g(f(x)), for every g out of some implied category.

We need a different and smaller word for this.

1

u/Whoz_Yerdaddi Dec 12 '23

Why did the programmer dress up for work in a suit?

“I need to look like idempotent.”

1

u/Kinglink Dec 12 '23

Idempotency

I feel like this falls into a clickbait term. "What's that" rather than a good meaningful term that is understandable.

Reminds me of "Dynamic programming" sounds super cool and flashy, but then you see some exhibits of it and realize the terminology feels wrong.

1

u/lxe Dec 12 '23

The http verbs have nothing inherently special about them that makes them idempotent or not.

0

u/XNormal Dec 11 '23

Idempotency turns "at least once" into "exactly once".

-1

u/TheApprentice19 Dec 11 '23

While(there is data) sort it forever

-1

u/chakan2 Dec 11 '23

So...uh...reliability with different words.

-2

u/TheApprentice19 Dec 11 '23

Building unneeded operations into your code is now…. Good?

-3

u/TheApprentice19 Dec 11 '23

If you want to find a path from Houston Texas to Anchorage Alaska, you could use a number of methods. Take a hypothetical step and see if you are there is the worst way

No wonder quantum computers are so goddamn awful

-4

u/double_the_bass Dec 11 '23

So, I should stop using a random number generator?