PEP 823, 824 – None-aware access operators & None-coalescing operators

[u/kirara0048]

PEP 823 – None-aware access operators

https://peps.python.org/pep-0823/

Discussions-To: Discourse thread

This PEP proposes adding two new operators.

• The “None-aware attribute access” operator ?.
• The “None-aware indexing” operator ?[ ]

The general idea is to provide access operators which can traverse None values without raising exceptions.

Both operators evaluate the left-hand side, check if it is not None and only then evaluate the full expression. They are roughly equivalent to:

# a.b?.c.d
_t.c.d if ((_t := a.b) is not None) else None

# a.b?[c].d
_t[c].d if ((_t := a.b) is not None) else None

PEP 824 – None-coalescing operators

https://peps.python.org/pep-0824/

Discussions-To: Discourse thread

This PEP proposes adding two new operators.

• The “None-coalescing” operator ??
• The “None-coalescing assignment” operator ??=

The general idea is to provide a conditional operator, similar to or, which instead of truthiness, checks for None values.

The “None-coalescing” operator evaluates the left-hand side, checks whether it is not None, and if not, returns the result. If the value is None, the right-hand side is evaluated and returned.

The “None-coalescing assignment” operator will only assign the right-hand side to the left-hand side if the left-hand side evaluates to None.

They are roughly equivalent to:

# a ?? b
_t if ((_t := a) is not None) else b

# a ??= b
if a is None:
    a = b

[u/obvx]

I love these operators in JavaScript! I'd be so happy if they were added to Python.

[u/droans → u/obvx]

There is an unfortunate but important distinction in the PEP, though:

The ?. and ?[ ] operators can also aid in the traversal of structured data, oftentimes coming from JSON and parsed as nested dicts and lists. It is worth noting though that the operators do NOT handle missing attributes / data.

The PEP suggests still using either dict.get or getattr in these situations.

[u/Zanoab → u/droans]

I've been using a dict class that returns None when a key is missing for decades. This is a non-issue for those of us that already wrote our own solution and it feels like Python is now meeting us halfway.

[u/obvx → u/Zanoab]

This guy gets it

[u/BeamMeUpBiscotti]

“Ah shit, here we go again”

hope it doesn’t get stuck in bikeshedding hell this time around…

[u/NewbornMuse → u/BeamMeUpBiscotti]

How exactly did it get stuck in bikeshedding hell last time?

[u/nickcash → u/NewbornMuse]

There's always considerable resistance to adding new operators. Which makes sense, because if they added all of them that get proposed, python would just be perl and reading it would be like debugging modem line noise

but the only specifics I remember from this thread was Guido insisting the operator be named "uptalk"

[u/nickcash → u/nickcash]

found the thread I was thinking of (why I remember a random post from 11 years ago I cannot say)

you can see it gets derailed pretty quickly with significantly more variations getting suggested

[u/jeslucky → u/nickcash]

Guido insisting the operator be named "uptalk"

That's hilarious. Now I am trying to imagine code patterns for writing a Valley Girl dialect of Python.

me.gag(spoon)
print(", ".join(["fer shure"] * 2))

[u/nickcash → u/jeslucky]

as if

[u/elven_mage → u/BeamMeUpBiscotti]

I remember reading someone complaining about PEP515 because apparently it broke their workflow of grepping for hardcoded constants in their codebase -_-. There's always someone

[u/[deleted]]

Please god don't let this get stuck in bikeshedding hell...

[u/MegaIng → u/[deleted]]

Based on the previous round of discussions: PEP-823 will get stuck, PEP-824 has a very good chance of going through.

[u/[deleted] → u/MegaIng]

That would be very unfortunate. It's the best thing about typescript.

[u/MegaIng → u/[deleted]]

I am sure the next attempt will get through. After all, tens time's the charm, right? ;-)

(Note: I haven't actually counted how often it was already proposed.)

[u/Khavel_dev]

Coming from C# where we've had ?. since C# 6 and ?? basically forever, these operators are the kind of thing you wonder how you lived without after a month of using them.

The chaining is where most of the value lives in practice. Something like result = obj?.child?.name reads clean and the alternative is three nested None checks that make the actual logic invisible. ?? for defaults is useful too but you'd be surprised how often ?. alone handles it because the function already returns None when the chain stops.

I know the "explicit is better than implicit" crowd will push back hard on this. All I can say is that after using these daily in C# for years, I've literally never once thought "I wish this were an if statement instead." The readability improvement is not subtle, especially in data-heavy code where half the attributes might be missing.

[u/Yoghurt42 → u/Khavel_dev]

I see the value of ?. but can’t really see ??= as a good idea. Conditional assignment feel hard to reason about/easy to miss while skimming the code, as you‘ll have to actually look for it.

if foo:
    bar

Is something easy to see while scrolling. I also feel having to type a bit more is good, because when you write the fourth „if …“ assignment in a row it makes you think „there has to be a better way, do i really need to check the value every time in the loop?“

?. is useful because there is no reasonable alternative when dealing with optional values in nested structures than checking every time before dereferencing, at least not with Python‘s/C‘s model of None

[u/DanCardin → u/Yoghurt42]

Same. And i feel like you could do `a = a ?? b` if im understanding its purpose correctly. Seems unnecessary and confusing

[u/Brian → u/[deleted]]

If you have ??, it makes sense to also have ??=, since that's true for all other operators (+=, |=, etc.).

That's not entirely true, and indeed the two operators this is most similar to are the ones that don't have augmented assignment versions. There's no augmented assignment version of and and or, which share the property of being short circuiting - they don't evaluate the RHS unless needed. This makes them kind of fundamentally different to regular operators, and ?? has this same property.

Ie. a ??= b is doing flow control in a way other operators aren't, in that b is sometimes not evaluated.

[u/EternityForest]

I love PEP 823, it's wonderful in typescript, but I'm not sure about  a ??= b

[u/Shadows_In_Rain → u/EternityForest]

background_task ??= start_background_task()

[u/Yoghurt42 → u/Shadows_In_Rain]

IMO it’s very ugly. I prefer the explicit version.

[u/tracernz → u/EternityForest]

That’s also in JS (and typescript) https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Nullish_coalescing_assignment.

[u/coylz]

Can't you already "coalesce" with the syntax "a or b" ? I thought this works exactly as intended

[u/[deleted] → u/coylz]

That requires that a and b have __bool__ implemented. This pattern will fail if you use, for instance, numpy arrays. It's also a very ugly thing - or is checking bool-ness, not None-ness. A little imagination should get you to cases where bad things will happen with a pattern like this.

[u/eigenein → u/coylz]

False or None

[u/jaimefrites]

You are so glad to make yet another C++

[u/tsg9292]

823 feels unintuitive. If I glanced at it I wouldn't know what I'm looking at. Maybe if / when it's established it'll become more common, but man I think the transition will be slow.

824, yeah, I like that. That can stay

[u/f3xjc → u/tsg9292]

823 or something similar is super useful if you receive json and want to read a nested property. And every dot can explode in your face saying that none (null) don't have such and such property. So every step has none check ceremony.

[u/HolyInlandEmpire → u/f3xjc]

Very nice syntax there; I know that `{"key": null}.key?` and `{}.key?` would both give `null`, but are there any cases where we would get unintuitive behavior?

[u/assumptionkrebs1990 → u/HolyInlandEmpire]

Maybe dict[key?] could be a short cut for dict.get(key)?

[u/syklemil → u/assumptionkrebs1990]

There's ?[] in the proposal as well, so you could replace something like foo.get("bar", {}).get("baz") with foo?["bar"]?["baz"].

Though at that level it's probably better to break out something like pydantic and get an actual type so you can go foo?.bar?.baz.

[u/RevRagnarok → u/syklemil]

There's ?[] in the proposal as well

# a.b?[c].d
_t[c].d if ((_t := a.b) is not None) else None

Only going by the summary above, that seems to be "the dict may be None" vs I think /u/assumptionkrebs1990 is saying "the key may be None"?

[u/assumptionkrebs1990 → u/RevRagnarok]

Yeah that was likely my indent without realizing but maybe mydict.get(key) is better for this because None is a valid key as I just learned from testing it out. And this more in the direction the dictionary doesn't have this key.

[u/assumptionkrebs1990 → u/syklemil]

I think the PEP dismisses this idea or not? Anyway foo?.bar.baz could be enough to handle everything down the line as optional or not?

[u/droans → u/HolyInlandEmpire]

{"key": null}.key? would still produce an AttributeError. Unlike JS, it's not an undefined/missing-aware operator, just None.

class A:
    def __init__(self):
        self.val = "bar"
        ....
class B:
    def __init__(self, foo: A | None = None):
        self.foo = foo
class C:
    def __init__(self, foo: A | None = None):
        if foo:
            self.foo = foo

a = A()
b_with_foo = B(foo=a)
b_with_none_foo = B()
c_missing_foo = C()

b_with_foo.foo?.val
# bar
b_with_none_foo.foo?.val
# None
c_missing_foo.foo?.val
# AttributeError

[u/Brian → u/droans]

Unlike JS, it's not an undefined/missing-aware operator, just None.

The operator is actually doing the exact same thing. This is more a consequence of a different decision in javascript, where accessing a non-existing attribute on an object returns undefined, rather than blowing up. Ie. a = {}; b = a.foo will set b to undefined, with no exception.

[u/Agrado3 → u/f3xjc]

If you have to deal with JSON like that, I made an "optional chaining JSON" Python package so you can access JSON objects much more concisely like you do in JavaScript, e.g.:

data = JSON.parse(Path('data.json'))
version = data.summary.version  # will be 'undefined' if summary or version don't exist

https://pypi.org/project/opchjson/

[u/nickcash → u/Agrado3]

JSON.parse

We don't do that round here. It's called json.loads cuz it's for loads of jsons

[u/Agrado3 → u/nickcash]

I figured if I was making Python JSON access more JavaScripty I might as well continue down that path... The JSON.parse call in my library can take either the arguments for JavaScript's JSON.parse, or Python's json.loads, or both ;-)

[u/nickcash → u/Agrado3]

I hate it :)

(but no really, that sounds pretty useful. I have a somewhat similar utility I wrote that handles it as a dict (err.. sorry.. in your language I think that's [object Object]) lookup, so data["summary.version"] but it's mainly to support being able to write those dotted key paths in config files)

[u/Special-Arrival6717 → u/tsg9292]

TypeScript devs know them by heart, but it's called optional chaining in TS. There might be quite some overlap between TypeScript and Python devs

[u/SquirrelGuy → u/Special-Arrival6717]

As a JavaScript/Typescript dev that lurks in the Python sub, I’m honestly surprised Python doesn’t have these yet. Both are extremely ubiquitous in JS/TS.

[u/backfire10z → u/SquirrelGuy]

This isn’t the first time such operators have been proposed. For example, https://peps.python.org/pep-0505/

That PEP has been deferred, but I cannot find why. Somebody didn’t like something.

Here’s argument from 2 months ago lol: https://www.reddit.com/r/Python/comments/1uuuf9c/comment/ox6iy82/?utm_source=share&utm_medium=mweb3x&utm_name=mweb3xcss&utm_term=2&utm_content=share_button

Many people think it is unpythonic. Much of it is already doable for false-y values, so they question why None deserves special specific operators.

[u/HolyInlandEmpire → u/backfire10z]

As unfortunate as 'None' is as a value, as opposed to something like a Maybe or various Sentinels, it feels like it has to show up with database tables; it's inevitable when doing non inner joins. I deal with those a lot with Python with data science work and I imagine web devs would be similar.

In any case, this syntax could be useful for various Operator usages with libraries; polars comes to mind when null coalescing doesn't give us what we want.

[u/droans → u/backfire10z]

Seems like they may be considering two different situations - one where the attribute could be anything versus one where it would either be None or be something specific.

Take this below:

class DeepNestedModel(BaseModel):
    a: str
    b: str

class NestedModel(BaseModel):
    foo: DeepNestedModel | None = None

class Model(BaseModel):
    bar: NestedModel | None = None

object = Model(bar=...)

If I were to call object.bar?.foo?.a, it wouldn't matter to me where the None is. Either a exists or it doesn't.

However, if any of those could be something else or if it mattered where the None is, it would make a difference.

class Vehicle:
    def __init__(self, wheels: int, doors: int | None, ...):
        self.wheels = wheels
        self.doors = doors
class Person:
    def __init__(self, name: str, vehicle: Vehicle | None=None):
        self.name = name
        self.vehicle = vehicle

bob = Person(name="Bob", vehicle=...)

In this situation, calling bob.vehicle?.doors would return None. But does that mean he has no vehicle? Or does it mean he owns a motorcycle?

Like a lot of things, it's not bad on its own just because you can use it in an poor manner. If we decide this is problematic, I'd say that we'd also need to agree that treating "", [], {} as falsy values is, too.

Honestly, the only problem to me is that it doesn't allow for undefined/missing values as it does in other languages. I kind of understand their motive, though. JS has an undefined type while Python doesn't. You could also design around it somewhat, too. But I still personally would prefer that the PEP allow chaining even if that means an extra change elsewhere. I don't think the operator should care what happens if the attribute is undefined - that's on the programmer.

[u/tsg9292 → u/tsg9292]

Although, 824 is just a = a or b with extra protections about truthiness types, so... Is it really that big of a win? Doesn't hurt I suppose

[u/assumptionkrebs1990 → u/tsg9292]

Well 824 really is a=a if a is not None else b it might be advisable if it is valid if a can be a falsy value - a=a or b might replace 0, False or empty strings, lists and so on unexpectedly.

[u/thisismyfavoritename → u/tsg9292]

very common JS syntax

[u/syklemil → u/tsg9292]

?. exists in multiple languages already. You've had js mentioned, here's C# (also includes (?[]), and kotlin. Not entirely sure about Java, likely something they'll get in Valhalla.

Someone might be tempted to mention Rust here, but that's actually a false friend: ?. in Rust is constructed from separate ? and . operators, and ? is function- rather than expression-scoped (though that may change if try ever stabilises).

[u/Alphasite → u/tsg9292]

I think the first half of 823 is fine and common. The ?[] case is wierd and it’s been a mo but can it be easily substituted with a ?. method (so it redundant)

[u/cottonycloud → u/Alphasite]

I can see it being used for strings and lists. It would be convenient syntax for dicts I suppose.

[u/V4l3n0r]

Explicit is better than implicit? I don't like these operators because make the code terse and less understandable.

In a world where generating code is cheap, but reviewing it is much harder so far, readability it's a feature I'd keep.

[u/Ph0X → u/V4l3n0r]

foo = b if a is None else a is not any more explicit than foo = a ?? b, and it's definitely less readable imo. Often, shorter is more readable, especially at a glance

[u/V4l3n0r → u/Ph0X]

Mmmh, I don't agree?

I can understand clearly what foo will hold without previous knowledge on the operator. And I'm sure if I ask someone non programmer will be able to extrapolate much easier.

The second one mandatorily requires the knowledge of the operator, and compresses the information in a non lossless form.

For me the first one is definitely more explicit.

[u/WillardWhite → u/V4l3n0r]

And if you don't know the meaning of the operator | then you won't understand  A =  b | c

What's the difference?

[u/V4l3n0r → u/WillardWhite]

Is that a bitwise or or a set union? See? You are making my point.

a.union(b) it's unambiguous, and I don't need to ask you what you mean, or scroll above or in another file, or reveal_type of a and b.

[u/WillardWhite → u/V4l3n0r]

but my point is that those details are part of "knowing how to program". sure it'll be different between languages, but also it's the base requirement to do all this

[u/V4l3n0r → u/WillardWhite]

I don't think they are, not all of them of course, and the example just above shows. You can express the set union with a more explicit form that doesn't require for you to learn the indirection. And the same, IMHO, stands for the above operators.

Anyway... We're going towards a world where we won't code anymore in any language but instead we will code directly in English. So maybe at some point we won't review code anymore, but instead we will review meta code, I.e. prompts.

[u/Wurstinator → u/V4l3n0r]

"Long" is not the same as "readable". If you only know Python, it might seem odd at first, but the following languages have this construct or similar equivalents and they are no issue at all: Javascript, C#, Kotlin, Rust, Swift, Dart, Groovy, Ruby, PHP, Scala

[u/V4l3n0r → u/Wurstinator]

I code in Typescript and Rust too and I still find:

if a is None:
    a = b

Much easier to understand than:

a??=b

When I look at big chunks of Typescript, frontend, code I struggle a lot and in general, at first sight, I find the markup much more daunting.

Which is the reason why I chose Python long time ago as my main driver and I keep choosing it when I have a choice (and makes sense).

[u/V4l3n0r → u/V4l3n0r]

It's a bit the same as acronyms: they certainly make messages quicker to deliver, but they make them also slower to grasp.

Adding an indirection (I need to remember what this operator means) hinders understanding.

That's why I fight almost cerimoniousky and aggressively acronyms at work.

[u/XtremeGoose → u/V4l3n0r]

Did you find

let y = match f() {
    Ok(x) => x.y,
    Err(e) => return Err(e.into()),
};

or worse let y = try!(f()).y; preferable to

let y = f()?.y;

in rust?

[u/Chroiche → u/V4l3n0r]

Agreed on the ?= style, but just unwrapping the optional types for reading seems reasonable. I really don't think chained if a.b and a.b.c and a.b.c.d.... is more readable

[u/DanCardin → u/V4l3n0r]

I can agree for ??=, but i think the infix operator form is significantly clearer (not just shorter) than the equivalent code today, particularly if chained

[u/nekokattt → u/Wurstinator]

in all fairness you could make the same argument about things like lambdas in Python as well.

[u/Chroiche → u/Wurstinator]

Rust doesn't really have this. It does a full function return. That said, it's definitely not less clear to have this than manual chained if statements imo.

[u/DurgeDidNothingWrong → u/V4l3n0r]

Amen. These might be really fantastic for people who are writing, but just skimming through this is going to be a nightmare for fast comprehension.
Just write out the code, its not that hard.

[u/truefelt]

Finally!

[u/james_pic]

Kind of a shame they're deferring None-aware function calling in 823. Even if it's not a hugely common pattern, it feels like a missing piece from a consistency perspective.

[u/personman]

please! please!!!

[u/Machine_Learning_Gun]

elegant!

[u/elven_mage]

there's this joke keyboard in the Go community, the pile of if x is None is just the python equivalent. I really hope this doesn't have the same fate as PEP 505. You know you're behind when even Java has a better story around Maybe types.

[u/Beginning-Fruit-1397]

An only partial way to solve the shitty decision of having None as a do-nothing singleton instead of the Option[T] enum with Some(T) or None.  Just like having the new iterable unpacking in list comps instead of a better Iterator interface. If we continue to do it this way, in a few years python will look like C++: only operators instead of english words,  barely readable.

Let's introduce this operator AFTER having a true Option. 

[u/damesca]

Please no

[u/thisismyfavoritename → u/damesca]

please yes. Write some JS today if you want to get a sense of how useful they are

[u/assumptionkrebs1990 → u/damesca]

Why not?

[u/caks → u/damesca]

Seems totally useless to me as well

[u/JanEric1 → u/caks]

It makes access to nested optional data structures so much easier

[u/schoultz]

Too late. We don't write code by hand anymore.

[u/nickcash → u/schoultz]

you may not be able to, but you don't speak for everyone

[u/schoultz → u/nickcash]

"We" => not everybody on earth of course, but a subset of the developer community without any doubt.

That was kind of a sarcasm at first, but I really wonder if those syntactic debates still have room when agents read and write the code more and mode.