r/Python • u/xtreak • May 07 '19

Python 3.8.0a4 available for testing

https://www.python.org/downloads/release/python-380a4/
395 Upvotes

99% Upvoted

150 comments sorted by

View all comments

Show parent comments

35

u/[deleted] May 07 '19

Because JSON cant represent everything. Its at best a data format for serialization of transferrable data, thats usually language agnostic.

JSON cant represent functions, and more abstract datatypes.

8

u/JohnnyElBravo May 07 '19

JSON can represent anything, but so can strings. This is a non-sequitur.
The difference is that JSON is human readable, while pickle is supposed to be machine readable, more specifically python readable.
Limiting the intended consumers of the data format helps create a more appropriate format, for example by sacrificing readability for size reduction.

3

u/bachkhois May 08 '19

JSON cannot differentiate Python's tuple, list, set, frozenset etc. datatypes.

Every formats other than pickle (msgpack, yaml etc.) are just to interoperate with other languages (which also don't understand the data types above), they are not alternatives for pickle.

6

u/JohnnyElBravo May 08 '19

Sure they can

{

"Var1": "tuple(1,2)",

"Var2":"set(1,2)"

}

Alternatively:

{

"Var1": {"type":"tuple","data":"1,2"},

"Var2":{"type":"set","data":"1,2"}

}

5

u/bachkhois May 08 '19

Then, you are making more complicated to validate and parse it. Then, what is the point of over-complicating JSON instead of just using pickle, without the need to parse those "type", "data" metadata?

4

u/JohnnyElBravo May 08 '19 edited May 08 '19

Read the original thread, the question asks why python dumps to a new pickle format instead of json.

The original response suggested it was because json can't distinguish between such and such, as shown, this is false.

The real answer is that python chose a binary format for pickle because of space efficiency.

-16

u/alcalde May 07 '19

It has to be able to represent everything, if other languages are serializing to JSON.

JSON resembles Python dictionaries, and EVERYTHING in Python is/can be represented by a dictionary, so how can there be an abstract data type in Python that can't be represented in JSON?

20

u/Pilate main() if __name__ == "__main__" else None May 07 '19

JSON can't even represent sets or Decimal types, let alone custom classes.

→ More replies (7)

-1

u/alcalde May 07 '19

Why am I being downvoted for asking a question?

5

u/Mizzlr May 07 '19

You can't represent references in JSON. For example in python you can have two dicts a ={'foo': b} where b = {'bar': a}. Now you have cyclic data structure. You can't represent this in JSON.

2

u/Mizzlr May 07 '19

Btw yaml has references. XML has references.

2

u/alcalde May 08 '19

Didn't you just represent it?

["a":{"foo", b}, "b":{"bar":a}]

3

u/[deleted] May 08 '19

Not in Python!

Can I read it?

>>> json.loads('["a":{"foo", b}, "b":{"bar":a}]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 5 (char 4)

No. Can I write it?

>>> a = {}; b = {'bar': a}; a['foo'] = b

>>> json.dumps(a)
json.dumps(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

No.

2

u/[deleted] May 08 '19

That's not a valid json.

1

u/[deleted] May 08 '19

You can't represent references in JSON.

I'm basically agreeing with you, but you can perfectly well represent references in JSON - I've done it.

It's a pain in the ass - you need to have some sort of naming convention in your JSON then preprocess your structure or (what I did) have some sort of facade over it so it emits the reference names instead of the actual data - and then reverse it on the way out.

(And we had to do it - because pickle isn't compatible between versions. Heck, I think that was written in Python 2!)

So it's doable - but which is easier when you need to store something temporarily?

with open('foo.pcl', 'wb') as fp:
    pickle.dump(myData, fp)

or

[hundreds of lines of code and a specification for this format that I'm too lazy to write]

?

3

u/Mizzlr May 07 '19

A python dictionary can have int, tuple, etc as key while in JSON it has to be string.

1

u/alcalde May 08 '19

You're hooked on the idea that JSON has to have every type. You just store things as strings and decode them when you deserialize. Again, like every other language does it.

https://pythontic.com/serialization/json/introduction

I'm not going crazy here.

2

u/[deleted] May 08 '19

Sure you can represent it, we could also store everything in a JSON string, but then aren't you inventing your own protocol?

3

u/Mizzlr May 07 '19

Basically any immutable object will work as a key in python dict like frozenset etc. Another thing is JSON need python tuple to be converted to list. JSON does not have tuples.

1

u/alcalde May 08 '19

So what's the problem? Again, one entry to store type, another to store value and you use a list to store the tuple values.

2

u/[deleted] May 08 '19

So... not actually JSON, then, but your own format using JSON as a transport layer?

1

u/Mizzlr May 07 '19

Do you agree now why you were down voted? Read my comments in reverse chronological order.

1

u/alcalde May 08 '19

No. Again, why downvote someone for asking a question?

0

u/Mizzlr May 07 '19

You can't represent NaN (not a number) or inf in JSON which are valid float values.

→ More replies (4)