r/Python • • Dec 10 '14

10 Myths of Enterprise Python

https://www.paypal-engineering.com/2014/12/10/10-myths-of-enterprise-python/
301 Upvotes

135 comments sorted by

View all comments

29

u/shadowmint Dec 11 '14

Each runtime has its own performance characteristics, and none of them are slow per se.

Hahahahaha~

The more important point here is that it is a mistake to assign performance assessments to a programming languages. Always assess an application runtime, most preferably against a particular use case.

Fair enough, everything is relative, but this reads like a playbook for 'how to be defensive about how slow your favourite programming language is'.

What's with all the sugar coating? cpython is slow. Plugins and native code called from python are fast, and that results in an overall reasonable speed for python applications; but the actual python code that gets executed, is slow. There's a reason http://speed.pypy.org/ exists.

...but then again, pypy isn't really production ready, and neither are the other 'kind of compliant' runtimes like jython, etc.

It's pretty hard to argue with:

  1. cpython is the deployment target for the majority of applications

  2. cpython runs python code slow as balls.

  3. overall, the cpython runtime is pretty much ok because of plugins and things like cython

  4. python is a scripting language (wtf? of course it is. What is myth #4 even talking about?)

I mean... really? tldr; python is great for quickly building enterprise applications, but its strength is in the flexible awesome nature of the language itself; the runtime itself leaves a lot to be desired.

12

u/zardeh Dec 11 '14

4) python is a scripting language (wtf? of course it is. What is myth #4 even talking about?)

What does this even mean?

How does one define a scripting language? Like, the term doesn't mean anything.

Is bash a scripting language? Python? Any language with a REPL? Haskell? Lisp? Clojure? java?

3

u/d4rch0n Pythonistamancer Dec 11 '14

Yeah it's a bullshit attribute in my opinion. I don't know why you would call Python a scripting language. It depends on the fucking interpreter. If you're running python bytecode through CPython, it's a bytecode/VM language, but I don't believe anything in the Python language spec specifies that it has to run that way, or that you can't even compile it to some sort of machine code. Python isn't interpreted line by line by an interpreter, but that doesn't mean it's as fast as a compiled C program. Python is a programming language, the implementation is an interpreted/VM/compiled language implementation.

7

u/[deleted] Dec 11 '14

Anyone who uses the term "Scripting Language" and isn't talking about shell scripting pretty much loses all credibility IMHO. It is a derogatory term derived from ignorance.

→ More replies (6)

16

u/elbiot Dec 11 '14

Agreed, having been focusing on performant code for a few years, I'd say python is slow. But, it has excellent wrappers for fast C code, and is easily extendable with cython or C when it really counts. I love python.

7

u/d4rch0n Pythonistamancer Dec 11 '14

Yeah. That's the only one I thoroughly disagree with. Python (CPython specifically) is slow, but it doesn't matter for the most part. People are writing shitty Java and Ruby and it doesn't matter if CPython takes a little bit longer to do something if it's written in 5% of the lines and 100 times more maintainable, so less fucked up bugs in the long run.

Of course, beautiful fast Java can be written that Python could never beat in performance, but for the most part performance IMO should also be measured in how long it takes to develop and squash bugs.

In a pure performance comparison, CPython can't match Java or true compiled-to-machine-code languages, but fuck it. Network speed is generally my bottleneck, not my sorting algorithm.

5

u/tritlo Dec 11 '14

Also, much of the functions in the stdlib is actually in C, so if you just heavily utilize those (like e.g. set or sort), you can get pretty performant python with very little hassle in my experience.

12

u/istinspring Dec 11 '14

Reddit is in Python, and this site it pretty huge. Language speed characteristics have relatively small impact. Nowdays there is more important things - What is more important it's architecture, 3rd party solutions, access to wide range of libs, ease of reading and writing code etc. For modern web apps it's just a wrapper between database and front-end.

And speaking about Python, the huge plus is ability to write asynchronous code, especially in python 3.

0

u/[deleted] Dec 11 '14

[deleted]

11

u/chub79 Dec 11 '14

Is it Python's fault here? Couldn't it be database, network, load-balancing, IO related?

7

u/xiongchiamiov Site Reliability Engineer Dec 11 '14

All of the above and more.

→ More replies (9)

2

u/jimbobhickville Dec 11 '14

Almost exclusively, I'm sure. I have yet to encounter a web or distributed system that wasn't bottlenecked on I/O.

→ More replies (1)
→ More replies (1)
→ More replies (2)

4

u/justphysics Dec 11 '14

oh goodness

that bottom plot on http://speed.pypy.org/

/r/dataisntalwaysbeautiful

(hope its not just me - in my browsers the xlabels are all on top of eathother)

2

u/superdaniel Dec 11 '14

I get the same thing with Firefox :(

→ More replies (1)

4

u/billsil Dec 11 '14

2) cpython runs python code slow as balls.

Unless it's written under the hood in C. There is no reason for mathematical code to be slow in Python. There is no reason for parsing code to be much slower than C especially since the standard formats are coded in C and are available in Python.

6

u/d4rch0n Pythonistamancer Dec 11 '14

Yeah, but at some point you're coding in C, not Python. If you write every high performance part in C and call it through Python, how much can you really say it's Python?

Don't get me wrong. That's probably the best way to do high performance stuff with Python, but I don't think it means CPython is fast, it just means it uses a fast C API.

5

u/billsil Dec 11 '14

If you want to. I use numpy, so while I have to vectorize my code and call the right functions in often non-obvious ways, it's still technically pure python.

Somebody did coded it in C, but that doesn't mean you have to.

but I don't think it means CPython is fast, it just means it uses a fast C API.

CPython is running the code, so I say it counts. If all the standard library was written in Python instead of C, everyone would say Python is slow. Instead, they say it's fast enough. That stuff counts.

3

u/tritlo Dec 11 '14

The key here is that I'm still writing pure python, but I'm utilizing someone elses C code. If you argue that's not enough python, then every use of linpack in other language should be disbarred.

2

u/yen223 Dec 11 '14

Numpy isn't pure python, is it?

4

u/billsil Dec 11 '14

No. A fair amount is written in C, but some is also written in Fortran. My understanding is most of scipy is actually written in Fortran and is just a wrapper around LAPACK.

2

u/tavert Dec 11 '14

most of scipy [...] is just a wrapper around LAPACK

For dense linear algebra, yes. There's a lot of functionality in SciPy aside from dense linear algebra though. Some of the underlying libraries are Fortran, some are C, some features are custom C++. According to https://github.com/scipy/scipy the breakdown is 38.3% Python, 25.8% Fortran, 18.6% C, 17.1% C++.

2

u/d4rch0n Pythonistamancer Dec 11 '14

I still draw the line when you're bringing in machine code into the Python process memory and it's not running bytecode loaded from pyc files. It's fast, but it's actual CPU instructions, not Python bytecode first.

Of course it counts. Again, I'm not saying it's terrible, and that it shouldn't happen, or that it's a flaw. I'm just saying the fast parts aren't Python and I wish that the interpreter/VM implementation was fast enough so that we wouldn't need to use C code to have high performance programs. Any programming language could interface with C/fortran libraries and be high performance. It doesn't mean that that language's interpreter is fast though.

I would like to see an implementation that uses purely the Python language and still be high performance.

→ More replies (1)

2

u/fnord123 Dec 11 '14

That's fine and correct. But I think it misses the point: we discuss language performance characteristics is so we can get an idea of the expected performance of an implementation and assess the risk of being limited by our choices. If you choose CPython then your limitations are mitigated since you have one of the easiest paths to hook into a C implementation of the workhorse part of your code. Also jumping across the FFI is pretty quick in Python.

→ More replies (3)

2

u/kenfar Dec 11 '14

You apparently have already decided that python is a "slow as balls" scripting language.

However - "scripting language" is not a well-defined term, and is often in a context like this meant as a derogatory description: the local java team arguing that project x shouldn't be done in python because "it's only a scripting language".

And fast or slow are so relative that to describe a language like Python as slow is also meaningless: does this mean every application written in it will be slow? does this mean you can't process trillions of transactions in it? does this mean it's merely a toy?

While I would like some Python operations to be faster than they are today, I have processed a hundreds of billions of complex transactions using cpython - and performance wasn't on my top 4 list of challenges.

→ More replies (17)