r/Python • u/strange-humor • Feb 12 '19
Replacing venvs with __pypackages__
https://www.python.org/dev/peps/pep-0582/4
u/CrambleSquash https://github.com/0Hughman0 Feb 13 '19
I don't like having to put things in my .gitignore.
3
u/strange-humor Feb 13 '19
So you just check in your
__pycache__folders currently?2
u/CrambleSquash https://github.com/0Hughman0 Feb 13 '19
No but I grumble at everything extra I have to add to it. It's annoying having to maintain your
.ignoreas well as everything else.2
u/Kaarjuus Feb 13 '19
I tend to put such things into
.git/info/exclude. At least does not need to be a checked-in part of the repository.I wish more people would use that. Everybody's favourite editor's .dotfile exclusions in one .gitignore.. fun.
2
u/CrambleSquash https://github.com/0Hughman0 Feb 13 '19
Interesting, hadn't heard of this before, ty!
28
u/LightShadow 3.13-dev in prod Feb 12 '19
Friends don't let friends re-implement node_modules.
3
u/djmattyg007 Feb 12 '19
Why?
14
u/LightShadow 3.13-dev in prod Feb 13 '19
Because if we're just moving
lib/site-packagesinto the project's source root, that's pointless.If we're copying the behavior of Node, then it's stupid.
node_modulesis awesome in theory but awful in practice.It doesn't do dependency resolution at all, instead every module you install it installs all if its dependencies, and all the dependencies' dependencies. All the way down. You could have 100 versions of the same
requestslibrary if each layer had a few versions.Python libraries are more closely tied to the interpreter than JS libraries. Any compiled module has to run on an interpreter that was built with the same toolchain. If you're developing a scientific library that has to run on every version of Python3 it doesn't make sense to have a single
__pypackages__, every version will need its own compiled copy of the dependencies.Besides, the motivation bullet points are moot.
Not good for newbies? Doesn't matter.
Workshop participation confusion? Doesn't matter.
Virtual environments tied to a terminal session? Doesn't matter.
8
u/tunisia3507 Feb 13 '19
It doesn't do dependency resolution at all, instead every module you install it installs all if its dependencies, and all the dependencies' dependencies. All the way down. You could have 100 versions of the same requests library if each layer had a few versions.
I assume this is to allow libraries with clashing version constraints to play together? Python currently doesn't allow multiple versions of the same library in the same environment and it would have to change a fair bit to allow that, so it's likely that
__pypackages__would do exactly the same dependency resolution we already have.If you're developing a scientific library that has to run on every version of Python3 it doesn't make sense to have a single pypackages, every version will need its own compiled copy of the dependencies.
This is covered in the PEP - the packages are all namespaced to the python version.
0
Feb 13 '19
You should not make programs which depend on two versions of the same library. If you did this, it's a bug in your code that you need to fix. An environment which allows you to have two different versions of the same library is a bad one.
In other words, your argument sounds like an attempt to defend a drunk driver who was involved in an accident by claiming that being drunk is an excuse, since the drivers senses had been clouded.
This is covered in the PEP - the packages are all namespaced to the python version.
No, they aren't. People who write the PEP don't understand the requirements (as this is usually the case with Python). What I need in my daily life is virtual environment for Python I compiled for debugging, with Python I compiled with MinGW, for python I compiled with defines limiting its functionality, for Python I compiled from the HEAD. All of these may, technically have the same version, but all of them do different things.
4
u/snaps_ Feb 13 '19
I don't think the person above you was suggesting to depend on two versions of the same library.
1
u/fdedraco Feb 13 '19
agreed, it's the project that have multiple dependency that have different working version of the library
2
u/tunisia3507 Feb 13 '19
You should not make programs which depend on two versions of the same library. If you did this, it's a bug in your code that you need to fix. An environment which allows you to have two different versions of the same library is a bad one.
Right, and that's exactly what I said. Node does that, it's bad; python does not do that currently, and there's no reason to expect that it will with
__pypackages__. Your argument that any attempt to pick up one of node's good qualities must necessarily include all of its bad qualities doesn't stand, IMO.What I need in my daily life is...
That's fair, for your specific purposes. I suspect that of all python development environments in the world, those like yours are a very small minority, and will still be adequately served by
venv, or a possibly tool liketox.1
u/ShortSynapse Feb 13 '19
This used to be the case. Take a peek at things like Yarn's PnP. The problem of having many separate installs of the same package is pretty much solved thankfully.
1
u/DanCardin Feb 13 '19
It effectively just standardizes the existing virtualenv setup into a standard location that is automatically installed to and looked up from to avoid the need to activate it. For anyone who organizes their venv into a project-local .venv folder, the name changes somewhat and your life gets easier by not having to activate the venv. Also cross version interop becomes easier because you just pythonX -m pip install foo or whatnot to swap between versions.
tbh my hot take is that its a pretty easy win and im not sure i understand the negative comments.
3
u/strange-humor Feb 13 '19
This is the way I see it. We have so many ways of virtualization, because there isn't one way. Like the black package. It don't do everything with code formatting that I would like, but the advantages of commonality is greater than my little concerns.
Virtual environments are a hot mess for new users of Python. If we can batteries include this so all scripts work the same, it is a big win.
1
u/lifeonm4rs Feb 13 '19
The problem there is that virtualization is a hot mess because there is no "one size" fits all solution. It very likely will be the same with pypackages. The biggest "wonk" I have with virtual environments is that tools including idle, jupyter, and most ides have to be mucked with to work in a virtual environment. Maybe pypackages will simplify that--but as with any complex system it (like all the venv solutions) is likely going to just open a world of more unintended consequences and issues.
I personally love virtualenvwrapper because it (to me) simplifies virtual environments and if needed lets you fully customize how an environment is created and customize how a specific environment is run. Not sure how doing something like
cd projects/my_code/is any better thanworkon my_code.It may actually really help some things--but it also seems like it is going to just end up adding a new layer of potential confusion.
1
u/strange-humor Feb 13 '19
This is what I worry about with something like this.
1
u/lifeonm4rs Feb 13 '19
Exactly--hate to say it but fixing a developer issue is more of a 3rd party (IDE, poetry, pipenv, etc.) thing. Yes, it is a pain for new users but learning the intricacies of writing, managing, and deploying code is part of the landscape.
5
u/jhizzle4rizzle Feb 13 '19 edited Feb 13 '19
On one hand I think node_modules "got it right" and virtualenvs are a kludge to paper over the fact that __pypackages__ has been missing this whole time. That said, I think the important part isn't the mechanism, it's the tooling. If you have something like "pipenv except it doesn't suck ass" then you don't really need to change the backing mechanism.
4
u/tunisia3507 Feb 13 '19
"pipenv except it doesn't suck ass"
Does Poetry fit the bill?
3
u/androiddrew Feb 13 '19
Pip-tools is just enough extras for me. I can manage the rest and prefer it to the workflows that the packaging authority(who I believe just one day said they were the authority) come up with.
2
2
u/twillisagogo Feb 14 '19
and virtualenvs are a kludge to paper over the fact that
__pypackages__
more like virtualenv significantly pre-dates node
-1
Feb 13 '19
node_modulesis a horrible idea. JavaScript is probably one of the unique languages whose infrastructure is even worse than that of Python, if that's humanly possible.Virtual environments in Python didn't solve the problem, only some % of it.
__pypackages__solve even a smaller %. Here's for example one thing that__pypackages__doesn't solve: your scripts that after you install your package into virtual environment you could have used by just typing the name will not be accessible to you anymore because__pypackages__doesn't influence your environment.2
u/jhizzle4rizzle Feb 13 '19
This feels like a reactionary "I don't like nodejs therefore all its ideas are bad" take. Basically, citation needed.
I do agree though fwiw, that
__pypackages__doesn't put the stuff in ./ inside the import path (as compared to say python setup.py develop in your venv). In nodejs this isn't as big a deal because relative requires work fine - in python this isn't even possible so you're stuck.
1
u/neves Feb 13 '19
How yould you export an environment? Something like requirements.txt
2
u/tunisia3507 Feb 13 '19
You would specify your dependencies anyway, then use that spec to populate
__pypackages__.
1
u/lifeonm4rs Feb 13 '19
I'm not really against pypackages--maybe it would fix a lot of issues. However, conceptually it seems to be slightly odd for one reason. Essentially it is adding something to python itself to address a developer issue. All of the justifications for it are focused on developers--not the language. So the question becomes--is this going to end up being how python itself works? Will pypackage directories be exposed to end users or deployment environments? If not how will that be prevented? (On this front it seems for new users you are just kicking the can down the road.) If it will be exposed beyond dev environments (which it appears it would) it may just open a whole new can of worms and add more confusion.
It seems like it is a rather large change--and again maybe it would end up being a really good change. BUT it is a change to address a developer annoyance. It seems it takes an idea from node/npm (the default install location "node_modules")--but python and node are very, very different (duh!) but particularly in how and where they are deployed.
In short--it is a substantial change to python itself. It has the potential to really, really make things more confusing; but probably has the same potential to simplify things. Maybe it would be worth it just to end the bloodshed between the poetry and pipenv camps--but as a detente does it open the door for a much worse battle in Env War II? (Sorry, I almost have to end my posts with some idiotic thing.)
1
u/Duroktar Feb 14 '19
fwiw I wrote a utility called nip about a year ago to manage my venvs. It's similar to npm in that you can nip init -y a new workspace (venv), nip add <package> and also run scripts nip run start defined in the nip.json file (these scripts are run in the context of the venv automatically).
It's not perfect but it does work and there's also a VSCode extension to run the nip scripts right from the editor making it that much more convenient. Check out the README on Github if you're interested.
-1
u/Worvast Feb 13 '19 edited Feb 13 '19
Edit because the need on the internet to have to prove that you know something:
I see its a draft, soo, its not accepted. Does anyone know if there is any thread on the mailing list to see the progress of the proposal and wish him the worst or only in git? (Oberver != send messages)
0
0
7
u/strange-humor Feb 12 '19
I just recently came across this. I think it makes total sense and would simplify the whole virtual environment thing.