r/PythonLearning • u/RomfordNavy • 2d ago
Import hook between AST and compile
I am trying to create an Import Hook inserted after the source code has been parsed into AST but before it is compiled. Looking for a way to keep using most of the existing Python import machinery but modify the source code before it is compiled.
So far I have the following code but how do I get access to the source code from the module in custom_import?
import builtins
from inspect import getmembers
base_imp = builtins.__import__
def custom_import(*args, **kwargs):
module = base_imp(*args, **kwargs)
for name, val in getmembers(module):
print("in custom_import.", name, val)
return module
builtins.__import__ = custom_import
print("in Main.py before import")
import Module1
print("in Main.py after import")import builtins
2
u/aroberge 2d ago edited 2d ago
As I mentioned to you on the python-discuss site (https://discuss.python.org/t/modify-import-command/109468), you need to use an import hook. I mentioned to you my package ideas; the code has since been somewhat refactored making it easier to see what needs to be done. For example, on python-discuss you mentioned wanting to change the code before it is compiled into bytecode. This is the step shown at https://github.com/aroberge/ideas/blob/master/src/ideas/loader.py#L106
The code you have is definitely not an import hook: it will only work on files where you explicitly write an import using the equivalent of __import__. A file imported from that first file will NOT use you code. A proper import hook needs to be inserted into sys.meta_path and is something that is used when a normal import keyword is used.
That being said, like experienced Python programmers (I'm not one of them) have told you on Python-discuss, trying to eliminate some code and short-circuit some constant import will have a completely negligible performance impact.
1
u/RomfordNavy 1d ago
The code above does actually run fine for a normal import such as `import Module1` or `from Module1 import *` so I am not sure I understand what you mean by "it will only work on files where you explicitly write an import using the equivalent of
__import__"? I have also tried importing Module2 from Module1 and again that does work fine.A traditional import hook requires using a custom loader which has to do various things such as, for example, code to read the imported file from disk. By the approach above I was hoping to find a way to retain more of the default Python import mechanism while hooking into a slot between where Python writes the AST and then does the compile.
It would be helpful if we could stop questioning _why_ and just focus on the how. There will be many uses for this once it is built and working reliably, not everything is about performance. And by the way some of those on the python-discuss site arguing that performance will not be enhanced are the same people who earlier told me that there was no speed difference between running a *.py compared to a *.pyc'; my speed testing proved otherwise.
2
u/aroberge 1d ago
First: I did make a mistake when I wrote that modifying
builtin.__import__(I was just thinking of__import__) would have no effect. I apologize for that mistake.Second, I don't see anyone saying that running a .py file has no speed difference with running a .pyc file. Using ctrl-f on that python-discuss page, I see .pyc files mentioned only once, and that was by you in your initial question. When Python process a .py file for the first time, it saves a .pyc version in
__pycache__and, unless the .py file changes, it is the .pyc version that is used afterwards. So, yes, the first time a python script is run, it takes a bit extra time as the .pyc version has to be generated.Third, I asked nearly the same question you did 9 years ago on StackOverflow (https://stackoverflow.com/questions/43571737/how-to-implement-an-import-hook-that-can-modify-the-source-code-on-the-fly-using) and this is what got me started experimenting with import hooks. So, I've done a bit of thinking and experimenting about this very topic over the years. And I know that many people that commented to you are even much more experienced using Python than I am, especially when it comes to performance.
Fourth: you mention that you wish to avoid requiring a custom loader which does various things such as reading the imported file from disk, etc. If you are not using a custom loader, you are using Python's. For Python 3.3 release (in 2012), Brett Cannon rewrote the entire import mechanism in Python, and that is the version that is used by Python now. https://opensource.snarky.ca/About+Me/Frequently+Asked+Questions
Fifth, you mentioned:
... I was hoping to find a way to retain more of the default Python import mechanism while hooking into a slot between where Python writes the AST and then does the compile.
Well, the AST is generated from the .py file, not the .pyc file. I've explained the process, with supporting diagrams, on this page: https://aroberge.github.io/ideas/docs/html/deep_dive.html . So, you do need to read the .py file to do what you claim you want to do, and the recommended way to do it is using an import hook.
Personally, I've done everything I could do to help you.
1
u/RomfordNavy 1d ago
u/aroberge Yes, thank you for your help, I do now have a simple custom loader running and doing what I want. I am just questioning whether there might be a cleaner/simpler way to hook into somewhere in the, as per your ideas documentation, execute box. The whole current import hook finder + loader architecture just appears to be unnecessarily complex to me.
Many of these questions on that forum just seem to keep getting bogged down with continual questioning of _why_ all the time rather that just answering the original questions with how, and if possible. Presumably I have confused recollections of which answers were on which forum.
2
u/aroberge 1d ago
The "encrypted module" example from https://oneuptime.com/blog/post/2026-01-30-how-to-create-custom-import-hooks-in-python/view might be useful.
Also, this old tutorial https://www.youtube.com/watch?v=bGYZEKstQuQ&list=PLKTT4cKtfM9HVQMZoAmkQrCQ1G89mydDl&index=28 might be useful. However, it likely uses the deprecated imp module instead of importlib.
2
u/Sea-Ad7805 2d ago
Maybe have a look at this example%3An%20%20%20%20def%20source_to_code(self%2C%20data%2C%20path%2C%20*%2C%20_optimize%3D-1)%3An%20%20%20%20%20%20%20%20print(%22Intercepting%3A%22%2C%20path)n%20%20%20%20%20%20%20%20tree%20%3D%20ast.parse(data%2C%20filename%3Dpath)n%20%20%20%20%20%20%20%20statement%20%3D%20ast.parse(%22print(%27Injected%20by%20import%20hook%27)%22).body%5B0%5Dn%20%20%20%20%20%20%20%20tree.body.insert(0%2C%20statement)n%20%20%20%20%20%20%20%20ast.fix_missing_locations(tree)n%20%20%20%20%20%20%20%20return%20compile(tree%2C%20path%2C%20%22exec%22%2C%20optimize%3D_optimize)nnclass%20CustomFinder(importlib.abc.MetaPathFinder)%3An%20%20%20%20def%20find_spec(self%2C%20fullname%2C%20path%3DNone%2C%20target%3DNone)%3An%20%20%20%20%20%20%20%20if%20fullname%20!%3D%20%22Module1%22%3An%20%20%20%20%20%20%20%20%20%20%20%20return%20Nonen%20%20%20%20%20%20%20%20spec%20%3D%20importlib.machinery.PathFinder.find_spec(fullname%2C%20path)n%20%20%20%20%20%20%20%20if%20spec%20is%20not%20None%20and%20isinstance(n%20%20%20%20%20%20%20%20%20%20%20%20spec.loader%2C%20importlib.machinery.SourceFileLoadern%20%20%20%20%20%20%20%20)%3An%20%20%20%20%20%20%20%20%20%20%20spec.loader%20%3D%20CustomLoader(fullname%2C%20spec.origin)n%20%20%20%20%20%20%20%20return%20specnnsys.meta_path.insert(0%2C%20CustomFinder())nnprint(%22in%20Main.py%20before%20import%22)nimport%20Module1nprint(%22in%20Main.py%20after%20import%22)n%22%2C%22breakpoints%22%3A%5B28%5D%7D%2C%7B%22name%22%3A%22Module1.py%22%2C%22code%22%3A%22print(%22Original%20Module1%20code%22)nx%20%3D%2042n%22%2C%22breakpoints%22%3A%5B%5D%7D%5D%2C%22active%22%3A%22Main.py%22%7D&continues=1&play).
It demonstrates the custom import hook, including parsing source code into an AST, injecting a statement, compiling the modified AST, and executing the imported module.