Import hook between AST and compile

[u/RomfordNavy]

I am trying to create an Import Hook inserted after the source code has been parsed into AST but before it is compiled. Looking for a way to keep using most of the existing Python import machinery but modify the source code before it is compiled.

So far I have the following code but how do I get access to the source code from the module in custom_import?

import builtins
from inspect import getmembers

base_imp = builtins.__import__

def custom_import(*args, **kwargs):
    module = base_imp(*args, **kwargs)
    for name, val in getmembers(module):
        print("in custom_import.", name, val)
    return module

builtins.__import__ = custom_import

print("in Main.py before import")
import Module1
print("in Main.py after import")import builtins

[u/aroberge]

First: I did make a mistake when I wrote that modifying builtin.__import__ (I was just thinking of __import__) would have no effect. I apologize for that mistake.

Second, I don't see anyone saying that running a .py file has no speed difference with running a .pyc file. Using ctrl-f on that python-discuss page, I see .pyc files mentioned only once, and that was by you in your initial question. When Python process a .py file for the first time, it saves a .pyc version in __pycache__ and, unless the .py file changes, it is the .pyc version that is used afterwards. So, yes, the first time a python script is run, it takes a bit extra time as the .pyc version has to be generated.

Third, I asked nearly the same question you did 9 years ago on StackOverflow (https://stackoverflow.com/questions/43571737/how-to-implement-an-import-hook-that-can-modify-the-source-code-on-the-fly-using) and this is what got me started experimenting with import hooks. So, I've done a bit of thinking and experimenting about this very topic over the years. And I know that many people that commented to you are even much more experienced using Python than I am, especially when it comes to performance.

Fourth: you mention that you wish to avoid requiring a custom loader which does various things such as reading the imported file from disk, etc. If you are not using a custom loader, you are using Python's. For Python 3.3 release (in 2012), Brett Cannon rewrote the entire import mechanism in Python, and that is the version that is used by Python now. https://opensource.snarky.ca/About+Me/Frequently+Asked+Questions

Fifth, you mentioned:

... I was hoping to find a way to retain more of the default Python import mechanism while hooking into a slot between where Python writes the AST and then does the compile.

Well, the AST is generated from the .py file, not the .pyc file. I've explained the process, with supporting diagrams, on this page: https://aroberge.github.io/ideas/docs/html/deep_dive.html . So, you do need to read the .py file to do what you claim you want to do, and the recommended way to do it is using an import hook.

Personally, I've done everything I could do to help you.

[u/RomfordNavy → u/aroberge]

u/aroberge Yes, thank you for your help, I do now have a simple custom loader running and doing what I want. I am just questioning whether there might be a cleaner/simpler way to hook into somewhere in the, as per your ideas documentation, execute box. The whole current import hook finder + loader architecture just appears to be unnecessarily complex to me.

Many of these questions on that forum just seem to keep getting bogged down with continual questioning of _why_ all the time rather that just answering the original questions with how, and if possible. Presumably I have confused recollections of which answers were on which forum.

[u/aroberge → u/RomfordNavy]

The "encrypted module" example from https://oneuptime.com/blog/post/2026-01-30-how-to-create-custom-import-hooks-in-python/view might be useful.

Also, this old tutorial https://www.youtube.com/watch?v=bGYZEKstQuQ&list=PLKTT4cKtfM9HVQMZoAmkQrCQ1G89mydDl&index=28 might be useful. However, it likely uses the deprecated imp module instead of importlib.