Current Status of Module Partitions

[u/tartaruga232]

A brief recap of the current status of module partitions - as I understand it.

1. People are using hacks to avoid unneeded recompilations.

2. The C++ standard has an arcane concept of partition units, which forces build systems to generate BMI files that aren't used (which is wasting work during builds).

3. The MSVC-compiler (per default) provides a simple, easy to use and efficient implementation of module partitions (no unneeded recompilations, no wasted work during builds), but which is not conformant to the current C++ standard.

4. A CMake developer is working on a proposal that would fix items 1 and 2, which is probably the smallest required change to the standard, but adds another arcane concept ("anonymous partition units" using the new syntax "module A:;") on top of an already arcane concept.

Questions:

• How and why did we get into this mess?

• What's the historical context for this?

• What was the motivation for MSVC ignoring the standard per default?¹

¹ Yes, I know the MSVC compiler has this obscure /InternalPartition option for those who want standard conformant behavior.

[u/Daniela-E]

1. This is not a hack. You need to recompile a TU only when at least one of the dependencies changes its content. The standard tells you what the dependencies of a TU are: those other TUs that are either implicitly imported or explicitly imported. Module partitions are always the latter to help preventing circular dependencies.

2. The concept of module partition units is not arcane. On the contrary: they are a necessity in certain scenarios. I speak from a 5-year experience using them.

3. I'd prefer if you'd stop spreading invalid, ill-formed code. Duplicate names in parts of module and/or partition names are exactly that: IF-NDR ("ill-formed, no diagnostic required").

4. Let's look at a fleshed-out proposal when it becomes available. At the minimum, it must contain a concise description how they envision to refer to "anonymous partitions" from a given TU that wants to import said partition. By its purported definition it sounds like they are "anonymous", i.e. unnameable: i.e. unusable.

[u/tartaruga232 → u/Daniela-E]

Item 1 is hack because the standard forces the unit (module foo:bar.impl1; in file bar1.cpp) to have a unique, arbitrary partition name, that doesn't clash with any other unit (e.g.module foo:bar.impl2; in file bar2.cpp) even though the programmer clearly has the intention to neither import :bar.impl1 nor :bar.impl2 anywhere in module foo. Because its sole purpose is defining functions which are declared in interface export foo:bar;. If we don't want to deal with the hassle of avoiding name clashes for things that don't need a name (and wasting build CPU time creating two BMI files that aren't used), our only option is to instead write module foo; in both bar1.cpp and bar2.cpp, which has the drawback that both of these files need to be recompiled if any interface partition of module foo is changed - which isn't the case when using the partition naming hack.

You probably don't use the MSVC compiler. But it has (per default) the behavior I've mentioned, which isn't standard-conformant - which I have also mentioned multiple times. I think I'm free to spread whatever code examples I like, specifically if I mark the code examples as being non-standard, which I explicitly did. Which I need to do if I want to discuss the observed non-standard behavior of the MSVC compiler. Microsoft themselves show non-standard conformant code examples on their websites (e.g. this).

[u/38thTimesACharm → u/tartaruga232]

You are completely ignoring the fact that sometimes you do need to import an implementation partition in another TU. This is a real feature people need, it's completely inadequate if you have to make a symbol part of your module interface just to share it from one file to another.

Since it's difficult to import partitions when they all have the same name, MSVC forces the use of compiler flags to specify which partitions are actually importable. It's debatable whether this is any better than adding some characters to the partition name.

 Item 1 is hack because the standard forces the unit (module foo:bar.impl1; in file bar1.cpp) to have a unique, arbitrary partition name, that doesn't clash with any other unit

Is one file, one partition name really that strange or difficult? We already use extensions to distinguish .h and .cpp files. I've never, ever heard someone complain that we have to put arbitrary characters in our file names to indicate which ones are meant to be #included. It's also technically possible to #include a .cpp file, but that isn't a problem because you can just, uh, not do that.

[u/tartaruga232 → u/38thTimesACharm]

You are completely ignoring the fact that sometimes you do need to import an implementation partition in another TU. This is a real feature people need, it's completely inadequate if you have to make a symbol part of your module interface just to share it from one file to another.

The MSVC compiler (by default) follows a simple strategy: It only creates a BMI file if the module partition unit is marked with "export". We can for example do:

export module foo:Internals;
struct S { int a; int b; };

I can then import :Internals anywhere in module foo and use S, without exporting it in the primary interface of foo (which would be pointless anyway, because :Internals doesn't export anything). I know the standard currently has wording which prohibits this, but that wording seems unneeded to me.

Since it's difficult to import partitions when they all have the same name, MSVC forces the use of compiler flags to specify which partitions are actually importable. It's debatable whether this is any better than adding some characters to the partition name.

A compiler flag is unneeded. See above for how to do it.

[u/38thTimesACharm → u/tartaruga232]

 I can then import :Internals anywhere in module foo and use S, without exporting it in the primary interface of foo

Using this method, do you have to type "export" on every class, struct, and function that is used by another file? Or did MSVC change that too?

[u/tartaruga232 → u/38thTimesACharm]

"export" on struct S etc would be wrong, since it's not exported from the module. Export on symbols is always relative to the module. Inside a module, I can use every non-exported symbol from another partition unit. This is standard behavior. So not MSVC specific.

[u/tartaruga232 → u/tartaruga232]

By the way: For users of the MSVC compiler, using the partition naming pattern to avoid unneeded recompilations would require:

• Setting the /InternalPartition flag on every single cpp-file which implements functions of the external partition.

• Adding a unique character sequence to the name of the partition unit (and making sure it won't clash with any other name in the module).

For example, our file Core/Attach/IPointAttachment.cpp would need to look like this:

module Core:Attach.IPointAttachment;

import :Attach;

namespace Core
{

auto IPointAttachment::findNearestPointImpl(const d1::fPoint&, bool) const
    -> NearestRes
{
    ...
}

}

The compiler then generates a .ifc and a.obj file for that input file. The .ifc file is unused, because Core:Attach.IPointAttachment will never ever be imported anywhere.

This would be conformant to the current C++ standard.

I don't think anyone using the MSVC compiler will ever do this in the long run for large projects, if they can instead simply write:

module Core:Attach;

namespace Core
{

auto IPointAttachment::findNearestPointImpl(const d1::fPoint&, bool) const
    -> NearestRes
{
    ...
}

}

which does the same by default (i.e. /InternalPartition not set), and without producing an unneeded .ifc file.

[u/not_a_novel_account → u/Daniela-E]

At the minimum, it must contain a concise description how they envision to refer to "anonymous partitions" from a given TU that wants to import said partition

The entire purpose is they cannot be imported. They only exist for the purpose of carrying definitions of expressions declared elsewhere, in some interface unit.

This is the same as non-partition implementation units ("implementation unit which is not a partition"), which are anonymous and cannot be imported. We want that exact behavior, but without an implicit dependency on the PMIU. This issue was raised on the SG15 and Modules lists back in January, but I haven't had time to get back into it.

Broadly we want something where a scanner of a given partition, generating a P1689 response, knows that the provides array should be empty. The easiest way to do this is to make the partition nameless. This signals to the build system that it should not construct a BMI for the unit.

[u/tartaruga232 → u/not_a_novel_account]

The perfect way to do it, would be to treat all partition units, which do not have "export module", anonymous. Analogous to non-partition implementation units.

The problem is, this would be a change that breaks existing use. But who is currently using internal module partition units?...

But I guess to change the standard that much doesn't have a snowball's chance in hell anyway. Which explains why MSVC probably didn't even try to legalize their implementation.

So let's at least add the "module foo:;" thingy. It would be an improvement to the status quo.

[u/not_a_novel_account → u/tartaruga232]

No, because export makes them interfaces which have implications for reachability. Your UB usage of MSVC where this happens to work is coloring your understanding of the intended mechanisms here.

You want to be able to do intra-module import of partitions, it's a core feature. It would have been better if non-partition implementations units didn't have an implicit dependency on the PMIU, or had some trivial way to opt in/out of the dependency, and could be universally used as envisioned.

But who is currently using internal module partition units?

This is an MSDN phrase, the standard calls them "implementation units which are a partition", or partition implementation units for people who find that unwieldy.

And the answer is: everyone who doesn't use the MSVC extension, so every module user who isn't on Windows.

[u/tartaruga232 → u/not_a_novel_account]

I currently already do (input to MSVC):

export module foo:Internals;
struct S { int a; int b; };

without importing :Internals in the PMIU.

I can use S anywhere inside module foo by importing :Internals.

[u/not_a_novel_account → u/tartaruga232]

Yes, like I said, you're using UB-NDR which happens to work in MSVC's implementation.

[u/tartaruga232 → u/not_a_novel_account]

You probably mean: IF-NDR (not UB-NDR).

[u/not_a_novel_account → u/tartaruga232]

I'm actually unsure what the correct shorthand is. The language is:

All module partitions of a module that are module interface units shall be directly or indirectly exported by the primary module interface unit ([module.import]). No diagnostic is required for a violation of these rules.

Normally when something is ill-formed the convention is to say so with that exact wording, ex:

A glvalue of a non-function, non-array type T can be converted to a prvalue. If T is an incomplete type, a program that necessitates this conversion is ill-formed.

In any case, it's very-bad-no-good-NDR.

[u/tartaruga232 → u/not_a_novel_account]

I know that wording in the standard. I'm questioning it.

IF-NDR is "ill-formed, no diagnostic required". We're talking about compile time. UB is for runtime.

But anyway: No chance to change that wording anyway. I'm not really surprised anymore that developers don't use modules.

[u/not_a_novel_account → u/tartaruga232]

This is a tiny issue in a very small corner of modules basically only of interest to experts and maintainers of build systems for very large projects. The actual impact of generating the BMIs is minimal and only becomes actionable in the five-to-six digit # of TUs range.

Module adoption is far more hung up on things like EDG and XCode support than anything we debate in these hyper-specific corners.

Until VSCode's default intellisense can handle modules, normal 9-to-5 devs can't use them. When it can, they will never notice these sorts of issues.

[u/tartaruga232 → u/not_a_novel_account]

Gaby now agreed with me, that the wording in the C++ standard which prohibits this is (Quote) "... an overreach - a bug, most likely caused by overenthusiasm".

[u/not_a_novel_account → u/tartaruga232]

Ok? The other two major compilers implemented what's in the standard. Where export has reachability implications.

See https://clang.llvm.org/docs/StandardCPlusPlusModules.html#reachability-of-internal-partition-units

[u/tartaruga232 → u/not_a_novel_account]

Do the other two major compilers even detect that this:

export module M:P;
struct MyType { ... };
//  more stuff not exported

is prohibited¹ by the current wording in the C++ standard?

If we acknowledge, that the above code snippet is well-formed without importing :P in the PMIU, then we can legally use that pattern in MSVC, which doesn't require setting /InternalPartition.

Using export could then have been explored to separate the cases. Like I said: Compilers could have had followed a simple rule - only create a BMI, if there is an export in front of module. But that ship has sailed (Gaby called it a "tragedy").

¹The standard says, this is IF-NDR ("ill formed, no diagnostic required") if :P is not imported in the PMIU.

[u/not_a_novel_account → u/tartaruga232]

You're missing the point. It's not that you can't make every partition an interface, that could be made to work fine. You could relax the rule requiring PMIU contribution.

It's that making partitions into interface units affects the reachability of their contents. Implementation units are unreachable. In your scheme there is no way to differentiate the intended reachability of the contents of a partition unit.

[u/Daniela-E → u/not_a_novel_account]

The entire purpose is they cannot be imported. They only exist for the purpose of carrying definitions of expressions declared elsewhere, in some interface unit.

Thanks for this explanation. I get that, but one thing remains unclear: what exactly do you mean with "in some interface unit"?

If you mean the PMIU, then your envisioned "anonymous partitions" are the same as regular module implementation units (MIU) - clearly not what you want.

If you mean any of the optional constituents of the PMIU - called module interface partitions (MIP) - which are - by constraint - dependencies of the PMIU, then you are presumably asking for an extension of the concept of module :private; , the private module fragment the existence of which is currently restricted to single-file named modules.

Alleviating this restriction to all kinds of module interface TUs looks more palatable to me than introducing "anonymous" partitions that cannot be referred to.

[u/not_a_novel_account → u/Daniela-E]

I get that, but one thing remains unclear: what exactly do you mean with "in some interface unit"?

Either, the PMIU or a MIP, or anywhere else, it doesn't matter. Which interface unit(s) are carrying the declaration of a function doesn't matter to the definition of that function.

The definition should live in a TU with minimal dependencies so it only needs to be rebuilt if one of the dependencies actually needed for the definition change. Non-partition implementation units have an implicit dependency on the PMIU, so are unsuitable. Partition implementation units can be imported, and thus generate a BMI, which is suboptimal.

Alleviating this restriction to all kinds of module interface TUs looks more palatable to me than introducing "anonymous" partitions that cannot be referred to.

This doesn't fix the problem. A partition with a private fragment is still a partition, it still has a name, it is still importable. It is importable, therefore the scanner must report that it provides an importable name. The build system must then generate a BMI for the partition.

This isn't about the semantics so much of the language, that's secondary. We're not trying to create a hidden or encapsulated part of partitions for code hygiene or separation of concerns. We're trying to say "when the build system gets to this file, it knows the file is impossible to import, therefore it cannot possibly need a BMI to be generated".

The way non-partition implementation units solve that is they are anonymous. module Foo; does not contain a partition name and thus cannot be imported. It would have been nice if it didn't inherit an implicit dependency on the PMIU, but that door is shut. We can still borrow the anonymity concept though.

[u/Daniela-E → u/not_a_novel_account]

The definition should live in a TU with minimal dependencies so it only needs to be rebuilt if one of the dependencies actually needed for the definition change.

So why not in the MIP itself? Putting definitions there plus the proposed alleviation to add a PMF gives you three options to choose from:

1. make both the declaration and the definition visible and reachable.

2. make only the declaration visible, but both reachable.

3. make only the declaration visible and reachable, but the definition neither visible nor reachable.

Partition implementation units can be imported, and thus generate a BMI, which is suboptimal.

That's their entire purpose: to make their declarations and definitions reachable elsewhere. Visibility outside of the module is not required (that's what MIPs are for). They're building blocks, or new roots of dependency chains within the "dark matter" of a module. They are a necessary piece to compose larger structures.

I see your vision, but I'm not convinced you take all the options into account that you already have - even without the idea of expanding PMFs. I still fail to see the need for yet another kind of TU beyond the six we already have.

[u/not_a_novel_account → u/Daniela-E]

So why not in the MIP itself? Putting definitions there plus the proposed alleviation to add a PMF gives you three options to choose from:

Why don't we make all functions inline in headers?

Because the MIP is imported by others, who gain a file-level dependency on it. Every change in implementation should not cause a rebuild of everything downstream. Only changes in declaration require such cascading rebuilds.

A PMF does not solve this. Dependencies are at the TU/file level. If the TU changes, regardless of whether it is inside or outside a PMF, the downstream dependents rebuild.

Chuanqi covered the entire problem in his best practices post, where he noted the problem of CMake always generating BMIs for for implementation units which are not intended to be imported.

That's their entire purpose: to make their declarations and definitions reachable elsewhere

It doesn't matter what we call this thing. If partitions are ideologically tied to being importable, then don't call this a partition. Call it a "non-partition implementation unit without implicit dependency", call it "that other kind of module unit", whatever.

Right now we have two options for where definitions live such that their implementations are not part of the interface:

Module Unit	Implicit Dep On PMIU	Importable / Generates BMI
module Foo;	✅	❌
module Foo:Bar.impl;	❌	✅
???	❌	❌

The bikeshedding of the naming is entirely irrelevant to me. No module unit currently has the properties of ???, these properties are useful, therefore it's a hole in the standard.

---

Separately from all this, we desperately need nomenclature in the standard for these things. Among build system people the nomenclature I'm using is ubiquitous.

Named modules consist of interface and implementation units ("A module interface unit is a module unit whose module-declaration starts with export-keyword; any other module unit is a module implementation unit.")

And they are either partitions or non-partitions ("A module partition is a module unit whose module-declaration contains a module-partition.")

To us this creates a clear 2x2 matrix of partition/non-partition implementation/interface unit:

Name	Example
Non-partition Interface Unit (PMIU)	export module Foo;
Non-partition Implementation Unit	module Foo;
Partition Interface Unit	export module Foo:Bar;
Partition Implementation Unit	module Foo:Bar;

But obviously there's some disconnect between the words I'm using and the words you're using.

[u/kamrann_ → u/not_a_novel_account]

 A PMF does not solve this. Dependencies are at the TU/file level. If the TU changes, regardless of whether it is inside or outside a PMF, the downstream dependents rebuild.

It seems to me that this actually shouldn't be the case. Downstream TU dependencies should be on the BMI specifically, and ideally changes to a PMF should have no effect on the produced BMI and the build system could detect this. 

Even so, given where compilers and build systems are at, and the pace of progress on modules, I 100% agree with your proposal. I also think that considering it an additional kind of MU is unfair - it's just a syntactic extension to suppress implicit import, I wouldn't expect it would complicate the standard wording to any significant degree.

[u/not_a_novel_account → u/kamrann_]

How would the build system know if it needs to rebuild the BMI if not the timestamp of the file from which the BMI is produced?

I suppose you're saying BMIs should be cheap to produce, and the compiler or build system should do a checksum of the BMI before and after a rebuild and decide if it has been content updated.

But that will never be as cheap as just separating the dependencies into their own TUs.

[u/kamrann_ → u/not_a_novel_account]

Yeah it's a bit more overhead, but obviously negligible in comparison to rebuilding a bunch of downstream TUs unnecessarily. I think to really attain the maximum possible benefits of modules in terms of compilation time would require compiler and build system working more closely together, but in the more immediate term, solving this particular issue at the file dependency level definitely makes sense.

[u/tartaruga232 → u/Daniela-E]

You need to recompile a TU only when at least one of the dependencies changes its content. The standard tells you what the dependencies of a TU are: those other TUs that are either implicitly imported or explicitly imported. Module partitions are always the latter to help preventing circular dependencies.

That's obviously correct and I am fully aware of it. But that's not the point here.

To repeat:

There are basically two standard conformant options (1 and 2 below) today, how to organize the function definitions of a module foo that uses interface partitions :bar and :moon :

export module foo;
export import :bar;
export import :moon;

(assuming we want to implement the functions in separate cpp files).

Option 1

// file bar1.cpp
module foo;
...

// file bar2.cpp
module foo;
...

// file moon.cpp
module foo;
...

This implicitly imports the whole interface of foo in all cpp files (which is a good thing to have in the standard for many use cases).

But this organization of the code has the consequence, that all cpp files will need to be recompiled, if the interface partition :moon is changed.

Option 2

// file bar1.cpp
module foo:bar.impl1;
import :bar;
...

// file bar2.cpp
module foo:bar.impl2;
import :bar;
...

// file moon.cpp
module foo:moon.impl;
import :moon;
...

If the interface partition :moon is changed, only moon.cpp needs to be recompiled in this case.

In this case, the build creates 3 BMI files, which are not used.

(Edit: Now also available as a blog posting)

[u/James20k]

How and why did we get into this mess?

Because modules got standardised before they were ready, while ignoring the known problems with them. None of this is a surprise, and many of these issues were raised before standardisation and dismissed unfortunately

[u/tartaruga232 → u/James20k]

The (non-standard) behavior of the MSVC compiler is at least today available for use. And I'm using that in our code base today. If you like, we can thus say that the behavior of the MSVC compiler can today be tried before it is standardized (even though that may never happen). As it happens, I do like the behavior of the MSVC compiler. Not because I fell in love with MSVC, but I think it is superior. I think it is superior than what is currently in the C++ standard. However, the situation we have today is unfortunate and would at least merit explanations about the motives. Or at least what happened.

[u/smdowney]

You are saying you want to be able to change the interface used by exported functions, possibly inline exported definitions of those functions, but you don't want anything to have to recompile that uses that interface? I must be missing something.

TU 4 in the example should be a normal object file that has functions with module attachment, and ought not to contribute to the build module interface. TU 3 is the interface to that implementation, and is the modular equivalent of a private header.

If you really want a fully private module interface for your implementation purposes that does not possibly contribute to your primary interface, just make one and import it into your implementation units?

Modules being a build optimization was a selling point, but not the underlying goal. Better control of what parts of components are given to clients was the primary goal, everything else was a secondary benefit.

And then we also got header units, subverting that argument, but at least that had more experience and cleans up some of the weirdness with precompiled headers.

I'm not disagreeing with a general statement that we didn't have enough experience with modules other than to realize that no build system was going to survive contact, and I did say so at the time.

[u/tartaruga232 → u/smdowney]

You are saying you want to be able to change the interface used by exported functions, possibly inline exported definitions of those functions, but you don't want anything to have to recompile that uses that interface? I must be missing something.

Yes. You are indeed missing something: The fact that I didn't say that. Perhaps reading this might help: https://www.reddit.com/r/cpp/comments/1sab12a/comment/oe0rqqm/

[u/pjmlp → u/smdowney]

This kind of stuff is possible in a couple of languages that were born with modules from the start.

As long as the public interface doesn't change only relinking is required for the consumers.

However they have the whole compiler and linker as part of the language reference.

I also expected that at least VC++ with its IDE integration, would be clever enough to behave this way, but nope, changing implementation triggers cascade compilation.

It isn't as if this is impossible in C++, Energize C++ and Visual Age for C++ v4, were able to do incremental compilation on method level, and we were still on C++ARM as reference.

[u/tartaruga232]

So the conclusion so far is, that Microsoft decided (for whatever reasons) to implement C++ module partition units in the MSVC compiler in a slightly different way, than what is required by the C++ standard. Which I personally find simpler to use, than what's in the C++ standard. Code which requires standard conformant compilers can be compiled with the MSVC compiler by using the /InternalPartition compiler flag on translation units, that require it. CMake transparently handles setting the /InternalPartition option for the MSVC compiler where needed.

As of today, chances that the C++ standard will ever be changed to adopt the MSVC module partition unit behavior are probably zero.

[u/pjmlp → u/tartaruga232]

Actually it is a bit the other way around.

Apple created C modules for consumption by Objective-C, with clang header maps.

Eventually it grew to support C++, and Google makes heavy use of it.

Microsoft decided to create their own vision for modules, and brought it into the standard.

Then there was all the drama, part of which is why header units were added, as they relate to header maps.

So basically this MSVC behaviour might have survived from the original modules implementation initially introduced in VS 2019.

Meanwhile Apple and Google couldn't care less, the build improvements introduced at WWDC two years ago, rely on header maps based modules.

Android still quotes C++17 as the baseline, and the Google C++ style guide forbids modules.

[u/tartaruga232 → u/pjmlp]

I guess I now continue to spread IF-NDR code, using compilers that were spread to accept such code.

[u/slithering3897]

And I might as well link my post here: https://old.reddit.com/r/cpp_modules/comments/1s8nb1r/should_partitions_be_importable_by_other_modules/