Problems with the planned new cross-compilation model

[finagolfin]

Description

@compnerd put together a document on a planned new cross-compilation model last summer, which replaces the flags -sdk and -resource-dir with the flags -sdk and -sysroot instead. I've since seen some issues with this new model, in order of importance:

1. Explicitly specifying an -sdk has a bunch of bugs and doesn't work very well. The most common way to ship Swift core modules and runtime libraries is next to the compiler instead and explicitly specifying an external -resource-dir works reasonably, though bugs have crept in there too. However, explicitly specifying an external -sdk, particularly a full SDK that contains both a C/C++ sysroot and a Swift resource directory, has historically been the least used configuration and comes with a lot of bugs.
   For example, I just saw that the Windows CI tries to build the Foundation macros for the Windows host by using a 6.0.3 Windows SDK with the trunk 6.2 Swift compiler. As explained there, that works fine now because the Swift 6.2 compiler likely quietly ignores the 6.0.3 resource directory in that 6.0.3 SDK and uses the 6.2 stdlib modules next to the compiler instead, but when I enforced that it must use the 6.0.3 SDK alone in that pull, the compile failed as expected. We will need to shake out all such -sdk leaks before we can rely on this flag for cross-compilation.
2. The -sdk/-resource-dir flags look directly in those paths for the C sysroot and Swift runtime resources, but when using -sdk/-sysroot instead, -sdk looks in <sdkPath>/usr/lib/swift instead for the Swift-specific files. I understand why this was done, for backward compatibility of the -sdk flag, but this means you cannot specify an arbitrary Swift resource path anymore.
   For example, when building the compiler tools themselves in this repo, you will often see the flags -sdk / -resource-dir /home/finagolfin/build/Ninja-Release/swift-linux-aarch64/lib/swift/ used to compile against the system C/C++ sysroot and the freshly-built Swift stdlib. How do you replace that with -sdk/-sysroot? We'll probably have to add a third hidden flag -sdk-runtime that works just like -resource-dir currently does, if we ever want to remove the -resource-dir flag from building this repo itself.
3. The -sysroot flag is brand new and barely used, I see a couple bugs in its implementation already. This is easiest to remedy since it's new.

Regarding 1., since all current approaches use -sdk, we have no choice but to make it better. My in-progress work on making it work properly in #79621 is shaking out more bugs, which I will file and fix as I'm able. At the very least, this suggests we might have to go slower with cross-compilation SDKs while these problems are shaken out.

@compnerd, let me know your thoughts.

@al45tair, as PSG chair, maybe you have some input.

@etcwilde, you deal with these cross-compilation issues more than most, thoughts welcome.

@MaxDesiatov, the SDK bundles feature you helped create uses the current -sdk/-resource-dir flags a lot, perhaps you have some input.

[etcwilde]

So I don't think getting rid of the -resource-dir is the goal of that proposal, though certainly lessening the reliance on setting it. The compiler resources are meant to be things that are tied to a given version of the compiler. Swift hasn't been good about keeping them separate, so I'll use clang as an example, but the sanitizer runtimes and the clang builtin headers are specific to that exact clang and thus must ship as part of the toolchain. That's what the -resource-dir should point at. For Swift, we definitely want the clang-importer headers in a subdirectory of the resource directory. Some of the shims also probably belong in that directory too like HeapObject.h and Task.h, but not all of them. The C++ bridging code isn't stable and is also tied to a specific build of the compiler, so those should also be shipped in there too. Under normal circumstances, people shouldn't really need to touch this, but stuff happens.

Under the new definition, the -sdk flag should point to the Swift runtimes for the platform you're targeting, and then the -sysroot should point at the runtimes and libraries provided by the OS. This is mostly because most OS's don't come with a Swift stdlib and don't want to install the Swift runtimes into your sysroot, or can't because you're trying to build something for your local system and don't have privileges to do so.

I haven't personally lived on the flags yet since we haven't split stuff up and things aren't done cooking in this regard yet, but this will become more important as we split the SDK distribution from the per-platform toolchain monolith that we have today. My eventual goal is to be able to download an Ubuntu sysroot, an Ubuntu Swift SDK, and cross-compile a little CLI application for Ubuntu from my Mac without having to be super clever about it (or whatever OS's we decide to produce Swift SDKs for). I'm not denying that things are probably not working right at the moment, but that's the direction we're trying to push things in.

For example, when building the compiler tools themselves in this repo, you will often see the flags -sdk / -resource-dir /home/finagolfin/build/Ninja-Release/swift-linux-aarch64/lib/swift/ used to compile against the system C/C++ sysroot and the freshly-built Swift stdlib. How do you replace that with -sdk/-sysroot? We'll probably have to add a third hidden flag -sdk-runtime that works just like -resource-dir currently does, if we ever want to remove the -resource-dir flag from building this repo itself.

In this example, -sdk points at the standard libraries (swiftCore, concurrency, etc...) and their modules. -sysroot points at the C/C++ libraries coming from the system sysroot, and -resource-dir points at the compiler resources. This will make a lot more sense once the standard library and compiler build systems are fully split from each other. The standard library build won't be installing the just-built standard library into the compiler build directory since it's a separate project, so -sdk will need to point at where the stdlib was installed to.