Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SwiftTemplate fails to build due to Invalid manifest #1322

Open
markst opened this issue Apr 2, 2024 · 23 comments
Open

SwiftTemplate fails to build due to Invalid manifest #1322

markst opened this issue Apr 2, 2024 · 23 comments
Assignees

Comments

@markst
Copy link
Contributor

markst commented Apr 2, 2024

We're using a XcodeBuildToolPlugin to generate mocks using Sourcery as mentioned in this issue:
#1229

This works when running in Xcode, however occasionally when running tests using xcodebuild the following can occur:

⚠️ /Users/vagrant/Library/org.swift.swiftpm/configuration is not accessible or not writable, disabling user-level cache features.
⚠️ /Users/vagrant/Library/org.swift.swiftpm/security is not accessible or not writable, disabling user-level cache features.
⚠️ /Users/vagrant/Library/Caches/org.swift.swiftpm is not accessible or not writable, disabling user-level cache features.
❌ error: '2.2.2': Invalid manifest (compiled with: ["/Applications/Xcode-15.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/TemporaryDirectory.rZXXGv/vfs.yaml", "-L", "/Applications/Xcode-15.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Applications/Xcode-15.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "arm64-apple-macosx13.0", "-sdk", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk", "-F", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Applications/Xcode-15.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode-15.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk", "-package-description-version", "5.7.0", "/Users/vagrant/Library/Developer/Xcode/DerivedData/Listen-fvqyfexjyixxxlfwlponyonxlklz/SourcePackages/plugins/Listen.output/ListenTests/SourceryPlugin/SourceryPlugin/Build/SwiftTemplate/8AA32FFF-75B5-408E-9C19-FCDB218A2B59/2.2.2/Package.swift", "-Xfrontend", "-disable-implicit-concurrency-module-import", "-Xfrontend", "-disable-implicit-string-processing-module-import", "-o", "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/TemporaryDirectory.NscvFM/2.2.2-manifest"])
❌ error: fatalError
@markst
Copy link
Contributor Author

markst commented Apr 2, 2024

This repeatedly occurs when attempting to build using Bitrise continuous integration platform

@markst
Copy link
Contributor Author

markst commented Apr 2, 2024

We're currently using Sourcery 2.2.2.

I've attempting passing buildPath arguments to sourcery using both outputDirectory and NSTemporaryDirectory()

private func createBuildCommands(
    tool: PluginContext.Tool,
    configPath: Path,
    outputDirectory: Path
) -> [Command] {
    let cachePath = outputDirectory.appending(subpath: "SourceryPlugin/Cache")
    let buildPath = outputDirectory.appending(subpath: "SourceryPlugin/Build")
    return [
        .prebuildCommand(
            displayName: "Generate mocked types for target",
            executable: tool.path,
            arguments: [
                "--config", configPath,
                "--cacheBasePath", cachePath,
                "--buildPath", buildPath,
                "--disableCache",
                "--verbose"
            ],
            environment: ["DERIVED_SOURCES_DIR": outputDirectory],
            outputFilesDirectory: outputDirectory
        )

@markst
Copy link
Contributor Author

markst commented Apr 2, 2024

I'm now struggling to reproduce locally, the output appears to succeed:

        Raw compilation of SwiftTemplate took: 7.120736837387085

@markst
Copy link
Contributor Author

markst commented Apr 3, 2024

as a sidenote, the SwiftTemplate binary is built each time which leads me to think the executableCacheKey is changing despite no change in code?

image

@art-divin
Copy link
Collaborator

as a sidenote, the SwiftTemplate binary is built each time which leads me to think the executableCacheKey is changing despite no change in code?

image

Hi @markst , aside from other issues you've reported, for which there are billions of thanks ❤️ ,
SwiftTemplate binary is compiled once (in single threaded mode of running Sourcery), and then re-used. Other components are cached on per-execution basis.

You can verify this by using --verbose flag when running Sourcery starting 2.2.2 release. It is clearly shown when SwiftTemplate is reused from cache, and when it is re-compiled again due to a different cacheKey.

@swwol
Copy link
Contributor

swwol commented Apr 22, 2024

You can normally resolve this issue using swift package purge-cache

@markst
Copy link
Contributor Author

markst commented May 24, 2024

@art-divin possibly it's failing to get the cached build since it's failing to complete the build?

2024-05-24 17-13-41 2024-05-24 17_16_00

@markst
Copy link
Contributor Author

markst commented May 24, 2024

might be worth mentioning if I try to open the generated Package.swift in xcode it complains:
image

@art-divin
Copy link
Collaborator

Hello @markst ,

I wonder, can you try using Sourcery 2.2.1 version?


There is one concurrency issue to be resolved by the end of the upcoming week (next release should be done on the next weekend). Maybe it is related to the issue you are facing, but I suspect it is not. 🕵🏻‍♂️

@art-divin
Copy link
Collaborator

Investigating 🕵🏻

@art-divin art-divin self-assigned this Jun 19, 2024
@art-divin
Copy link
Collaborator

We're currently using Sourcery 2.2.2.

I've attempting passing buildPath arguments to sourcery using both outputDirectory and NSTemporaryDirectory()

private func createBuildCommands(
    tool: PluginContext.Tool,
    configPath: Path,
    outputDirectory: Path
) -> [Command] {
    let cachePath = outputDirectory.appending(subpath: "SourceryPlugin/Cache")
    let buildPath = outputDirectory.appending(subpath: "SourceryPlugin/Build")
    return [
        .prebuildCommand(
            displayName: "Generate mocked types for target",
            executable: tool.path,
            arguments: [
                "--config", configPath,
                "--cacheBasePath", cachePath,
                "--buildPath", buildPath,
                "--disableCache",
                "--verbose"
            ],
            environment: ["DERIVED_SOURCES_DIR": outputDirectory],
            outputFilesDirectory: outputDirectory
        )

Hello @markst ,

first thing I notice, is that you are using --disableCache flag. When this flag is set, cacheKey is ignored and SwiftTemplate is recompiled for every run.

This is not the root cause of your issue, but thing is, --cacheBasePath is ignored and unused because of the --disableCache flag.

@art-divin
Copy link
Collaborator

@markst by the way, the speed with which in the gif you've shared the build is failing, tells me that the compilation of SwiftTemplate fails for one or the other reason. Flag verbose should expose some details about these processes, would you mind checking the raw log output in Xcode's Report Navigator for this exact build phase - plugin containing sourcery invocation?

That would help a lot 👍🏻

@swwol
Copy link
Contributor

swwol commented Aug 2, 2024

I am seeing this same issue (or something that looks very similar) when building from a swiftpm plugin from xcode 16 beta 3 and Xcode 16 beta 4. The first xcode 16 beta doesn't appear to have the same issue, nor does Xcode 15.4. From my reading of the below error message - its maybe a sandboxing/file permissions issue? It is unable to import the PackageDescription module so fails to build the generated SwiftTemplate module.
Screenshot 2024-08-02 at 11 12 20

@swwol
Copy link
Contributor

swwol commented Aug 8, 2024

I'm not sure if my issue is exactly the same as yours, but in my case the issue is specifically with SwiftTemplate when sourcery in invoked from a build plugin in Xcode 16 beta 3+. I did an experiment running almost identical code from aCommand plugin - and in that case it works fine. My conclusion, from this behaviour and the error messages, is that in the latest xcode16 betas, the sandboxing around the build plugin has been tightened up so that the generated SwiftTemplate package cannot load in the external modules it need to compile.

@swwol
Copy link
Contributor

swwol commented Aug 8, 2024

This repeatedly occurs when attempting to build using Bitrise continuous integration platform

could you be using a different version of xcode on CI than locally?

@markst
Copy link
Contributor Author

markst commented Aug 9, 2024

Using Xcode 15.4 on local and Bitrise.
Mine is also using a build plugin, which does work fine locally.

Weirdly it's only certain templates which cause the issue, using a simpler stencil doesn't produce the error.

  command failed with exit status 75 (xcodebuild "-project" "/Users/vagrant/git/iView.xcodeproj" "-scheme" "ABC-iview" "test" "-destination" "id=3D607B78-B519-49C9-9893-69E10C677495" "-resultBundlePath" "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/XCUITestOutput1609981805/Test-ABC-iview.xcresult" "-retry-tests-on-failure" "-test-iterations" "3" "-xcconfig" "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/3140738280/temp.xcconfig"):
    error: '2.2.5': Invalid manifest (compiled with: ["/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/TemporaryDirectory.Of5BAl/vfs.yaml", "-L", "/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "arm64-apple-macosx13.0", "-sdk", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk", "-F", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk", "-package-description-version", "5.7.0", "/Users/vagrant/Library/Developer/Xcode/DerivedData/iView-gqtrnizhblfespbcrptzwvhjcwlb/SourcePackages/plugins/showpage.output/ShowPageTests/SourceryPlugin/SourceryGenerated/Build/SwiftTemplate/10FD6F02-FC4D-4AC8-AFE5-869EF4E370CA/2.2.5/Package.swift", "-Xfrontend", "-disable-implicit-concurrency-module-import", "-Xfrontend", "-disable-implicit-string-processing-module-import", "-o", "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/TemporaryDirectory.eSuSKt/2.2.5-manifest"])
    error: fatalError
    <unknown>:0: error: error opening '/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/C/clang/ModuleCache/PackageDescription-3441QLGOQ0HEV.swiftmodule' for output: Operation not permitted
    /Users/vagrant/Library/Developer/Xcode/DerivedData/iView-gqtrnizhblfespbcrptzwvhjcwlb/SourcePackages/plugins/showpage.output/ShowPageTests/SourceryPlugin/SourceryGenerated/Build/SwiftTemplate/10FD6F02-FC4D-4AC8-AFE5-869EF4E370CA/2.2.5/Package.swift:5:8: error: failed to build module 'PackageDescription' for importation due to the errors above; the textual interface may be broken by project issues or a compiler bug
    error: 2024-08-09 12:52:19.016 xcodebuild[7491:26436]  DVTFilePathFSEvents: Failed to start fs event stream.
    error: '2.2.5': Invalid manifest (compiled with: ["/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/TemporaryDirectory.Of5BAl/vfs.yaml", "-L", "/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "arm64-apple-macosx13.0", "-sdk", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk", "-F", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Applications/Xcode-15.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode-15.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk", "-package-description-version", "5.7.0", "/Users/vagrant/Library/Developer/Xcode/DerivedData/iView-gqtrnizhblfespbcrptzwvhjcwlb/SourcePackages/plugins/showpage.output/ShowPageTests/SourceryPlugin/SourceryGenerated/Build/SwiftTemplate/10FD6F02-FC4D-4AC8-AFE5-869EF4E370CA/2.2.5/Package.swift", "-Xfrontend", "-disable-implicit-concurrency-module-import", "-Xfrontend", "-disable-implicit-string-processing-module-import", "-o", "/var/folders/yy/6kcn9mkd5svdbqnznwf474f00000gn/T/TemporaryDirectory.eSuSKt/2.2.5-manifest"])
    error: fatalError

(this particular output is Xcode 15.2)

@markst
Copy link
Contributor Author

markst commented Aug 12, 2024

I can reproduce this when using swiftc directory in the output SwiftTemplate directory:

swiftc -swift-version 5 -sdk $(xcrun --sdk macosx --show-sdk-path) -package-description-version 5.7.0 ./Package.swift 
Package.swift:4:8: error: no such module 'PackageDescription'
import PackageDescription
       ^

It's confusing it's trying to build the Package.swift. I've updated my plugin to use buildCommand which does push the error further down the track. Even supplying the outputFiles seems to still attempt to build the Package.swift.

.buildCommand(
    displayName: "Generate mocked types for target",
    executable: tool.path,
    arguments: [
        "--config", configPath,
        "--cacheBasePath", cachePath.string,
        "--buildPath", buildPath.string,
        "--verbose"
    ],
    environment: ["OUTPUT_DIR": outputDirectory],
    outputFiles: [
        outputDirectory.appending("Mock.generated.swift")
    ]
)

@markst
Copy link
Contributor Author

markst commented Aug 19, 2024

Just to update here, the original issue only happens when using swifttemplate and not when using a stencil.

I tested using a really simple swifttemplate such as:

// Found <%= types.all.count %> types

@markst
Copy link
Contributor Author

markst commented Aug 20, 2024

I'm not sure if my issue is exactly the same as yours, but in my case the issue is specifically with SwiftTemplate when sourcery in invoked from a build plugin in Xcode 16 beta 3+.

So I tested using Xcode 16 Beta 4 locally and my plugin works fine.
When running on Bitrise.io using Xcode 16.1.0 I get the same issue:

Output
Last lines of the build log:
note: Run script build phase 'Generate mocked types for target' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'ShowPageTests' from project 'ShowPage')
Testing failed:
	'2.2.5': Invalid manifest (compiled with: ["/Applications/Xcode-16.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/1v/d12kc4b11ln61d6sncx9qh040000gn/T/TemporaryDirectory.Ye6OSk/vfs.yaml", "-L", "/Applications/Xcode-16.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Applications/Xcode-16.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "arm64-apple-macosx13.0", "-sdk", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.1.sdk", "-F", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Applications/Xcode-16.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.1.sdk", "-package-description-version", "5.7.0", "/Users/vagrant/Library/Developer/Xcode/DerivedData/iView-gqtrnizhblfespbcrptzwvhjcwlb/SourcePackages/plugins/showpage.output/ShowPageTests/SourceryPlugin/Build/SwiftTemplate/DC420EE6-FD89-4509-96B0-02D94ED19BC2/2.2.5/Package.swift", "-o", "/var/folders/1v/d12kc4b11ln61d6sncx9qh040000gn/T/TemporaryDirectory.DRpCvJ/2.2.5-manifest"])
	error opening '/var/folders/1v/d12kc4b11ln61d6sncx9qh040000gn/C/clang/ModuleCache/PackageDescription-2LK4PKUT2HR51.swiftmodule' for output: /var/folders/1v/d12kc4b11ln61d6sncx9qh040000gn/C/clang/ModuleCache/PackageDescription-2LK4PKUT2HR51.swiftmodule: Operation not permitted
	failed to build module 'PackageDescription' for importation due to the errors above; the textual interface may be broken by project issues or a compiler bug
	'2.2.5': Invalid manifest (compiled with: ["/Applications/Xcode-16.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/1v/d12kc4b11ln61d6sncx9qh040000gn/T/TemporaryDirectory.w7rzsC/vfs.yaml", "-L", "/Applications/Xcode-16.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Applications/Xcode-16.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "arm64-apple-macosx13.0", "-sdk", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.1.sdk", "-F", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Applications/Xcode-16.1.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode-16.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.1.sdk", "-package-description-version", "5.7.0", "/Users/vagrant/Library/Developer/Xcode/DerivedData/iView-gqtrnizhblfespbcrptzwvhjcwlb/SourcePackages/plugins/showpage.output/ShowPageTests/SourceryPlugin/Build/SwiftTemplate/DC420EE6-FD89-4509-96B0-02D94ED19BC2/2.2.5/Package.swift", "-o", "/var/folders/1v/d12kc4b11ln61d6sncx9qh040000gn/T/TemporaryDirectory.5QwXrI/2.2.5-manifest"])
	Testing cancelled because the build failed.
** TEST FAILED **

@swwol I'm curious to know how your Sourcery plugin is different from mine since it does seem we're experiencing similar issues when using SwiftTemplate specifically.

@markst
Copy link
Contributor Author

markst commented Aug 29, 2024

Created a demo project in order to test. Weirdly the project fails to build locally!

https://github.com/markst/sourcery-plugin-demo

@markst
Copy link
Contributor Author

markst commented Sep 3, 2024

We've gone with using https://github.com/Kolos65/Mockable instead

@swwol
Copy link
Contributor

swwol commented Sep 9, 2024

I'm not sure if my issue is exactly the same as yours, but in my case the issue is specifically with SwiftTemplate when sourcery in invoked from a build plugin in Xcode 16 beta 3+.

So I tested using Xcode 16 Beta 4 locally and my plugin works fine. When running on Bitrise.io using Xcode 16.1.0 I get the same issue:

Output
@swwol I'm curious to know how your Sourcery plugin is different from mine since it does seem we're experiencing similar issues when using SwiftTemplate specifically.

Looking at your comment above, one difference is I am using the plugin work directory for build and cache paths rather than output directory.

@markst
Copy link
Contributor Author

markst commented Sep 10, 2024

@swwol thanks for response! possibly changed since earlier comments. It is now using context.pluginWorkDirectory

https://github.com/markst/sourcery-plugin-demo/blob/main/spm-build-tools/Plugins/SourceryPlugin/SourceryPlugin.swift#L11

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants