Skip to content

Commit 71d49b0

Browse files
sebstosebsto
authored
[plugin] support for resources packaging on ubuntu (#467)
When including resources in the package and packaging on Ubuntu, `FileManager` throws a FilePermission error. The docker daemon runs as root and files to be copied are owned by `root` while the archiver runs as the current user (`ubuntu` on EC2 Ubuntu). The `FileManager` manages to copy the files but throws an error after the copy. We suspect the `FileManager` to perform some kind of operation after the copy and it fails because of the `root` permission of the files. See #449 (comment) for a description of the problem. This PR contains code to reproduce the problem, a very simple workaround, and an integration test. The workaround consists of - trapping all errors - verify if the error is the permission error (Code = 513) - verify if the files have been copied or not - if the two above conditions are met, ignore the error, otherwise re-throw it I would rather prefer a solution that solves the root cause rather than just ignoring the error. We're still investigating the root cause (see [this thread](https://forums.swift.org/t/filemanager-copyitem-on-linux-fails-after-copying-the-files/77282) on the Swift Forum and this issue on Swift Foundation swiftlang/swift-foundation#1125
1 parent b4f0164 commit 71d49b0

File tree

11 files changed

+243
-13
lines changed

11 files changed

+243
-13
lines changed

Diff for: ‎.github/workflows/integration_tests.yml

+13-2
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,11 @@ on:
1919
type: boolean
2020
description: "Boolean to enable the compilation of examples. Defaults to true."
2121
default: true
22+
archive_plugin_examples:
23+
type: string
24+
description: "The list of examples to run through the archive plugin test. Pass a String with a valid JSON array such as \"[ 'HelloWorld', 'APIGateway' ]\""
25+
required: true
26+
default: ""
2227
archive_plugin_enabled:
2328
type: boolean
2429
description: "Boolean to enable the test of the archive plugin. Defaults to true."
@@ -54,7 +59,7 @@ jobs:
5459
# We are using only one Swift version
5560
swift:
5661
- image: ${{ inputs.matrix_linux_swift_container_image }}
57-
swift_version: "6.0.1-amazonlinux2"
62+
swift_version: "6.0.3-amazonlinux2"
5863
container:
5964
image: ${{ matrix.swift.image }}
6065
steps:
@@ -98,6 +103,10 @@ jobs:
98103
name: Test archive plugin
99104
if: ${{ inputs.archive_plugin_enabled }}
100105
runs-on: ubuntu-latest
106+
strategy:
107+
fail-fast: false
108+
matrix:
109+
examples: ${{ fromJson(inputs.archive_plugin_examples) }}
101110
steps:
102111
- name: Checkout repository
103112
uses: actions/checkout@v4
@@ -107,8 +116,10 @@ jobs:
107116
# https://github.com/actions/checkout/issues/766
108117
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
109118
- name: Test the archive plugin
119+
env:
120+
EXAMPLE: ${{ matrix.examples }}
110121
run: |
111-
.github/workflows/scripts/check-archive-plugin.sh
122+
.github/workflows/scripts/check-archive-plugin.sh
112123
113124
check-foundation:
114125
name: No dependencies on Foundation

Diff for: ‎.github/workflows/pull_request.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,8 @@ jobs:
3636
# We pass the list of examples here, but we can't pass an array as argument
3737
# Instead, we pass a String with a valid JSON array.
3838
# The workaround is mentioned here https://github.com/orgs/community/discussions/11692
39-
examples: "[ 'APIGateway', 'APIGateway+LambdaAuthorizer', 'BackgroundTasks', 'HelloJSON', 'HelloWorld', 'S3_AWSSDK', 'S3_Soto', 'Streaming', 'Testing', 'Tutorial' ]"
40-
39+
examples: "[ 'APIGateway', 'APIGateway+LambdaAuthorizer', 'BackgroundTasks', 'HelloJSON', 'HelloWorld', 'ResourcesPackaging', 'S3_AWSSDK', 'S3_Soto', 'Streaming', 'Testing', 'Tutorial' ]"
40+
archive_plugin_examples: "[ 'HelloWorld', 'ResourcesPackaging' ]"
4141
archive_plugin_enabled: true
4242

4343
swift-6-language-mode:

Diff for: ‎.github/workflows/scripts/check-archive-plugin.sh

+13-3
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,17 @@
1313
##
1414
##===----------------------------------------------------------------------===##
1515

16-
EXAMPLE=HelloWorld
16+
log() { printf -- "** %s\n" "$*" >&2; }
17+
error() { printf -- "** ERROR: %s\n" "$*" >&2; }
18+
fatal() { error "$@"; exit 1; }
19+
20+
test -n "${EXAMPLE:-}" || fatal "EXAMPLE unset"
21+
1722
OUTPUT_DIR=.build/plugins/AWSLambdaPackager/outputs/AWSLambdaPackager
1823
OUTPUT_FILE=${OUTPUT_DIR}/MyLambda/bootstrap
1924
ZIP_FILE=${OUTPUT_DIR}/MyLambda/MyLambda.zip
2025

21-
pushd Examples/${EXAMPLE} || exit 1
26+
pushd "Examples/${EXAMPLE}" || exit 1
2227

2328
# package the example (docker and swift toolchain are installed on the GH runner)
2429
LAMBDA_USE_LOCAL_DEPS=../.. swift package archive --allow-network-connections docker || exit 1
@@ -33,5 +38,10 @@ file "${OUTPUT_FILE}" | grep --silent ELF
3338
# does the ZIP file contain the bootstrap?
3439
unzip -l "${ZIP_FILE}" | grep --silent bootstrap
3540

36-
echo "✅ The archive plugin is OK"
41+
# if EXAMPLE is ResourcesPackaging, check if the ZIP file contains hello.txt
42+
if [ "$EXAMPLE" == "ResourcesPackaging" ]; then
43+
unzip -l "${ZIP_FILE}" | grep --silent hello.txt
44+
fi
45+
46+
echo "✅ The archive plugin is OK with example ${EXAMPLE}"
3747
popd || exit 1

Diff for: ‎.licenseignore

+2-1
Original file line numberDiff line numberDiff line change
@@ -33,4 +33,5 @@ Package.resolved
3333
*.yaml
3434
*.yml
3535
**/.npmignore
36-
**/*.json
36+
**/*.json
37+
**/*.txt

Diff for: ‎Examples/ResourcesPackaging/.gitignore

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
.DS_Store
2+
/.build
3+
/Packages
4+
xcuserdata/
5+
DerivedData/
6+
.swiftpm/configuration/registries.json
7+
.swiftpm/xcode/package.xcworkspace/contents.xcworkspacedata
8+
.netrc

Diff for: ‎Examples/ResourcesPackaging/Package.swift

+57
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
// swift-tools-version: 6.0
2+
// The swift-tools-version declares the minimum version of Swift required to build this package.
3+
4+
import PackageDescription
5+
6+
// needed for CI to test the local version of the library
7+
import struct Foundation.URL
8+
9+
let package = Package(
10+
name: "ResourcesPackaging",
11+
platforms: [.macOS(.v15)],
12+
products: [
13+
.executable(name: "MyLambda", targets: ["MyLambda"])
14+
],
15+
dependencies: [
16+
.package(url: "https://github.com/swift-server/swift-aws-lambda-runtime.git", branch: "main")
17+
],
18+
targets: [
19+
.executableTarget(
20+
name: "MyLambda",
21+
dependencies: [
22+
.product(name: "AWSLambdaRuntime", package: "swift-aws-lambda-runtime")
23+
],
24+
path: ".",
25+
resources: [
26+
.process("hello.txt")
27+
]
28+
)
29+
]
30+
)
31+
32+
if let localDepsPath = Context.environment["LAMBDA_USE_LOCAL_DEPS"],
33+
localDepsPath != "",
34+
let v = try? URL(fileURLWithPath: localDepsPath).resourceValues(forKeys: [.isDirectoryKey]),
35+
v.isDirectory == true
36+
{
37+
// when we use the local runtime as deps, let's remove the dependency added above
38+
let indexToRemove = package.dependencies.firstIndex { dependency in
39+
if case .sourceControl(
40+
name: _,
41+
location: "https://github.com/swift-server/swift-aws-lambda-runtime.git",
42+
requirement: _
43+
) = dependency.kind {
44+
return true
45+
}
46+
return false
47+
}
48+
if let indexToRemove {
49+
package.dependencies.remove(at: indexToRemove)
50+
}
51+
52+
// then we add the dependency on LAMBDA_USE_LOCAL_DEPS' path (typically ../..)
53+
print("[INFO] Compiling against swift-aws-lambda-runtime located at \(localDepsPath)")
54+
package.dependencies += [
55+
.package(name: "swift-aws-lambda-runtime", path: localDepsPath)
56+
]
57+
}

Diff for: ‎Examples/ResourcesPackaging/Sources/main.swift

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
//===----------------------------------------------------------------------===//
2+
//
3+
// This source file is part of the SwiftAWSLambdaRuntime open source project
4+
//
5+
// Copyright (c) 2025 Apple Inc. and the SwiftAWSLambdaRuntime project authors
6+
// Licensed under Apache License v2.0
7+
//
8+
// See LICENSE.txt for license information
9+
// See CONTRIBUTORS.txt for the list of SwiftAWSLambdaRuntime project authors
10+
//
11+
// SPDX-License-Identifier: Apache-2.0
12+
//
13+
//===----------------------------------------------------------------------===//
14+
15+
import AWSLambdaRuntime
16+
import Foundation
17+
18+
let runtime = LambdaRuntime {
19+
(event: String, context: LambdaContext) in
20+
guard let fileURL = Bundle.module.url(forResource: "hello", withExtension: "txt") else {
21+
fatalError("no file url")
22+
}
23+
return try String(contentsOf: fileURL, encoding: .utf8)
24+
}
25+
26+
try await runtime.run()

Diff for: ‎Examples/ResourcesPackaging/hello.txt

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Hello World

Diff for: ‎Plugins/AWSLambdaPackager/Plugin.swift

+23-5
Original file line numberDiff line numberDiff line change
@@ -249,11 +249,29 @@ struct AWSLambdaPackager: CommandPlugin {
249249
let resourcesDirectoryName = artifactURL.lastPathComponent
250250
let relocatedResourcesDirectory = workingDirectory.appending(path: resourcesDirectoryName)
251251
if FileManager.default.fileExists(atPath: artifactURL.path()) {
252-
try FileManager.default.copyItem(
253-
atPath: artifactURL.path(),
254-
toPath: relocatedResourcesDirectory.path()
255-
)
256-
arguments.append(resourcesDirectoryName)
252+
do {
253+
try FileManager.default.copyItem(
254+
atPath: artifactURL.path(),
255+
toPath: relocatedResourcesDirectory.path()
256+
)
257+
arguments.append(resourcesDirectoryName)
258+
} catch let error as CocoaError {
259+
260+
// On Linux, when the build has been done with Docker,
261+
// the source file are owned by root
262+
// this causes a permission error **after** the files have been copied
263+
// see https://github.com/swift-server/swift-aws-lambda-runtime/issues/449
264+
// see https://forums.swift.org/t/filemanager-copyitem-on-linux-fails-after-copying-the-files/77282
265+
266+
// because this error happens after the files have been copied, we can ignore it
267+
// this code checks if the destination file exists
268+
// if they do, just ignore error, otherwise throw it up to the caller.
269+
if !(error.code == CocoaError.Code.fileWriteNoPermission
270+
&& FileManager.default.fileExists(atPath: relocatedResourcesDirectory.path()))
271+
{
272+
throw error
273+
} // else just ignore it
274+
}
257275
}
258276
}
259277

Diff for: ‎scripts/ubuntu-install-swift.sh

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
#!/bin/bash
2+
##===----------------------------------------------------------------------===##
3+
##
4+
## This source file is part of the SwiftAWSLambdaRuntime open source project
5+
##
6+
## Copyright (c) 2025 Apple Inc. and the SwiftAWSLambdaRuntime project authors
7+
## Licensed under Apache License v2.0
8+
##
9+
## See LICENSE.txt for license information
10+
## See CONTRIBUTORS.txt for the list of SwiftAWSLambdaRuntime project authors
11+
##
12+
## SPDX-License-Identifier: Apache-2.0
13+
##
14+
##===----------------------------------------------------------------------===##
15+
16+
sudo apt update && sudo apt -y upgrade
17+
18+
# Install Swift 6.0.3
19+
sudo apt-get -y install \
20+
binutils \
21+
git \
22+
gnupg2 \
23+
libc6-dev \
24+
libcurl4-openssl-dev \
25+
libedit2 \
26+
libgcc-13-dev \
27+
libncurses-dev \
28+
libpython3-dev \
29+
libsqlite3-0 \
30+
libstdc++-13-dev \
31+
libxml2-dev \
32+
libz3-dev \
33+
pkg-config \
34+
tzdata \
35+
unzip \
36+
zip \
37+
zlib1g-dev
38+
39+
wget https://download.swift.org/swift-6.0.3-release/ubuntu2404-aarch64/swift-6.0.3-RELEASE/swift-6.0.3-RELEASE-ubuntu24.04-aarch64.tar.gz
40+
41+
tar xfvz swift-6.0.3-RELEASE-ubuntu24.04-aarch64.tar.gz
42+
43+
export PATH=/home/ubuntu/swift-6.0.3-RELEASE-ubuntu24.04-aarch64/usr/bin:"${PATH}"
44+
45+
swift --version
46+
47+
# Install Docker
48+
sudo apt-get update
49+
sudo apt-get install -y ca-certificates curl
50+
sudo install -m 0755 -d /etc/apt/keyrings
51+
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
52+
sudo chmod a+r /etc/apt/keyrings/docker.asc
53+
54+
# Add the repository to Apt sources:
55+
# shellcheck source=/etc/os-release
56+
# shellcheck disable=SC1091
57+
. /etc/os-release
58+
echo \
59+
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
60+
$VERSION_CODENAME stable" | \
61+
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
62+
sudo apt-get update
63+
64+
sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
65+
66+
# Add the current user to the docker group
67+
sudo usermod -aG docker "$USER"
68+
69+
# LOGOUT and LOGIN to apply the changes
70+
exit 0

Diff for: ‎scripts/ubuntu-test-plugin.sh

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/bash
2+
##===----------------------------------------------------------------------===##
3+
##
4+
## This source file is part of the SwiftAWSLambdaRuntime open source project
5+
##
6+
## Copyright (c) 2025 Apple Inc. and the SwiftAWSLambdaRuntime project authors
7+
## Licensed under Apache License v2.0
8+
##
9+
## See LICENSE.txt for license information
10+
## See CONTRIBUTORS.txt for the list of SwiftAWSLambdaRuntime project authors
11+
##
12+
## SPDX-License-Identifier: Apache-2.0
13+
##
14+
##===----------------------------------------------------------------------===##
15+
16+
# Connect with ssh
17+
18+
export PATH=/home/ubuntu/swift-6.0.3-RELEASE-ubuntu24.04-aarch64/usr/bin:"${PATH}"
19+
20+
# clone a project
21+
git clone https://github.com/swift-server/swift-aws-lambda-runtime.git
22+
23+
# be sure Swift is install.
24+
# Youc an install swift with the following command: ./scripts/ubuntu-install-swift.sh
25+
26+
# build the project
27+
cd swift-aws-lambda-runtime/Examples/ResourcesPackaging/ || exit 1
28+
LAMBDA_USE_LOCAL_DEPS=../.. swift package archive --allow-network-connections docker

0 commit comments

Comments
 (0)