我正在尝试将一些简单的CI作业(例如: linters)运行到我的python存储库的lambda函数中。
我使用gitlab-ci来解决这个问题,它的基本架构(与问题没有直接关系,但它可能对选择有所帮助)如下:
CI作业被定义为一组外壳命令,在我的例子中只运行black --check .
(一个python linter):1.外壳命令被发送到我的lambda 2. lambda git克隆存储库3. lambda在子进程中执行命令4.返回结果
所以我的lambda看起来像这样:
import os
import stat
import sys
import json
import subprocess
import stat
import shutil
from git import Repo
def main(event, lambda_context):
# I've commented out, so that if you like trying to reproduce it
# at home, you don't need the additional dependencies :)
#Repo.clone_from("https://example.com/the/repo.git")
# the command sent by the CI having a shebang in it
# the quick and dirty solution we found is to write it into
# a shell script and then executing said script
script = open("/tmp/script.sh", "w")
script.write(event["command"])
script.close()
st = os.stat("/tmp/script.sh")
os.chmod("/tmp/script.sh", st.st_mode | stat.S_IEXEC)
# the copy is just to be extra-safe
copy_env_variables = os.environ.copy()
# for all the binary added by the requirements.txt
# we edit PYTHONPATH for black in the subshell to find the modules
lambda_root_dir = os.environ["LAMBDA_TASK_ROOT"]
copy_env_variables["PATH"] = copy_env_variables["PATH"] + f":{lambda_root_dir}/bin"
copy_env_variables["PYTHONPATH"] = (
copy_env_variables.get("PYTHONPATH", "") + ":" + lambda_root_dir
)
proc = subprocess.Popen(
"/tmp/script.sh",
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
env=copy_env_variables,
)
(stdout, _) = proc.communicate()
return {"return_code": proc.returncode, "output": stdout.decode("utf-8")}
而lambda与这个脚本打包在一起:(python3是python3.7,我在linux x86_64上)
python3 -m pip install -r requirements.txt -t ./
zip -r archive.zip . -x \*.pyc *.git*
requirements.txt如下所示:
gitpython
black
当我执行CI作业时,我得到了子进程的输出:
$ black --check .
Traceback (most recent call last):
File "/var/task/bin/black", line 6, in <module>
from black import patched_main
File "/var/task/black.py", line 15, in <module>
import regex as re
File "/var/task/regex/__init__.py", line 1, in <module>
from .regex import *
File "/var/task/regex/regex.py", line 402, in <module>
import regex._regex_core as _regex_core
File "/var/task/regex/_regex_core.py", line 21, in <module>
import regex._regex as _regex
ModuleNotFoundError: No module named 'regex._regex'
regex._regexp
的特殊之处在于它是一个已编译的模块,名为regex/_regex.cpython-37m-x86_64-linux-gnu.so
,但是如果我将它直接导入到我的lambda代码中,它就可以正常工作。
关于python处理来自.so库的模块的方式,我是不是遗漏了什么?
发布于 2019-11-05 17:44:01
问题来自于二进制文件中包含/usr/bin/python3
并指向python3.6,因此python3.6无法加载为python3.7编译的.so
奇怪的是,尽管Lambda是使用python3.7运行时部署的。在lambda中,python3.7位于/var/lang/bin/python3.7
中,所以目前我已经将我的配置项修改为/var/lang/bin/python3.7
中的symlink python3.7,这样pip
生成的shebang就是正确的lambda了,
发布于 2019-11-05 18:03:09
你在你的机器上打包lambda。Lambda运行时使用(可能)不同的linux发行版。您可能希望使用Docker启动amazonlinux:latest
镜像,并从那里打包lambda。(你可以在这里找到步骤https://smirnov-am.github.io/ci-ci-pipeline-for-aws-lambda-python-runtime/)
https://stackoverflow.com/questions/58698406
复制相似问题