Functionbeat被部署为lambda，但是状态显示没有收到cloudwatch日志的数据
Functionbeat被部署为lambda，但是状态显示没有收到cloudwatch日志的数据
提问于 2021-10-20 21:54:07
我遵循这个tutorial并为一个cloudwatch日志组配置和部署了functionbeat。我还添加了elastic search和kibana端点。以下是输出
MacBook-Pro:functionbeat-7.10.2-darwin-x86_64 user$ ./functionbeat setup -e
2021-10-20T22:57:35.097-0400 INFO instance/beat.go:645 Home path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Config path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2021-10-20T22:57:35.098-0400 INFO instance/beat.go:653 Beat ID: c9cbbe8c-319a-4577-be8e-de223fba4f6e
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "data": "/tmp", "home": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "logs": "/tmp/logs"}, "type": "functionbeat", "uuid": "c9cbbe8c-319a-4577-be8e-de223fba4f6e"}}}
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T22:39:38.000Z", "version": "7.10.2"}}}
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":8,"version":"go1.14.12"}}}
2021-10-20T22:57:35.101-0400 INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-10-13T02:09:16.051008-04:00","name":"Jijos-MacBook-Pro.local","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::1ce5:e86b:6d5c:88d9/64","192.168.1.11/24","fe80::84d7:afff:fe5e:107c/64","fe80::84d7:afff:fe5e:107c/64","fe80::46cf:bba0:8bb3:5b62/64","fe80::99c:e0d2:dbd5:e7aa/64","fe80::cdce:ea46:e3c:3feb/64","fe80::76aa:a49d:6061:d997/64","fe80::aede:48ff:fe00:1122/64"],"kernel_version":"20.6.0","mac":["a6:83:e7:8a:24:01","a4:83:e7:8a:24:01","86:d7:af:5e:10:7c","86:d7:af:5e:10:7c","82:ea:c5:62:c4:05","82:ea:c5:62:c4:04","82:ea:c5:62:c4:01","82:ea:c5:62:c4:00","82:ea:c5:62:c4:01","ac:de:48:00:11:22"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.16","major":10,"minor":16,"patch":0,"build":"20G165"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"BEEB9A65-6CAE-51E9-B16E-BCE2FBB9EED9"}}}
2021-10-20T22:57:35.101-0400 INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"cwd": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "exe": "./functionbeat", "name": "functionbeat", "pid": 13218, "ppid": 8720, "start_time": "2021-10-20T22:57:34.835-0400"}}}
2021-10-20T22:57:35.101-0400 INFO instance/beat.go:299 Setup Beat: functionbeat; Version: 7.10.2
2021-10-20T22:57:35.102-0400 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'functionbeat-7.10.2' as ILM is enabled.
2021-10-20T22:57:35.103-0400 INFO eslegclient/connection.go:99 elasticsearch url: https://l-es.xyz.io:443
2021-10-20T22:57:35.106-0400 INFO [publisher] pipeline/module.go:113 Beat name: Jijos-MacBook-Pro.local
2021-10-20T22:57:35.107-0400 INFO eslegclient/connection.go:99 elasticsearch url: https://l-es.xyz.io:443
2021-10-20T22:57:35.316-0400 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2021-10-20T22:57:35.366-0400 INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2021-10-20T22:57:35.420-0400 INFO template/load.go:117 Try loading template functionbeat-7.10.2 to Elasticsearch
2021-10-20T22:57:35.496-0400 INFO template/load.go:109 template with name 'functionbeat-7.10.2' loaded.
2021-10-20T22:57:35.496-0400 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
Index setup finished.在部署functionbeat时，我也能够成功地部署它
MacBook-Pro:functionbeat-7.10.2-darwin-x86_64 user$ ./functionbeat -v -e -d "*" deploy fn-cloudwatch-logs
2021-10-20T22:58:30.531-0400 INFO instance/beat.go:645 Home path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Config path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2021-10-20T22:58:30.532-0400 DEBUG [beat] instance/beat.go:697 Beat metadata path: /tmp/meta.json
2021-10-20T22:58:30.532-0400 INFO instance/beat.go:653 Beat ID: c9cbbe8c-319a-4577-be8e-de223fba4f6e
2021-10-20T22:58:30.535-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:126 add_cloud_metadata: starting to fetch metadata, timeout=3s
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for azure after 3.000470363s. result=[provider:azure, error=failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for aws after 3.000674614s. result=[provider:aws, error=failed requesting aws metadata: Get "http://169.254.169.254/2014-02-25/dynamic/instance-identity/document": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for openstack after 3.000710413s. result=[provider:openstack, error=failed requesting openstack metadata: Get "http://169.254.169.254/2009-04-04/meta-data/instance-id": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for gcp after 3.000737886s. result=[provider:gcp, error=failed requesting gcp metadata: Get "http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for digitalocean after 3.000758996s. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get "http://169.254.169.254/metadata/v1.json": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:129 add_cloud_metadata: fetchMetadata ran for 3.000778492s
2021-10-20T22:58:33.536-0400 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2021-10-20T22:58:33.536-0400 DEBUG [processors] processors/processor.go:120 Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], add_cloud_metadata={}
2021-10-20T22:58:33.538-0400 DEBUG [cli-handler] cmd/cli_handler.go:52 Starting deploy for: fn-cloudwatch-logs
2021-10-20T22:58:33.539-0400 DEBUG [aws] aws/cli_manager.go:119 Deploying function: fn-cloudwatch-logs
2021-10-20T22:58:33.539-0400 DEBUG [provider] aws/template_builder.go:90 Compressing all assets into an artifact
2021-10-20T22:58:35.284-0400 DEBUG [provider] aws/template_builder.go:96 Compression is successful (zip size: 22046351 bytes)
2021-10-20T22:58:35.339-0400 INFO [provider] aws/template_builder.go:155 No role is configured for function fn-cloudwatch-logs, creating a custom role.
2021-10-20T22:58:35.342-0400 DEBUG [aws] aws/cli_manager.go:69 Using cloudformation template:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"fnbfncloudwatchlogs": {
"Properties": {
"Code": {
"S3Bucket": "functionbeat-deploy-bucket-test-poc",
"S3Key": "functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip"
},
"Description": "lambda function for cloudwatch logs",
"Environment": {
"Variables": {
"BEAT_STRICT_PERMS": "false",
"ENABLED_FUNCTIONS": "fn-cloudwatch-logs"
}
},
"FunctionName": "fn-cloudwatch-logs",
"Handler": "functionbeat-aws",
"MemorySize": 128,
"ReservedConcurrentExecutions": 5,
"Role": {
"Fn::GetAtt": [
"fnbfncloudwatchlogsIAMRoleLambdaExecution",
"Arn"
]
},
"Runtime": "go1.x",
"Timeout": 3
},
"Type": "AWS::Lambda::Function"
},
"fnbfncloudwatchlogsIAMRoleLambdaExecution": {
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": {
"Fn::Join": [
"",
[
"lambda.",
{
"Ref": "AWS::URLSuffix"
}
]
]
}
}
}
]
},
"Path": "/",
"Policies": [
{
"PolicyDocument": {
"Statement": [
{
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": [
{
"Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/fn-cloudwatch-logs:*"
}
]
}
]
},
"PolicyName": {
"Fn::Join": [
"-",
[
"fnb",
"lambda",
"fn-cloudwatch-logs"
]
]
}
}
],
"RoleName": "functionbeat-lambda-fn-cloudwatch-logs"
},
"Type": "AWS::IAM::Role"
},
"fnbfncloudwatchlogsLogGroup": {
"Properties": {
"LogGroupName": "/aws/lambda/fn-cloudwatch-logs"
},
"Type": "AWS::Logs::LogGroup"
},
"fnbfncloudwatchlogsPermission0": {
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"fnbfncloudwatchlogs",
"Arn"
]
},
"Principal": {
"Fn::Join": [
"",
[
"logs.",
{
"Ref": "AWS::Region"
},
".",
{
"Ref": "AWS::URLSuffix"
}
]
]
},
"SourceArn": {
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":logs:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":log-group:",
"/aws/containerinsights/translator-eks-ci-blue-cluster/application",
":*"
]
]
}
},
"Type": "AWS::Lambda::Permission"
},
"fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication": {
"Properties": {
"DestinationArn": {
"Fn::GetAtt": [
"fnbfncloudwatchlogs",
"Arn"
]
},
"FilterPattern": "mylog_",
"LogGroupName": "/aws/containerinsights/translator-eks-ci-blue-cluster/application"
},
"Type": "AWS::Logs::SubscriptionFilter"
}
}
}
2021-10-20T22:58:35.343-0400 DEBUG [aws.executor] executor/executor.go:53 The executor is executing '6' operations for converging state
2021-10-20T22:58:35.343-0400 DEBUG [aws] aws/op_ensure_bucket.go:33 Verifying presence of S3 bucket: functionbeat-deploy-bucket-test-poc
2021-10-20T22:58:35.548-0400 DEBUG [aws] aws/op_upload_to_bucket.go:44 Uploading file 'functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip' to bucket 'functionbeat-deploy-bucket-test-poc' with size 22046351 bytes
2021-10-20T22:58:36.286-0400 DEBUG [aws] aws/op_upload_to_bucket.go:57 Upload successful
2021-10-20T22:58:36.287-0400 DEBUG [aws] aws/op_upload_to_bucket.go:44 Uploading file 'functionbeat-deployment/fn-cloudwatch-logs/2_AZvlkBMEoQfFeV_dSW5B2VD927AWycnifwEPnFtcI/cloudformation-template-create.json' to bucket 'functionbeat-deploy-bucket-test-poc' with size 4231 bytes
2021-10-20T22:58:36.328-0400 DEBUG [aws] aws/op_upload_to_bucket.go:57 Upload successful
2021-10-20T22:58:36.328-0400 DEBUG [aws] aws/op_cloudformation.go:48 Creating CloudFormation create stack request
2021-10-20T22:58:36.848-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-fn-cloudwatch-logs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2021-10-20T22:58:40.973-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:58:40.973-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:58:43.035-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:58:43.035-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:58:45.091-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:58:55.375-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:58:57.429-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:01.542-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:05.670-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:07.727-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:18.052-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:19.735-0400 DEBUG [aws] aws/op_delete_file_bucket.go:38 Removing file 'functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip' on bucket 'functionbeat-deploy-bucket-test-poc'
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/op_delete_file_bucket.go:51 Remove successful
2021-10-20T22:59:19.808-0400 DEBUG [aws.executor] executor/executor.go:68 All operations successful
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/cli_manager.go:125 Successfully created function 'fn-cloudwatch-logs'
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/cli_manager.go:126 Deploy finish for function 'fn-cloudwatch-logs'
Function: fn-cloudwatch-logs, deploy successful
2021-10-20T22:59:19.810-0400 DEBUG [cli-handler] cmd/cli_handler.go:64 Deploy execution ended在亚马逊网络服务控制台中，我看到正在创建函数fn-cloudwatch-logs，我还在s3存储桶中看到了cloudformation模板。我还看到一个触发器被添加到cloudwatch日志组中，但是kibana中的functionbeat状态显示为No Data is being received。
我还尝试将一个示例日志数据提供给lambda函数，结果在cloudwatch中抛出了一个Could not parse events from cloudwatch错误
2021-10-21T19:17:32.605Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://******.io:443)): Connection marked as failed because the onConnect callback failed: 169.254.40.221 requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of 169.254.40.221.
2021-10-21T19:17:32.605Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://******.io:443)): Connection marked as failed because the onConnect callback failed: 169.254.40.221 requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of 169.254.40.221.我在配置中是否遗漏了什么？
回答 1
Stack Overflow用户
发布于 2021-11-02 01:47:16
这个EOF错误是因为它需要一些输入，但没有接收到。因为你直接调用的函数是beat lambda函数，而这个函数需要被自动调用，所以我们给出了触发器。问题可能是因为AWS无法与您的弹性云建立成功的连接。通常是协议问题(如果您将output.elasticsearch: host设置为localhost:9200，因为除非是公共url，否则AWS无法到达此本地主机url)或权限问题。如果你查看functionbeat lambda函数cloudwatch日志，你可以看到实际的问题。在functionbeat.yml中放入logging.level: debug以查看详细日志。
此外，在部署functionbeat之后，您不能立即在kibana中看到日志。在成功部署后，一旦订阅过滤器添加到日志组中，您就必须调用已添加订阅过滤器的函数，而不是functionbeat lambda函数。因为触发器被添加到function _ Because _ lambda函数。因此，只要有新的项目被添加到这个日志组中，它就会自动调用functionbeat lambda函数。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/69656531
复制相关文章
![python求平均值的怎么编写,python 怎么求平均值[通俗易懂]](https://ask.qcloudimg.com/http-save/yehe-8223537/9d0a8c05763138544dc208708a44873b.jpg)
点击加载更多
相似问题
Microsoft Graph API / Office Planner数据的权限访问令牌
Microsoft Graph Api -No权限在请求Me.Photo时访问令牌
Microsoft Graph API的权限
Microsoft Graph API -获取组-权限错误
Microsoft Graph API访问令牌验证失败(java)
添加站长 进交流群
领取专属 10元无门槛券
AI混元助手 在线答疑
关注 腾讯云开发者公众号
洞察 腾讯核心技术
剖析业界实践案例
腾讯云开发者
