服务
关于
CloudProse博客
聚光灯

Lambda目的地:我们从艰难的道路中学到的东西

您可能会说这与旅程有关,而不是目的地
贾里德·崔特(Jared Short Trek)
贾里德·肖特(Jared Short) | 2019年12月13日

最近引入的Lambda目标是一种将事件从AWS Lambda函数有效地定向到AWS中各种服务的新方法。过去一周,我们在Trek10的功能上进行了探讨,在此过程中,我认为可以分享一些惊喜和艰辛的经验教训。

如发行说明和文档所述,目标仅是异步(事件)Lambda调用以及来自流源的调用的功能。

除目的地外,我们还获得了许多其他配置来管理重试。与其他流引入的旋钮结合使用,还可以使流方案更加健壮。

您可以将以下服务定位到目标位置,从而轻松地在事件之间穿梭,而无需编写所有粘合代码。

Lambda目的地包括AWS Lambda,Amazon SNS,Amazon SQS,Amazon EventBridge

(资源: AWS博客)

它是什么样子的

在任何异步函数调用上(不流源,稍后再说), you can set a destination for onSuccessonFailure.

If your AWS Lambda function fully executes without any error, your responsePayload 和 some additional details are shipped off to your destination. If you function errors for any reason, we'll see that come through as an error.

让我们假设我们有一个简单的AWS Lambda函数,其中包含以下代码。

module.exports.handler = 异步的 event => {
 return event;
}

We want to ship this to an SQS queue onSuccess. We need to give the Lambda Execution Role rights to sqs:SendMessage to that queue (precisely as if we were doing this in code), 和 then set the onSuccess destination to the SQS queue ARN.

Next we invoke our function. aws lambda invoke --function-name test-destinations --invocation-type Event --payload '{"my": "event"}'.

如果检查队列,则会看到以下内容。

{
    "version": "1.0",
    "timestamp": "2019-12-13T20:04:08.088Z",
    "requestContext": {
        "requestId": "f14f5ab1-410f-4162-8c7b-c3f6b276a28c",
        "functionArn": "arn:aws:lambda:us-east-1:454679818906:function:sfn-lab-test-Stream-1UKZ0V094MA7T-StreamProcessor-12O2ALY0Z59LC:$LATEST",
        "condition": "Success",
        "approximateInvokeCount": 1
    },
    "requestPayload": {
        "my": "event"
    },
    "responseContext": {
        "statusCode": 200,
        "executedVersion": "$LATEST"
    },
    "responsePayload": {
        "my": "event"
    }
}

现在,关于学习事物 艰难的道路。您 不能 使用控制台中的“测试”按钮测试目标。它根本不会触发已配置的目标,我相信这是因为控制台调用是同步的,而不是事件类型(异步)。我们花了些力气反对这一点,单击测试,在控制台中看到执行成功,然后在目的地没有任何反应。这显然是回头看,但容易误解。

You'll also have noticed we get a ton of wrapper information 和 data. This can be particularly useful if a destination gets lots of events from different sources, or in the case of an error, we can investigate the requestPayload much closer 和 see what we can do to recover that data or from a bad state.

例如,给出以下代码。

module.exports.handler = 异步的 event => {
 throw new Error("failure example");
 return event;
}

We get the following result in our onFailure SQS Queue.

{
    "version": "1.0",
    "timestamp": "2019-12-13T20:06:50.820Z",
    "requestContext": {
        "requestId": "0972f748-a94c-4902-9ddb-8479e915c0b2",
        "functionArn": "arn:aws:lambda:us-east-1:454679818906:function:sfn-lab-test-Stream-1UKZ0V094MA7T-StreamProcessor-12O2ALY0Z59LC:$LATEST",
        "condition": "RetriesExhausted",
        "approximateInvokeCount": 3
    },
    "requestPayload": {
        "my": "event"
    },
    "responseContext": {
        "statusCode": 200,
        "executedVersion": "$LATEST",
        "functionError": "Unhandled"
    },
    "responsePayload": {
        "errorType": "Error",
        "errorMessage": "failure example",
        "trace": [
            "Error: failure example",
            " at Runtime.module.exports.handler (/var/task/app.js:5:9)",
            " at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
        ]
    }
}

This makes for some pretty powerful paradigms 和 robust architectures. We now know that our payload of {"my": "event"} failed to process, the error message, etc. This makes handling 和 debugging pretty straight forward. Push 所有 these into a queue 和 automatically process later, or queue for developer review.

流事件源是一种特殊的野兽

使用目标时,您必须意识到一些大惊喜,主要是如果您习惯于使用CLI或控制台调用lambda函数来测试/模拟流。

Stream-based lambdas have an entirely different way of creating the onFailure destination. You do not configure them on the function destination configuration, rather you do so in the EventSourceMapping. In fact, you cannot even set an onSuccess destination. Presumably, because of scale reasons, AWS doesn't want to give you the ability to run over the destination services with massively scaled stream infrastructure.

Also note, you can set both EventSourceMapping onFailure 和 the 异步的 onSuccessonFailure for the same function. Your destinations will be routed to based on the invocation type.

Your EventSourceMapping onFailure destination can only be one of SNS or SQS. No EventBridge, no other AWS Lambda functions. Once again, make sure that you are giving your function execution roles proper permissions to publish to your destination!

现在, 另一个艰难的教训. Even if you get smart from learning you can't use the console to test your destinations 和 use the CLI to invoke as an Event type, it will not get routed to your stream based onFailure configuration. It 必须 来自实际流源的事件。请注意这一点,并且知道要进行测试,您需要在DynamoDB中更改或创建记录,或者通过Kinesis推送内容。

流还支持一些非常巧妙的附加功能,例如定义每次调用要分批处理的事件数,批处理事件时在窗口中等待多长时间,在重试之前将批处理二等分(对于查明“毒丸”数据非常有用。您可以在上阅读有关这些功能的更多信息。 AWS博客.

{
 "Type" : "AWS::Lambda::EventSourceMapping",
 "Properties" : {
 "BatchSize" : Integer,
 "BisectBatchOnFunctionError" : Boolean,
 "DestinationConfig" : DestinationConfig,
 "Enabled" : Boolean,
 "EventSourceArn" : String,
 "FunctionName" : String,
 "MaximumBatchingWindowInSeconds" : Integer,
 "MaximumRecordAgeInSeconds" : Integer,
 "MaximumRetryAttempts" : Integer,
 "ParallelizationFactor" : Integer,
 "StartingPosition" : String
 }
}

我还建议您检查一下文档(CloudFormation最终是最有用的),以了解差异。 异步目标配置EventSourceMapping目标配置.

重要说明和注意事项

我们还遇到了其他一些陷阱和经验教训,前面已经提到了一些,但是这里是列表。以真正的节日形式...

当事情发生变化或得到修复或更新时,我们将尽量保持最新状态。

  • 您的lambda执行策略必须具有放置,发布或调用目标源的权限。就像在业务逻辑代码中处理事务一样。
  • Streams (Kinesis, DynamoDB) EventSourceMappings only support onFailure. If you want to do something with the events on success, you do it in your code.
  • The console invocations of lambda functions do not test destinations, only CLI calls of the Event type.
  • When "adding" a destination in the console, if a destination already exists for the Async onFailure or onSuccess 和 you try to point it to another destination, it silently overwrites the previous one. It's more of an "update" than an "add" in those cases. Semantics, but worth knowing.
  • 您无法在控制台中更新Stream EventSourceMappings。如果要编辑批量大小或启用二等分,减少重试次数等,则必须完全删除并重新创建它。
  • 截至目前,CloudFormation似乎无法正确识别流EventSourceMapping更新(我们已创建了支持票证)。
  • 所有信息图表和文档中都缺少SQS作为您去目的地的功能的来源。我们仍在进行实验,但这是奇怪的。
  • 特别是周围的文档之间的区别 流目的地异步的 可能更精确。

有用的图表

非常感谢 阿甘(Forrest Brazeal),谁为此职位贡献了研究。

作者
贾里德·崔特(Jared Short Trek)
贾里德·肖特(Jared Short)