aws lambda layer構築中にThe plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.でコケる。

lambda layerを構築していたが、何故かデプロイできなくなった。

とりあえずデバッグフラグ。

kilin@kilin:~/$ env | grep TF_LOG
TF_LOG=DEBUG

確認。

2024-04-27T21:38:53.960+0900 [TRACE] evalApplyProvisioners: module.guess_api.aws_lambda_layer_version.lambda_layer is tainted, so skipping provisioning
2024-04-27T21:38:53.961+0900 [TRACE] maybeTainted: module.guess_api.aws_lambda_layer_version.lambda_layer was already tainted, so nothing to do
2024-04-27T21:38:53.961+0900 [TRACE] terraform.contextPlugins: Schema for provider "registry.terraform.io/hashicorp/aws" is in the global cache
2024-04-27T21:38:53.961+0900 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.guess_api.aws_lambda_layer_version.lambda_layer
2024-04-27T21:38:53.961+0900 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for module.guess_api.aws_lambda_layer_version.lambda_layer
2024-04-27T21:38:53.971+0900 [TRACE] statemgr.Filesystem: not making a backup, because the new snapshot is identical to the old
2024-04-27T21:38:53.973+0900 [TRACE] statemgr.Filesystem: no state changes since last snapshot
2024-04-27T21:38:53.973+0900 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2024-04-27T21:38:53.976+0900 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2024-04-27T21:38:53.976+0900 [ERROR] vertex "module.guess_api.aws_lambda_layer_version.lambda_layer" error: Request cancelled
2024-04-27T21:38:53.977+0900 [TRACE] vertex "module.guess_api.aws_lambda_layer_version.lambda_layer": visit complete, with errors
2024-04-27T21:38:53.980+0900 [TRACE] dag/walk: upstream of "module.guess_api.aws_lambda_function.guess_api_lambda (expand)" errored, so skipping
2024-04-27T21:38:53.981+0900 [TRACE] dag/walk: upstream of "module.guess_api.aws_lambda_function.guess_api_lambda" errored, so skipping
2024-04-27T21:38:53.982+0900 [TRACE] dag/walk: upstream of "module.guess_api (close)" errored, so skipping
2024-04-27T21:38:53.982+0900 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" errored, so skipping
2024-04-27T21:38:53.982+0900 [TRACE] dag/walk: upstream of "root" errored, so skipping
2024-04-27T21:38:53.984+0900 [TRACE] terraform.contextPlugins: Schema for provider "registry.terraform.io/hashicorp/aws" is in the global cache
2024-04-27T21:38:53.984+0900 [TRACE] statemgr.Filesystem: not making a backup, because the new snapshot is identical to the old
2024-04-27T21:38:53.985+0900 [TRACE] statemgr.Filesystem: no state changes since last snapshot
2024-04-27T21:38:53.985+0900 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
╷
│ Error: Request cancelled
│ 
│   with module.guess_api.aws_lambda_layer_version.lambda_layer,
│   on ../modules/lambda/main.tf line 88, in resource "aws_lambda_layer_version" "lambda_layer":
│   88: resource "aws_lambda_layer_version" "lambda_layer" {
│ 
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵
2024-04-27T21:38:53.998+0900 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2024-04-27T21:38:53.999+0900 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
2024-04-27T21:38:54.002+0900 [DEBUG] provider: plugin exited

よくわからない。ログ実際のログはTRACEしている箇所もある。ひとまずログが長すぎてどうしようもないためERRORだけで引っ掛けてみる。

TF_LOG=ERROR
terraform apply

しかし、

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.guess_api.aws_lambda_layer_version.lambda_layer: Creating...
module.guess_api.aws_lambda_layer_version.lambda_layer: Still creating... [10s elapsed]
2024-04-27T21:45:42.466+0900 [ERROR] plugin.(*GRPCProvider).ApplyResourceChange: error="rpc error: code = Canceled desc = context canceled"
2024-04-27T21:45:42.495+0900 [ERROR] vertex "module.guess_api.aws_lambda_layer_version.lambda_layer" error: Request cancelled
╷
│ Error: Request cancelled
│ 
│   with module.guess_api.aws_lambda_layer_version.lambda_layer,
│   on ../modules/lambda/main.tf line 88, in resource "aws_lambda_layer_version" "lambda_layer":
│   88: resource "aws_lambda_layer_version" "lambda_layer" {
│ 
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵
(lambda-env) kilin@kilin-B450-I-AORUS-PRO-WIFI:~/Documents/Project/climbApp/terraform/dev$

駄目みたい。

一応確認したが、terraform planしても特に問題が見当たらない。

2024-04-27T21:36:45.765+0900 [INFO]  backend/local: plan operation completed

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # module.guess_api.aws_lambda_function.guess_api_lambda will be updated in-place
  ~ resource "aws_lambda_function" "guess_api_lambda" {
        id                             = "guessApi-dev"
      ~ layers                         = [
          - "arn:aws:lambda:ap-northeast-1:369311474110:layer:lambda_layer_climb:4",
        ] -> (known after apply)
        tags                           = {}
        # (27 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # module.guess_api.aws_lambda_layer_version.lambda_layer will be created
  + resource "aws_lambda_layer_version" "lambda_layer" {
      + arn                         = (known after apply)
      + compatible_runtimes         = [
          + "python3.10",
        ]
      + created_date                = (known after apply)
      + filename                    = "../modules/lambda/python.zip"
      + id                          = (known after apply)
      + layer_arn                   = (known after apply)
      + layer_name                  = "lambda_layer_climb"
      + signing_job_arn             = (known after apply)
      + signing_profile_version_arn = (known after apply)
      + skip_destroy                = false
      + source_code_hash            = "rmQ+NESv0D/ezoyMc/Z8zBTWn6fK0hnrV9FpJRKGlQA="
      + source_code_size            = (known after apply)
      + version                     = (known after apply)
    }

Plan: 1 to add, 1 to change, 0 to destroy.

この辺の記事を読んでもイマイチよくわからず。

github.com

原因がわからないと思ったが、そもそもultralyticsで色々放り込んだな？と思い出し、ファイル容量を確認する。

du -sh ./*

2.4GBのデプロイ用zipファイルが存在した。これだ。

サイズを小さくして再度apply。

  Enter a value: yes

module.guess_api.aws_lambda_layer_version.lambda_layer: Creating...
module.guess_api.aws_lambda_layer_version.lambda_layer: Creation complete after 5s [id=arn:aws:lambda:ap-northeast-1:369311474110:layer:lambda_layer_climb:5]
module.guess_api.aws_lambda_function.guess_api_lambda: Modifying... [id=guessApi-dev]
module.guess_api.aws_lambda_function.guess_api_lambda: Modifications complete after 8s [id=guessApi-dev]

Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

OK。

lambda layerのアップロードサイズ上限は50MB(圧縮後)

terraform側では上限に引っかかった旨のエラーは表示されない様子だった。 zipファイルサイズを小さくしてデプロイし直すとデプロイできたので、サイズ上限に引っかかったということがわかる。

docs.aws.amazon.com

こちらによると圧縮zipで50MBで解凍後に250MB以下でないといけないらしい。今回はlayerサイズのクォータ制限を回避する形で対応を行った。

lizminim

忘れていても思い出せるように。

aws lambda layer構築中にThe plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.でコケる。

lambda layerのアップロードサイズ上限は50MB(圧縮後)