Components

Type: aws:core:glueAddPartitionsArc

Synopsis

The AWS Glue add partitions arc updates the given catalog with newly arrived partitions.

Description

This are listens for lot availability events and extracts the partition values from the dataset
manifest.

It then adds those partitions to the given AWS Glue table specified by the sink pathURI, having the
format:
    glue://[catalog]/database/table

Where the catalog is only required if writing to a different account.

The resulting manifest file will contain uris of the format:
    glue://[catalog]/database/table/value1/value2/...

partitionType: value|named
    the default named takes on the form name=value, name is the partition name and value is the
    partition value

namedPartitionDelimiter: defaults to '='
    the delimiter used to separate the partition name from the partition value

partitionResults: none|all|added
    none: do not list any partitions in the manifest, just the glue:// uri
    all: list all partitions found in the source manifest
    added: list only partitions that did not previously exist in the table

Template

Required

{
  "type" : "aws:core:glueAddPartitionsArc",
  "name" : null,
  "sources" : { },
  "sinks" : { }
}

All

{
  "type" : "aws:core:glueAddPartitionsArc",
  "name" : null,
  "exclude" : false,
  "sources" : { },
  "sinks" : { },
  "workload" : {
    "workloadProps" : {
      "partitionType" : "named",
      "namedPartitionDelimiter" : "=",
      "partitionResults" : "added"
    },
    "runtimeProps" : {
      "memorySizeMB" : 512,
      "retryAttempts" : 3,
      "timeoutMin" : 15,
      "architecture" : "ARM_64"
    }
  }
}