🚀 Bindplane's first Launch Week goes live on June 2nd! New features launching all week.Explore now

AWS S3 Event

The AWS S3 Event Receiver consumes S3 event notifications for object creation events (s3:ObjectCreated:*) and emits the S3 object as the string body of a log record.

Supported Platforms

PlatformSupported
Linux
Windows
macOS

Available in the Bindplane Distro for OpenTelemetry Collector v1.74.0+.

Prerequisites

  • An AWS account with access to S3 and SQS.
  • An SQS queue configured to receive S3 event notifications.
  1. Ensure the collector has permission to read and delete messages from the SQS queue.
  2. Ensure the collector has permission to read objects from the S3 bucket.

How It Works

  1. The receiver polls an SQS queue for S3 event notifications.
  2. When an object creation event (s3:ObjectCreated:*) is received, the receiver downloads the S3 object.
  3. The receiver reads the object into the body of a new log record.
  4. Non-object creation events are ignored but removed from the queue.
  5. If an S3 object is not found (404 error), the corresponding SQS message is preserved for retry later.

Configuration Fields

FieldTypeDefaultRequiredDescription
sqs_queue_urlstringtrueThe URL of the SQS queue to poll for S3 event notifications.
standard_poll_intervalduration15falseThe interval (in seconds) at which the SQS queue is polled for messages.
max_poll_intervalduration2falseThe maximum interval (in seconds) at which the SQS queue is polled for messages.
polling_backoff_factorfloat2falseThe factor by which the polling interval is multiplied after an unsuccessful poll.
workersint5falseThe number of workers to process events.
visibility_timeoutduration300falseHow long (in seconds) messages received from the queue will be invisible to other consumers.
max_log_sizeint1048576falseThe maximum size of a log record in bytes. Logs exceeding this size will be split
max_logs_emittedint1000falseThe maximum number of log records to emit in a single batch. A higher number will result in fewer batches, but more memory.

Example Configuration

observIQ docs - AWS S3 Event