Process Kubernetes Cluster Events for ClickHouse
Pre-processes Kubernetes cluster-level events for ClickHouse ingestion. Structures and normalizes event data for optimal ClickHouse compression, redacts sensitive information (secret references), ensures resource attributes are populated for the otel_logs schema, maps event types to OTLP severity levels, and deduplicates repeated warning events. Retains high-cardinality data for full visibility since ClickHouse handles large datasets efficiently.
Included Processors
Detect Resource Attributes
Detects the resource type and adds any missing resource attributes based on the environment. Select the correct detector for your infrastructure (e.g., gcp, eks, ec2, aks, azure) so that resources are correctly identified and enriched with the appropriate metadata. ClickHouse stores these in the ResourceAttributes map column of the otel_logs table.
Redact Sensitive Information
Redacts secret references and sensitive data from cluster events.
Standardize ClickHouse Formatting
Standardizes event data formatting for optimal ClickHouse compression. Normalizes event reason and type to lowercase, ensures boolean and numeric fields are typed consistently, and formats event counts as integers.
Map Event Type to Severity
Maps Kubernetes event types to OTLP severity levels for the ClickHouse SeverityText and SeverityNumber columns.
Deduplicate Cluster Warnings
Deduplicates repeated K8s warning events within a 60-second window.
Delete Empty Fields
Removes null and empty values to reduce storage overhead.