Process Logs for ClickHouse
Comprehensive log hygiene pipeline for ClickHouse ingestion. Parses JSON bodies into structured attributes, coalesces common field name variants into standard names, parses severity into OTEL-standard levels, standardizes attribute naming conventions, masks sensitive data, and groups logs by resource for optimal ClickHouse compression and query performance.
Included Processors
Parse JSON Body
Parses JSON log body into attributes for downstream processing.
Coalesce Message Fields
Normalizes msg/log field variants to a standard "message" attribute.
Coalesce Severity Fields
Normalizes lvl/level field variants to a standard "severity" attribute.
Map Severity to OTEL Standard
Maps common severity text values from the coalesced "severity" attribute into OTEL-standard severity_text and severity_number fields for optimal ClickHouse SeverityNumber column queries.
Mask Sensitive Fields
Redacts emails, IPs, credit cards, SSNs, and other PII from log data.
Batch Logs
Batches logs for optimal ClickHouse insert performance.
Delete Empty Fields
Removes null values and empty maps to optimize storage.