Process NGINX Logs for Elasticsearch
Optimizes NGINX access and error logs for Elasticsearch ingestion via OTLP. Parses the combined log format, filters health check and static asset requests, maps HTTP status codes to OTLP severity levels, masks sensitive data, removes high-cardinality request identifiers, samples successful responses, and deduplicates error bursts.
Included Processors
Parse NGINX Combined Log Format
Parses NGINX combined log format extracting IP, timestamp, method, path, status code, response size, referrer, and user agent.
Parse NGINX Timestamp
Parses the NGINX timestamp into the log record time field.
Filter Health Checks
Drops health check and monitoring endpoint requests.
Filter Static Asset Requests
Drops requests for static assets (CSS, JS, images, fonts).
Map HTTP Status to Severity
Maps HTTP status codes to OTLP severity levels for Elasticsearch severity-based filtering and alerting.
Mask Sensitive Data
Redacts sensitive data from request paths and user information.
Add ECS-Compatible Fields
Adds Elastic Common Schema fields for HTTP access log data, enabling compatibility with built-in Kibana web server dashboards.
Sample Success Responses
Samples 50% of successful HTTP 2xx response logs to reduce volume.
Deduplicate Error Responses
Deduplicates repeated HTTP error logs within a 30-second window.
Delete Empty Fields
Removes null and empty values to reduce storage overhead.
Example Telemetry
Log
| Name | Before | After |
|---|---|---|
| observed_time | 2023-12-25T21:15:30Z | 2023-12-25T21:15:30Z |
| time | 2023-12-25T21:15:30Z | 2023-12-25T18:15:30Z |
| severity_text | WARN | |
| severity_number | 0 | 13 |
Body
203.0.113.45 - - [25/Dec/2023:10:15:30 -0800] "GET /old-page HTTP/1.1" 301 512 "https://www.example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"| http_version | HTTP/1.1 |
| ip | **** |
| method | GET |
| path | /old-page |
| referrer | https://www.example.com/ |
| response_size | 512 |
| status_code | 301 |
| timestamp | 25/Dec/2023:10:15:30 -0800 |
| user_agent | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 |
Attributes
| Name | Before | After |
|---|---|---|
| event.dataset | nginx.access | |
| event.kind | event |
Resource
| Name | Before | After |
|---|---|---|
| deployment.environment | production | |
| service.name | nginx |