Live Workshop: Integrate Google SecOps with Bindplane - Join Us on February 26th at 11 AM ET!Sign Up Now

Google Cloud PubSub

Description

Read messages from a Google Cloud PubSub topic and generate logs, metrics, or traces from the result.

important

The OpenTelemetry receiver this source uses is in beta and is subject to change.

Supported Platforms

PlatformSupported
Linux✓
Windows✓
macOS✓

Configuration

FieldDescription
Telemetry TypesThe types of telemetry to read from PubSub.
Project IDThe Google Cloud project ID where the PubSub client will connect to.
Subscription NameName of the pre-existing PubSub subscription to read messages from.
Payload EncodingUse "none" for attribute based detection. If the "content-type" attribute is missing from data, this is used to determine the encoding. Metrics and traces can only be OTLP Proto. See Attribute Labels for more information.
Payload CompressionUse "none" for attribute based detection. If the "content-encoding" attribute is missing from data, this is used to determine the compression. See Attribute Labels for more information.
Client IDThe client ID to be used by PubSub to make load balancing decisions.
User AgentThe user agent that will be used by the PubSub client to connect to the service.
TimeoutTimeout for all API calls, default is 12 seconds if unset.
EndpointOverride of the default PubSub endpoint. Leave empty for default.
InsecureIf using a custom endpoint, determines if connection should be insecure.

Authentication

Authentication is handled via the GOOGLE_APPLICATION_CREDENTIALS environment variable. This should point to a file containing the service account key for the project.

To set this for the collector, you need to modify the service file depending on your OS.

Linux

Use a Systemd override:

bash
1sudo systemctl edit observiq-otel-collector

Then run the following command to reload the systemd configuration:

bash
1sudo systemctl daemon-reload
2sudo systemctl restart observiq-otel-collector
Windows

Start powershell as adminstrator and run the following command:

powershell
1reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\observiq-otel-collector" /v Environment /t REG_MULTI_SZ /d "GOOGLE_APPLICATION_CREDENTIALS=<path to key file>" /f

Then restart the collector service:

powershell
1Restart-Service observiq-otel-collector
MacOS

Add GOOGLE_APPLICATION_CREDENTIALS to the EnvironmentVariables dict in the launchd service file /Library/LaunchDaemons/com.observiq.collector.plist (other values are shown for context):

xml
1<key>EnvironmentVariables</key>
2<dict>
3    <key>GOOGLE_APPLICATION_CREDENTIALS</key>
4    <string><path to key file></string>
5    <key>OIQ_OTEL_COLLECTOR_HOME</key>
6    <string>/opt/observiq-otel-collector/</string>
7    <key>OIQ_OTEL_COLLECTOR_STORAGE</key>
8    <string>/opt/observiq-otel-collector/storage</string>
9</dict>

Then restart the collector:

bash
1sudo launchctl unload /Library/LaunchDaemons/com.observiq.collector.plist
2sudo launchctl load /Library/LaunchDaemons/com.observiq.collector.plist

Attribute Labels

The Google Cloud PubSub receiver looks for the following attributes on the message to determine the message encoding and compression:

  • content-type - How the message content should be interpreted.
    • If this value is set to application/protobuf, another attribute, ce-type, is expected to determine telemetry type. This attribute should be one of org.opentelemetry.otlp.traces.v1, org.opentelemetry.otlp.metrics.v1, or org.opentelemetry.otlp.logs.v1.
  • content-encoding - Despite being called the "content-encoding" this controls the compression of the message.

If these attributes are present on the message data, the Payload Encoding and Payload Compression source parameters are ignored and should be set to "none".

If these attributes are not present on the message data, the receiver will use the Payload Encoding and Payload Compression parameters to determine the encoding and compression.

Metrics and traces can only be OTLP Proto. Logs can be OTLP JSON, OTLP Proto, or Raw Text. For more information, see the OpenTelemetry Receiver Documentation.

Example Configuration

In this example, we are parsing log data from the given PubSub subscription. The expected attributes are not present on the PubSub message data, so the Payload Encoding and Payload Compression parameters are used to determine the encoding and compression.

If these attributes were present, then the Payload Encoding and Payload Compression parameters should be set to "none".

Web Interface

observIQ docs - Google Cloud PubSub Source

Standalone Source

yaml
1apiVersion: bindplane.observiq.com/v1
2kind: Source
3metadata:
4  id: googlecloudpubsub
5  name: googlecloudpubsub
6spec:
7  type: googlecloudpubsub
8  parameters:
9    - name: telemetry_types
10      value: ['Logs']
11    - name: project
12      value: 'my_project_id'
13    - name: subscription
14      value: 'my_subscription_name'
15    - name: encoding
16      value: 'raw_text'
17    - name: compression
18      value: 'gzip'