CVE-2021-41090

Grafana Agent is a telemetry collector for sending metrics, logs, and trace data to the opinionated Grafana observability stack. Prior to versions 0.20.1 and 0.21.2, inline secrets defined within a metrics instance config are exposed in plaintext over two endpoints: metrics instance configs defined in the base YAML file are exposed at /-/config and metrics instance configs defined for the scraping service are exposed at /agent/api/v1/configs/:key. Inline secrets will be exposed to anyone being able to reach these endpoints. If HTTPS with client authentication is not configured, these endpoints are accessible to unauthenticated users. Secrets found in these sections are used for delivering metrics to a Prometheus Remote Write system, authenticating against a system for discovering Prometheus targets, and authenticating against a system for collecting metrics. This does not apply for non-inlined secrets, such as *_file based secrets. This issue is patched in Grafana Agent versions 0.20.1 and 0.21.2. A few workarounds are available. Users who cannot upgrade should use non-inline secrets where possible. Users may also desire to restrict API access to Grafana Agent with some combination of restricting the network interfaces Grafana Agent listens on through http_listen_address in the server block, configuring Grafana Agent to use HTTPS with client authentication, and/or using firewall rules to restrict external access to Grafana Agent's API.

Grafana Agent is a telemetry collector for sending metrics, logs, and trace data to the opinionated Grafana observability stack. Prior to versions 0.20.1 and 0.21.2, inline secrets defined within a metrics instance config are exposed in plaintext over two endpoints: metrics instance configs defined in the base YAML file are exposed at /-/config and metrics instance configs defined for the scraping service are exposed at /agent/api/v1/configs/:key. Inline secrets will be exposed to anyone being able to reach these endpoints. If HTTPS with client authentication is not configured, these endpoints are accessible to unauthenticated users. Secrets found in these sections are used for delivering metrics to a Prometheus Remote Write system, authenticating against a system for discovering Prometheus targets, and authenticating against a system for collecting metrics. This does not apply for non-inlined secrets, such as *_file based secrets. This issue is patched in Grafana Agent versions 0.20.1 and 0.21.2. A few workarounds are available. Users who cannot upgrade should use non-inline secrets where possible. Users may also desire to restrict API access to Grafana Agent with some combination of restricting the network interfaces Grafana Agent listens on through http_listen_address in the server block, configuring Grafana Agent to use HTTPS with client authentication, and/or using firewall rules to restrict external access to Grafana Agent's API.

Impact

Some inline secrets are exposed in plaintext over the Grafana Agent HTTP server:

• Inline secrets for metrics instance configs in the base YAML file are exposed at /-/config
• Inline secrets for integrations are exposed at /-/config
• Inline secrets for Consul ACL tokens and ETCD basic auth when configured for the scraping service at /-/config.
• Inline secrets for the Kafka receiver for OpenTelemetry-Collector tracing at /-/config.
• Inline secrets for metrics instance configs loaded from the scraping service are exposed at /agent/api/v1/configs/{name}.

Inline secrets will be exposed to anyone being able to reach these endpoints.

Secrets found in these sections are used for:

• Delivering metrics to a Prometheus Remote Write system
• Authenticating against a system for discovering Prometheus targets
• Authenticating against a system for collecting metrics (scrape_configs and integrations)
• Authenticating against a Consul or ETCD for storing configurations to distribute in scraping service mode
• Authenticating against Kafka for receiving traces

Non-inlined secrets, such as *_file-based secrets, are not impacted by this vulnerability.

Patches

Download v0.20.1 or any version past v0.21.2 to patch Grafana Agent. These patches obfuscate the listed impacted secrets from the vulnerable endpoints.

The patches also disable the endpoints by default. Pass the command-line flag --config.enable-read-api to opt-in and re-enable the endpoints.

Workarounds

If for some reason you cannot upgrade, use non-inline secrets where possible. Not all configuration options may have a non-inline equivalent.

You also may desire to restrict API access to Grafana Agent, with some combination of:

• Restrict network interfaces Grafana Agent listens on through http_listen_address in the server block. 127.0.0.1 is the most restrictive, 0.0.0.0 is the default.
• Configure Grafana Agent to use HTTPS with client authentication.
• Use firewall rules to restrict external access to Grafana Agent's API.

Grafana Agent is a telemetry collector for sending metrics, logs, and trace data to the opinionated Grafana observability stack. Prior to versions 0.20.1 and 0.21.2, inline secrets defined within a metrics instance config are exposed in plaintext over two endpoints: metrics instance configs defined in the base YAML file are exposed at /-/config and metrics instance configs defined for the scraping service are exposed at /agent/api/v1/configs/:key. Inline secrets will be exposed to anyone being able to reach these endpoints. If HTTPS with client authentication is not configured, these endpoints are accessible to unauthenticated users. Secrets found in these sections are used for delivering metrics to a Prometheus Remote Write system, authenticating against a system for discovering Prometheus targets, and authenticating against a system for collecting metrics. This does not apply for non-inlined secrets, such as *_file based secrets. This issue is patched in Grafana Agent versions 0.20.1 and 0.21.2. A few workarounds are available. Users who cannot upgrade should use non-inline secrets where possible. Users may also desire to restrict API access to Grafana Agent with some combination of restricting the network interfaces Grafana Agent listens on through http_listen_address in the server block, configuring Grafana Agent to use HTTPS with client authentication, and/or using firewall rules to restrict external access to Grafana Agent's API.

Grafana Agent is a telemetry collector for sending metrics, logs, and trace data to the opinionated Grafana observability stack. Prior to versions 0.20.1 and 0.21.2, inline secrets defined within a metrics instance config are exposed in plaintext over two endpoints: metrics instance configs defined in the base YAML file are exposed at /-/config and metrics instance configs defined for the scraping service are exposed at /agent/api/v1/configs/:key. Inline secrets will be exposed to anyone being able to reach these endpoints. If HTTPS with client authentication is not configured, these endpoints are accessible to unauthenticated users. Secrets found in these sections are used for delivering metrics to a Prometheus Remote Write system, authenticating against a system for discovering Prometheus targets, and authenticating against a system for collecting metrics. This does not apply for non-inlined secrets, such as *_file based secrets. This issue is patched in Grafana Agent versions 0.20.1 and 0.21.2. A few workarounds are available. Users who cannot upgrade should use non-inline secrets where possible. Users may also desire to restrict API access to Grafana Agent with some combination of restricting the network interfaces Grafana Agent listens on through http_listen_address in the server block, configuring Grafana Agent to use HTTPS with client authentication, and/or using firewall rules to restrict external access to Grafana Agent's API.

Impact

Some inline secrets are exposed in plaintext over the Grafana Agent HTTP server:

• Inline secrets for metrics instance configs in the base YAML file are exposed at /-/config
• Inline secrets for integrations are exposed at /-/config
• Inline secrets for Consul ACL tokens and ETCD basic auth when configured for the scraping service at /-/config.
• Inline secrets for the Kafka receiver for OpenTelemetry-Collector tracing at /-/config.
• Inline secrets for metrics instance configs loaded from the scraping service are exposed at /agent/api/v1/configs/{name}.

Inline secrets will be exposed to anyone being able to reach these endpoints.

Secrets found in these sections are used for:

• Delivering metrics to a Prometheus Remote Write system
• Authenticating against a system for discovering Prometheus targets
• Authenticating against a system for collecting metrics (scrape_configs and integrations)
• Authenticating against a Consul or ETCD for storing configurations to distribute in scraping service mode
• Authenticating against Kafka for receiving traces

Non-inlined secrets, such as *_file-based secrets, are not impacted by this vulnerability.

Patches

Download v0.20.1 or any version past v0.21.2 to patch Grafana Agent. These patches obfuscate the listed impacted secrets from the vulnerable endpoints.

The patches also disable the endpoints by default. Pass the command-line flag --config.enable-read-api to opt-in and re-enable the endpoints.

Workarounds

If for some reason you cannot upgrade, use non-inline secrets where possible. Not all configuration options may have a non-inline equivalent.

You also may desire to restrict API access to Grafana Agent, with some combination of:

• Restrict network interfaces Grafana Agent listens on through http_listen_address in the server block. 127.0.0.1 is the most restrictive, 0.0.0.0 is the default.
• Configure Grafana Agent to use HTTPS with client authentication.
• Use firewall rules to restrict external access to Grafana Agent's API.

Instance config inline secret exposure in Grafana in github.com/grafana/agent