Sep 16, 2025
ClusterLogForwarder on OpenShift
Introduction
The goal of this article is to present the information necessary to demonstrate cluster log forwarding from an OpenShift cluster using the ClusterLogForwarder functionality based on the Vector collector from Red Hat OpenShift Logging 6.x.
The OpenShift Logging Operator provides the ClusterLogging and ClusterLogForwarder CRs. The latter provides the ability to forward logs to external destinations. Logs are not stored on the OpenShift cluster by default. The optional Loki operator with access to object storage is used to provide log storage functionality.
Prerequisites
- An OpenShift 4.x cluster
- A receiver that can ingest the forwarded logs
Running VictoriaLogs
Most likely you will already have a service in the environment that will ingest the logs that are forwarded from OpenShift. However, if you are looking to test this functionality in a lab environment you might be looking for a quick and dirty solution to receive the logs to validate your pipeline configuration. One such option is to use VictoriaLogs.
There are multiple options for deploying VictoriaLogs but the easiest is to simply run the container.
Create a folder to hold the collected logs
mkdir /data/victoria-logs-data
Run the pod
podman run --name victoria-logs --rm -it -p 9428:9428 -v /data/victoria-logs-data:/victoria-logs-data:Z docker.io/victoriametrics/victoria-logs:v1.26.0 -storageDataPath=victoria-logs-data
Use Podlet to generate Podman Quadlet files
There is a great utility called Podlet that will transform a podman run command into the proper syntax for the systemd unit file. You can download the podlet binary or just use podman to run it as a container without downloading and installing:
podman run ghcr.io/containers/podlet --install podman run --name victoria-logs --rm -it -p 9428:9428 -v /data/victoria-logs-data:/victoria-logs-data:Z docker.io/victoriametrics/victoria-logs:v1.26.0 -storageDataPath=victoria-logs-data
The output should look something like this:
victoria-logs.container
[Container]
ContainerName=victoria-logs
Exec='-storageDataPath=victoria-logs-data'
Image=docker.io/victoriametrics/victoria-logs:v1.26.0
PodmanArgs=--interactive --tty
PublishPort=9428:9428
Volume=/data/victoria-logs-data:/victoria-logs-data:Z
[Install]
WantedBy=default.target
Save that above output to a file: /usr/share/containers/systemd/victoria-logs.container
Use systemd to control the new service
sudo systemctl daemon-reload
sudo systemctl start victoria-logs.service
sudo systemctl status victoria-logs.service
Query the logs
Logs stored in VictoriaLogs can be queried at the /select/logsql/query HTTP endpoint. The LogsQL query must be passed via query argument. For example, to query all of the logs:
curl http://<hostname>:9428/select/logsql/query -d 'query=*'|jq '.'
Live tailing
# All logs
curl http://<hostname>:9428/select/logsql/tail -d 'query=*'|jq '.'
# LogsQL to tail on a specific host
curl http://<hostname>:9428/select/logsql/tail -d 'query=hostname:"<my_FQDN>"'|jq '.'
Setup OpenShift ClusterLogForwarder
The configuration below will start the pipeline using the default inputRefs for application and infrastructure. Next the pipeline will apply two different filters, one for adding additional labels to the logs and one to drop all messages that do not have the level of critical, error, or warning.
The add-labels “filter” is being used to add the labels for cluster_name and cluster_type.
The output uses the Elasticsearch sink type and the URL specifies the HTTP Query string parameters for _msg_field, _time_field and _stream_fields.
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
annotations:
observability.openshift.io/tech-preview-otlp-output: enabled
name: logging-forwarder-victora-logs
namespace: openshift-logging
spec:
collector:
resources:
limits:
memory: 32Gi
filters:
- name: add-labels
openshiftLabels:
cluster_name: sam
cluster_type: ocp-virt
type: openshiftLabels
- drop:
- test:
- field: .level
notMatches: (?i)critical|error|warning
name: keep-critical
type: drop
managementState: Managed
outputs:
- elasticsearch:
index: ocp-virt
url: http://elmer.example.com:9428/insert/elasticsearch/_bulk?_stream_fields=log_type,hostname,stream,kubernetes.pod_name,kubernetes.container_name,kubernetes.pod_namespace&_time_field=@timestamp&_msg_field=message,msg,_msg,log.msg,log.message,log&fake_field=1
version: 8
name: elasticsearch-output
type: elasticsearch
pipelines:
- filterRefs:
- keep-critical
- add-labels
inputRefs:
- application
- infrastructure
name: logs-to-es
outputRefs:
- elasticsearch-output
serviceAccount:
name: collector
VictoriaLogs
Here is an example of the raw JSON of a log queried from VictoriaLogs using the “All logs” query from above.
{
"_time": "2025-10-27T18:12:29.894389919Z",
"_stream_id": "0000000000000000105e188d66f8ce2d2682acf51e42ea2a",
"_stream": "{hostname=\"sam.example.com\",kubernetes.container_name=\"manager\",kubernetes.pod_name=\"vault-secrets-operator-controller-manager-769bf97ff-mfch6\",log_type=\"infrastructure\"}",
"_msg": "{\"level\":\"error\",\"ts\":\"2025-10-27T18:12:29Z\",\"logger\":\"cachingClientFactory\",\"msg\":\"Failed to get NewClientWithLogin\",\"controller\":\"vaultstaticsecret\",\"controllerGroup\":\"secrets.hashicorp.com\",\"controllerKind\":\"VaultStaticSecret\",\"VaultStaticSecret\":{\"name\":\"vss-etcd-backup\",\"namespace\":\"openshift-etcd-backup\"},\"namespace\":\"openshift-etcd-backup\",\"name\":\"vss-etcd-backup\",\"reconcileID\":\"8ec961f7-85c7-40f6-9ccc-9660dac6a9f1\",\"cacheKey\":\"approle-9b008cbf5ce63e51f52c70\",\"error\":\"Error making API request.\\n\\nNamespace: openshift-config\\nURL: PUT https://vault.example.com:8200/v1/auth/vso/login\\nCode: 503. Errors:\\n\\n* Vault is sealed\",\"stacktrace\":\"github.com/hashicorp/vault-secrets-operator/vault.(*cachingClientFactory).Get\\n\\t/home/runner/work/vault-secrets-operator/vault-secrets-operator/vault/client_factory.go:483\\ngithub.com/hashicorp/vault-secrets-operator/controllers.(*VaultStaticSecretReconciler).Reconcile\\n\\t/home/runner/work/vault-secrets-operator/vault-secrets-operator/controllers/vaultstaticsecret_controller.go:94\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:216\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:461\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:421\\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\\n\\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:296\"}",
"hostname": "sam.example.com",
"kubernetes.annotations.capabilities": "Seamless Upgrades",
"kubernetes.annotations.categories": "Security",
"kubernetes.annotations.containerImage": "registry.connect.redhat.com/hashicorp/vault-secrets-operator@sha256:84ac7003c4cb90c35f1e296585bb58187a86d64f972133e2661839bdb532ddb8",
"kubernetes.annotations.createdAt": "2025-09-26T21:50:51Z",
"kubernetes.annotations.description": "The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.",
"kubernetes.annotations.features.operators.openshift.io/cnf": "false",
"kubernetes.annotations.features.operators.openshift.io/cni": "false",
"kubernetes.annotations.features.operators.openshift.io/csi": "false",
"kubernetes.annotations.features.operators.openshift.io/disconnected": "true",
"kubernetes.annotations.features.operators.openshift.io/fips-compliant": "false",
"kubernetes.annotations.features.operators.openshift.io/proxy-aware": "true",
"kubernetes.annotations.features.operators.openshift.io/tls-profiles": "false",
"kubernetes.annotations.features.operators.openshift.io/token-auth-aws": "false",
"kubernetes.annotations.features.operators.openshift.io/token-auth-azure": "false",
"kubernetes.annotations.features.operators.openshift.io/token-auth-gcp": "false",
"kubernetes.annotations.k8s.v1.cni.cncf.io/network-status": "[{\n \"name\": \"ovn-kubernetes\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.128.0.195\"\n ],\n \"mac\": \"0a:58:0a:80:00:c3\",\n \"default\": true,\n \"dns\": {}\n}]",
"kubernetes.annotations.kubectl.kubernetes.io/default-container": "manager",
"kubernetes.annotations.olm.operatorGroup": "global-operators",
"kubernetes.annotations.olm.operatorNamespace": "openshift-operators",
"kubernetes.annotations.openshift.io/scc": "restricted-v2",
"kubernetes.annotations.operators.operatorframework.io/builder": "operator-sdk-v1.33.0",
"kubernetes.annotations.operators.operatorframework.io/project_layout": "go.kubebuilder.io/v3",
"kubernetes.annotations.repository": "https://github.com/hashicorp/vault-secrets-operator",
"kubernetes.annotations.seccomp.security.alpha.kubernetes.io/pod": "runtime/default",
"kubernetes.annotations.support": "HashiCorp",
"kubernetes.container_id": "cri-o://1f8fef8ce9b813a725bf664253ebf3a1cd7d52cd323d1d25c9f09d106f5b84e6",
"kubernetes.container_image": "registry.connect.redhat.com/hashicorp/vault-secrets-operator@sha256:84ac7003c4cb90c35f1e296585bb58187a86d64f972133e2661839bdb532ddb8",
"kubernetes.container_image_id": "registry.connect.redhat.com/hashicorp/vault-secrets-operator@sha256:56251cd14b99a6078edf44ba01c2fbd42d123466c5efa24911fed0566e8a876e",
"kubernetes.container_iostream": "stderr",
"kubernetes.container_name": "manager",
"kubernetes.labels.app_kubernetes_io_component": "controller-manager",
"kubernetes.labels.control-plane": "controller-manager",
"kubernetes.labels.pod-template-hash": "769bf97ff",
"kubernetes.namespace_id": "1e159892-99c2-4547-931c-a0026db8824c",
"kubernetes.namespace_labels.kubernetes_io_metadata_name": "openshift-operators",
"kubernetes.namespace_labels.openshift_io_cluster-monitoring": "true",
"kubernetes.namespace_labels.pod-security_kubernetes_io_audit": "restricted",
"kubernetes.namespace_labels.pod-security_kubernetes_io_audit-version": "latest",
"kubernetes.namespace_labels.pod-security_kubernetes_io_enforce": "privileged",
"kubernetes.namespace_labels.pod-security_kubernetes_io_enforce-version": "latest",
"kubernetes.namespace_labels.pod-security_kubernetes_io_warn": "restricted",
"kubernetes.namespace_labels.pod-security_kubernetes_io_warn-version": "latest",
"kubernetes.namespace_labels.security_openshift_io_scc_podSecurityLabelSync": "true",
"kubernetes.namespace_name": "openshift-operators",
"kubernetes.pod_id": "c45ee4cd-8f53-4b9f-a479-6ed318b0eedf",
"kubernetes.pod_ip": "10.128.0.195",
"kubernetes.pod_name": "vault-secrets-operator-controller-manager-769bf97ff-mfch6",
"kubernetes.pod_owner": "ReplicaSet/vault-secrets-operator-controller-manager-769bf97ff",
"level": "error",
"log_source": "container",
"log_type": "infrastructure",
"openshift.cluster_id": "71eae85f-f749-4bbc-a825-8417db701e4b",
"openshift.labels.cluster_name": "sam",
"openshift.labels.cluster_type": "ocp-virt",
"kubernetes.annotations.alm-examples": "[\n {\n \"apiVersion\": \"secrets.hashicorp.com/v1beta1\",\n \"kind\": \"VaultConnection\",\n \"metadata\": {\n \"name\": \"vaultconnection-sample\",\n \"namespace\": \"tenant-1\"\n },\n \"spec\": {\n \"address\": \"http://vault.vault.svc.cluster.local:8200\"\n }\n },\n {\n \"apiVersion\": \"secrets.hashicorp.com/v1beta1\",\n \"kind\": \"VaultAuth\",\n \"metadata\": {\n \"name\": \"vaultauth-sample\",\n \"namespace\": \"tenant-1\"\n },\n \"spec\": {\n \"vaultConnectionRef\": \"vaultconnection-sample\",\n \"method\": \"kubernetes\",\n \"mount\": \"kubernetes\",\n \"kubernetes\": {\n \"role\": \"sample\",\n \"serviceAccount\": \"default\"\n }\n }\n },\n {\n \"apiVersion\": \"secrets.hashicorp.com/v1beta1\",\n \"kind\": \"VaultStaticSecret\",\n \"metadata\": {\n \"name\": \"vaultstaticsecret-sample\",\n \"namespace\": \"tenant-1\"\n },\n \"spec\": {\n \"vaultAuthRef\": \"vaultauth-sample\",\n \"mount\": \"kvv2\",\n \"type\": \"kv-v2\",\n \"path\": \"secret\",\n \"refreshAfter\": \"5s\",\n \"destination\": {\n \"create\": true,\n \"name\": \"secret1\"\n }\n }\n }\n]",
"kubernetes.annotations.k8s.ovn.org/pod-networks": "{\"default\":{\"ip_addresses\":[\"10.128.0.195/23\"],\"mac_address\":\"0a:58:0a:80:00:c3\",\"gateway_ips\":[\"10.128.0.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.128.0.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.128.0.1\"},{\"dest\":\"169.254.0.5/32\",\"nextHop\":\"10.128.0.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.128.0.1\"}],\"ip_address\":\"10.128.0.195/23\",\"gateway_ip\":\"10.128.0.1\",\"role\":\"primary\"}}",
"kubernetes.annotations.operatorframework.io/properties": "{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"CSISecrets\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"HCPAuth\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"HCPVaultSecretsApp\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"SecretTransformation\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"VaultAuth\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"VaultAuthGlobal\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"VaultConnection\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"VaultDynamicSecret\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"VaultPKISecret\",\"version\":\"v1beta1\"}},{\"type\":\"olm.gvk\",\"value\":{\"group\":\"secrets.hashicorp.com\",\"kind\":\"VaultStaticSecret\",\"version\":\"v1beta1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"vault-secrets-operator\",\"version\":\"1.0.1\"}}]}",
"openshift.sequence": "1761588749944941137",
"timestamp": "2025-10-27T18:12:29.894389919Z"
}