VictoriaLogs single server k8s setup gotchas
TL;DR
Quick notes from our production rollout of the VictoriaLogs single‑server Helm chart:
- Use a dedicated node (preferably NVMe) for the VictoriaLogs server to avoid noisy‑neighbor interference.
- Tune retention and storage to match your log volume; VictoriaLogs compresses well but storage still has a cost.
- If you enable Vector as a log collector, limit metadata enrichment to reduce cardinality and resource usage.
- Enable dashboards and collect Vector metrics via VMPodScrape for operational visibility.
- Run a dedicated VMAlert instance for VictoriaLogs and scope rules with selectors.
- Follow LogsQL best practices (reduce label cardinality, avoid expensive regexes, limit time ranges).
Use case & scope
We recently deployed the VictoriaLogs single‑server Helm chart to our production clusters. In this post, I’ll share some gotchas and best practices we encountered during the setup.
Before diving into the setup, a quick shout‑out to Grafana Loki — a solid alternative. We started with Loki but found VictoriaLogs easier to set up and maintain for our needs, while still being cost effective.
Note: our production setup is optimized for a relatively small cluster (fewer than 80 nodes) with a focus on cost efficiency. Some recommendations below may not apply to much larger clusters.
Use a dedicated node for the server
Although this may run counter to strict cost‑saving goals, I strongly recommend allocating a dedicated node for the VictoriaLogs server. A single‑server deployment handles both storage and query workloads for the entire cluster; isolating it (and provisioning sufficient CPU, memory, and fast disks) prevents noisy‑neighbor interference. Prefer NVMe‑backed storage to achieve the high I/O throughput and low latency this workload requires. If you are using Azure consider an NVMe supported VM size for your dedicated nodepool. Remember that the performance and availability of the logging stack are critical for reliable monitoring and troubleshooting.
Adjust retention period and storage
By default, the victorialogs helm chart sets a retention period of 1 month for the logs collected by VictoriaLogs together with a storage request of 10Gi. Depending on your log volume and retention requirements, you may want to adjust these values to fit your needs. Another useful option to look and adjust is retention by disk space usage which can be set using absolute or (mutually exclusive) percentage values.
Even though VictoriaLogs usually compresses logs by 10x or more times evaluate the use case and usefulness of the logs you are collecting, as well as the cost of storage. Below is an example of how to modify the retention period and storage request in your values.yaml file:
1
2
3
4
5
6
7
8
---
server:
retentionPeriod: 7d
retentionDiskSpaceUsage: 19Gi
vmServiceScrape:
enabled: true
persistentVolume:
size: 20Gi
If you enable Vector, limit enrichment and tune resources
If you decide to enable Vector Kubernetes logs, be aware that by default it will enrich the logs with Kubernetes metadata (pod labels, annotations, namespace, etc.) which can lead to high cardinality (which in case of structured logs it is handled pretty good by VictoriaLogs provided you follow a couple of best practices) and increased resource usage. Depending on your use case, you may want to limit the metadata enrichment to only the necessary fields. You can do this by modifying the values.yaml file:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
vector:
enabled: true
customConfig:
sources:
k8s:
insert_namespace_fields: false
oldest_first: false
pod_annotation_fields:
container_id: ""
container_image_id: ""
pod_uid: ""
pod_annotations: ""
pod_labels: ""
extra_label_selector: "component!=ama-logs-agent,rsName!=ama-logs-rs,app!=konnectivity-agent"
node_annotation_fields:
node_labels: ""
namespace_annotation_fields:
namespace_labels: ""
Enable dashboards
The ROI (return on investment) is high: Vector’s dashboards make it easy to spot noisy pods at a glance, so you can exclude them from collection or prioritize investigation. The default dashboards are comprehensive and provide immediate operational value.
Enable dashboards via values.yaml:
1
2
3
4
5
dashboards:
enabled: true
labels:
grafana_dashboard: '1'
namespace: victoriametrics
To collect Vector metrics, create a VMPodScrape resource and send metrics to VictoriaMetrics:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: vector
namespace: victoriametrics
labels:
app.kubernetes.io/part-of: victorialogs
spec:
jobLabel: vector
namespaceSelector:
matchNames:
- victorialogs
selector:
matchLabels:
app.kubernetes.io/name: vector
podMetricsEndpoints:
- portNumber: 9090
interval: "10s"
honorLabels: true
path: /metrics
Deploy a dedicated VMAlert
If you are using the victoriametrics-k8s-stack chart, you will notice that it comes with a “default” VMAlert instance. However, as stated in the official documentation, it is recommended to deploy a dedicated VMAlert instance for VictoriaLogs. The main gotcha here is to use selectors for both VMalert resources so that each instance only picks up the relevant alerting rules. Below is an example of how to deploy a dedicated VMAlert instance for VictoriaLogs:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
labels:
app.kubernetes.io/component: alerting
app.kubernetes.io/part-of: victorialogs
name: vmetrics-vlogs
namespace: victoriametrics
spec:
datasource:
url: http://vlogs-victoria-logs-single-server.victorialogs.svc.cluster.local.:9428
evaluationInterval: 20s
extraArgs:
http.pathPrefix: /
remoteWrite.disablePathAppend: "true"
notifiers:
- url: http://vmalertmanager-vmetrics-victoria-metrics-k8s-stack-0.vmalertmanager-vmetrics-victoria-metrics-k8s-stack.victoriametrics.svc.cluster.local.:9093
port: "8080"
remoteRead:
url: http://vmsingle-vmetrics-victoria-metrics-k8s-stack.victoriametrics.svc.cluster.local.:8428
remoteWrite:
url: http://vmsingle-vmetrics-victoria-metrics-k8s-stack.victoriametrics.svc.cluster.local.:8428/api/v1/write
ruleSelector:
matchLabels:
app.kubernetes.io/part-of: victorialogs
ruleNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: victorialogs
selectAllByDefault: false
Note that in the example above, we are using ruleSelector, ruleNamespaceSelector and setting selectAllByDefault to false to ensure that the VMAlert instance only picks up the alerting rules that are relevant to VictoriaLogs. Likewise you will need to modify the ruleSelector and ruleNamespaceSelector of the default VMAlert instance to ensure that it only picks up the alerting rules that are relevant to VictoriaMetrics. This can be done by modifying the values.yaml file of the victoriametrics-k8s-stack chart:
1
2
3
4
5
6
vmalert:
spec:
ruleNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: victoriametrics
selectAllByDefault: false
Use LogsQL best practices
The official documentation includes a comprehensive LogsQL guide that’s worth reading to optimize your queries and dashboards. I recommend reviewing the LogsQL overview and applying the recommended best practices and performance tips to lower resource usage and improve query latency.
Conclusion
We have covered some of the gotchas and best practices when setting up the VictoriaLogs single server in Kubernetes. The server is easy to set up and maintain, and it provides a good overview of the cluster’s logs while being cost-effective. However, it is important to keep in mind the resource requirements and to configure the chart according to your needs. Using a dedicated node for the server, adjusting retention period and storage, enabling dashboards, deploying a dedicated VMAlert instance for VictoriaLogs, and following LogsQL best practices are some of the key considerations to keep in mind when setting up VictoriaLogs single server.