When you run more than one server, ssh + tail -f stops scaling. Centralized logging ships every log to one searchable place. The modern, fully open-source stack is Fluent Bit → OpenSearch → OpenSearch Dashboards — ELK-style, without licensing surprises. This guide builds it on Kubernetes.

Prerequisites

  • A Kubernetes cluster and kubectl + helm.
  • A StorageClass for OpenSearch's data volumes.
  • Enough headroom: OpenSearch wants a couple of GB of RAM per node to be happy.

The pieces

  • Fluent Bit — a tiny, fast log collector/forwarder (written in C, ~1 MB RAM per pod). Tails files, parses, enriches, ships. It replaces the heavier Logstash for collection.
  • OpenSearch — the search and analytics store (the community fork of Elasticsearch).
  • OpenSearch Dashboards — the UI (the Kibana equivalent) for searching and visualizing.
app logs → Fluent Bit (collect / parse / enrich) → OpenSearch (store / index) → Dashboards (search)

Install OpenSearch

helm repo add opensearch https://opensearch-project.github.io/helm-charts
helm install opensearch opensearch/opensearch -n logging --create-namespace \
  --set replicas=3 \
  --set persistence.size=50Gi
helm install dashboards opensearch/opensearch-dashboards -n logging

Three nodes give you master quorum and shard replicas, so one node loss doesn't lose data.

Deploy Fluent Bit as a DaemonSet

One pod per node tails every container's logs. A minimal, production-shaped config:

[SERVICE]
    Flush         5
    Log_Level     info

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            cri
    Tag               kube.*
    Mem_Buf_Limit     10MB
    Skip_Long_Lines   On

[FILTER]
    Name              kubernetes        # attach pod, namespace, labels
    Match             kube.*
    Merge_Log         On

[OUTPUT]
    Name              opensearch
    Match             *
    Host              opensearch-cluster-master.logging.svc
    Port              9200
    HTTP_User         admin
    HTTP_Passwd       ${OS_PASSWORD}
    Index             logs
    Logstash_Format   On                # daily indices: logs-2025.11.03
    Suppress_Type_Name On
    tls               On

The kubernetes filter is what makes logs useful — it attaches pod, namespace, and labels so you can filter by service. Install via helm install fluent-bit fluent/fluent-bit -f values.yaml.

Parse first, search later

Structure beats grep. Parse app logs into JSON fields at collection time so you can query status:500 AND service:checkout instead of scanning text. Either log JSON from the app (best) or add a Fluent Bit parser:

[PARSER]
    Name         json
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%dT%H:%M:%S.%L

Sizing, shards & security

Two things sink self-hosted logging: getting shard sizing wrong, and leaving the cluster open.

Shards. Every index is split into shards, and every shard is a Lucene index with real overhead. The classic mistake is oversharding — thousands of tiny shards from daily indices with too many primaries — which exhausts heap and slows everything. Aim for shards in the 10–50 GB range and set the primary count accordingly (for most log volumes, 1 primary + 1 replica per daily index is plenty). The replica is what lets you lose a node without losing data, so keep at least one.

Memory. OpenSearch is a JVM app: set the heap to ≤ 50% of RAM and never above ~31 GB (beyond that the JVM loses compressed object pointers and you get less usable memory). Leave the rest of RAM for the OS filesystem cache, which is what actually makes searches fast. At scale, run 3 dedicated master-eligible nodes so cluster coordination never competes with query load.

Security — do not skip this. An open OpenSearch on port 9200 is a well-known breach vector; there are countless stories of exposed clusters wiped by bots. So:

  • Never expose 9200 publicly. Keep it internal (ClusterIP / private network); reach Dashboards through an authenticated ingress only.
  • Enable TLS for node-to-node and client traffic (the Fluent Bit tls On above).
  • Turn on authentication and RBAC — separate roles for the Fluent Bit writer (index-only) and human readers, so a leaked shipping credential can't read or delete everything.
  • Put Dashboards behind SSO if you can.

(One naming note: OpenSearch calls its lifecycle feature ISM; Elasticsearch calls the equivalent ILM. Same idea, different acronym — don't let a tutorial's wording confuse you.)

Search, dashboards & alerting on logs

Collecting logs is half the job; the value is in using them. In OpenSearch Dashboards:

  • Discover is your interactive search. Because you parsed logs into fields, you write precise queries — status:500 AND service:checkout AND NOT path:/health — instead of grepping text. Save the searches you run during incidents.
  • Visualizations & dashboards turn logs into a picture: error rate over time, top failing endpoints, slowest requests, requests by pod. A per-service dashboard is a triage superpower during an outage.
  • Alerting. OpenSearch's alerting plugin runs saved queries on a schedule and notifies Slack/email when a threshold trips — e.g. "more than 50 status:500 from checkout in 5 minutes." This turns your log store into an active monitor, not just a forensic archive.

The habit that pays off: log structured JSON from your apps with consistent field names (request_id, user_id, latency_ms). Consistency across services is what makes cross-service search and dashboards possible.

Cluster architecture: node roles at scale

A three-node cluster where every node does everything is fine for a start, but at volume you separate node roles so one workload can't starve another:

  • Master-eligible nodes (3, dedicated) handle cluster coordination — small and stable.
  • Data nodes hold shards and do the heavy indexing/search work — scale these out horizontally.
  • Coordinating nodes (optional) receive queries and fan them out, offloading data nodes.
  • Ingest nodes run pipelines if you transform on the cluster side.

Pair this with hot–warm–cold architecture: recent, heavily-searched indices live on fast-SSD hot data nodes; older indices move to cheaper warm/cold nodes (driven by the ISM policy below). This is how you keep a large logging platform both fast and affordable.

Performance tuning: bulk, refresh & mappings

A few settings dominate OpenSearch indexing performance:

  • Bulk requests. Index in batches, not one document at a time — Fluent Bit and the bulk API both do this. Small, frequent writes are the classic cause of a struggling cluster.
  • Refresh interval. By default indices refresh every second (so new logs are searchable quickly). For high-throughput logging you can raise refresh_interval to 30s to cut indexing overhead — you trade a little search latency for a lot of write throughput.
  • Mappings. Don't let OpenSearch guess types forever. Define an index template so fields get the right types, disable indexing on fields you never search, and avoid mapping high-cardinality free text as keyword where you don't need aggregations. Bad mappings waste disk and slow queries.

Beyond logs: metrics & traces

Logs answer "what happened," but the mature goal is observability — logs, metrics, and traces together. OpenSearch can store metrics and trace data too (via OpenTelemetry), or you pair it with Prometheus for metrics (see our monitoring guide) and a tracing backend. The point: logs tell you an error occurred; metrics tell you how often and how bad; traces tell you where in the request path it happened. A logging pipeline is the entry point to that fuller picture.

A worked example: from container log to searchable field

It helps to trace one log line end to end. A pod writes to stdout:

{"level":"error","service":"checkout","status":500,"latency_ms":812,"msg":"payment declined"}
  1. The container runtime writes it to /var/log/containers/checkout-xxxx.log on the node.
  2. The Fluent Bit tail input reads the line; the cri parser strips the runtime wrapper.
  3. The kubernetes filter attaches kubernetes.pod_name, namespace, and labels.
  4. A json parser (or Merge_Log On) turns the message into real fields: level, service, status, latency_ms.
  5. The opensearch output ships it to today's index logs-2025.11.03.
  6. In Dashboards you search service:checkout AND status:500 and sort by latency_ms — instantly.

Every hop is configuration, not code. Once you can picture this path, debugging the pipeline ("where did my field go?") becomes obvious — check the parser and filter stage that should have produced it.

Multiline logs, enrichment & parsing pitfalls

Two real-world issues bite every logging project:

  • Multiline logs — a Java stack trace is one logical event spread over 30 lines. Without handling, each line becomes a separate, useless log entry. Use Fluent Bit's multiline parser (built-in modes for common stack traces, or a custom rule) to stitch them into one event before shipping. This is the most common "my logs are a mess" cause.
  • Enrichment — add context the raw log lacks: environment, region, app version, a trace ID. Some comes free from the kubernetes filter; the rest you add with filters or by logging it from the app. Rich, consistent fields are what make cross-service queries powerful.

Parsing pitfalls to avoid: logging unstructured text (always prefer JSON from the app); inconsistent field names across services (user_id vs userId vs uid — pick one convention); mapping a high-cardinality field (a full URL, a UUID) as an aggregatable keyword, which explodes storage; and time zone / timestamp mismatches that scatter events across the wrong day. Decide a logging standard early and enforce it — it's far cheaper than fixing millions of malformed documents later.

OpenSearch vs the alternatives (Loki, hosted)

OpenSearch/ELK is powerful but not the only option, and knowing when it fits saves you money and ops:

  • OpenSearch (this stack) — full-text search and rich aggregations over structured logs, plus dashboards. Best when you genuinely search and analyze logs across many fields. The cost is running and sizing a stateful cluster.
  • Grafana Loki — indexes only labels (not full text), storing log bodies cheaply in object storage. Far cheaper and simpler to run, and it lives natively next to Prometheus/Grafana — great when you mostly filter by labels (pod, namespace) and grep, rather than doing complex full-text analytics.
  • Managed services — AWS OpenSearch Service, Elastic Cloud, Google Cloud Logging — remove the oper= ational burden for a per-GB fee; sensible when you'd rather not run a cluster.

A common mature setup is both: cheap, high-volume logs in Loki for everyday filtering, and OpenSearch for the subset that needs real search and analytics. Choose by the question you ask most: "filter by label and read" leans Loki; "search and aggregate across fields" leans OpenSearch.

Control cost with Index State Management (ISM)

Logs grow forever if you let them. An ISM policy rolls indices over by size/age and deletes or moves old ones — the difference between a flat logging bill and one that grows every month:

{
  "policy": {
    "description": "hot 7d, delete at 30d",
    "states": [
      {"name": "hot", "actions": [{"rollover": {"min_size": "20gb", "min_index_age": "1d"}}],
       "transitions": [{"state_name": "delete", "conditions": {"min_index_age": "30d"}}]},
      {"name": "delete", "actions": [{"delete": {}}]}
    ]
  }
}
  • Hot (fast SSD) for recent, heavily-searched logs.
  • Warm/cold for older logs you rarely touch.
  • Delete after your retention window.

Verify it worked

  1. kubectl get pods -n logging — OpenSearch (3), Dashboards, and a Fluent Bit pod per node, all Ready.
  2. In Dashboards, create an index pattern logs-* and open Discover — you should see live logs with kubernetes.pod_name, namespace, and your parsed fields.
  3. Generate an error in a test app and search status:500 — it should appear within seconds.

Troubleshooting

  • No logs in OpenSearch → check Fluent Bit's own logs (kubectl logs ds/fluent-bit) for OUTPUT errors: wrong host, TLS mismatch, or auth failure are the usual causes.
  • cluster_block_exception / read-only indices → disk watermark hit; OpenSearch locked writes. Add storage or lower retention, then clear the block.
  • Cluster status yellow/red → yellow = unassigned replicas (fine on a 1-node test, fix by adding nodes); red = missing primary shards (investigate immediately).
  • Logs unstructured / one big message field → parser not applied; enable Merge_Log and a JSON parser, or log JSON from the app.
  • Fluent Bit dropping logs under load → raise Mem_Buf_Limit and enable filesystem buffering so a slow OpenSearch doesn't lose data.

Also alert on the pipeline itself: Fluent Bit output error rate and OpenSearch cluster status.

Where to go next

Our free course builds this end to end: a resilient OpenSearch cluster, a Fluent Bit DaemonSet with parsing, dashboards for a sample app, and an ISM policy that keeps storage flat — with the lab to practice searching every service's logs in one place.