Opentelemetry: Escape Hatch From the Observability Cartel

Making sense out of so much data is why datadog and sentry make so much money.

These are issues you'd experience with anything that spans your stack as a custom telemetry library would.

It needs to be treated as an integral part of whatever framework is being instrumented. And maintained by those same people.

I don't think it's fair to say "no one wants to invest in proper instrumentation" - the OpenTelemetry community has built a massive amount of instrumentation in a relatively short period of time. Yes, OpenTelemetry is still young and unstable, but it's getting better every day.

As the article notes, the OpenTelemetry Collector has plugins can convert nearly any telemetry format to OTLP and back. Many of the plugins are "official" and maintained by employees of Splunk, Datadog, Snowflake, etc. Not only does it break the lock-in, but it allows you to reuse all the great instrumentation that's been built up over the years.

> The core issue is that, with otel, observability platforms become just a UI layer over a database.

I think this is a good thing - when everyone is on the same playing field (I can use Datadog instrumentation, convert it to OTel, then export it to Grafana Cloud/Prometheus), vendors will have to compete on performance and UX instead of their ability to lock us in with "golden handcuffs" instrumentation libraries.

Of course you could also roll your own telemtry, which is generally no that difficult in a lot of frameworks. You don't always need something like OTEL.

They have a SQL like query language that I think can do most of what you're describing

i guess it depends on what you're used to

[0] https://www.ibm.com/products/instana/opentelemetry

[1] https://github.com/instana/instana-otel-collector

[2] https://play-with.instana.io/#/home

[disclaimer: I'm an IBMer]

It will always be overkill for just an app or two talking with eachother... till you grow, then it won't be overkill any more.

But still might be worth getting into it on smaller apps just thanks to wealth of tools available.

I haven't tried but it's probably possible to do the same with JS

I think Sentry was also expanding into tracing--might be worth a look to see if they're doing something that works better in their library

That said if your goal is basic performance metrics and nothing more, then tracing is overkill. Don’t even need an SDK, just monitor the compute nodes async with some basic system metrics. But if your goal is to narrow down behaviors within your app on a per-request basis there really is no way around tracing if you value your sanity.

The collector is a helm chart, someone on my project added it to our K8s clusters last week. It was like 30 lines of YAML/Terraform in total. Logs, trace forwarding, Prometheus scraping. That bit is easy.

Idk about deploying the ui/storage. I’ve used Grafana Loki stuff in Docker Compose locally without much head scratching for local development.

https://github.com/grafana/docker-otel-lgtm

Native k8s logs + Prometheus is probably more on the lighter weight side but you don't get traces. You could find some middle ground using the otel collector to extract trace metrics so you get RED metrics but you wouldn't have full traces

compared to dumping logs to a file (or a single instance Prometheus scraping /metrics) everything is frustrating, because there are so many moving parts anyway, you want to query stuff and correlate, but for that you need to propagate the trace id, and emit and store spans, and take care to properly handle async workers, and you want historical comparisons, and downsampling for retention, and of course auto-discovery/registration/labeling from k8s pods (or containers or whatever) and source code upload and release tagging from CI, and ...