Monitoring Your Elixir App with Telemetry

This is an update to the Monitoring Your Elixir App with Prometheus post. Since then, Telemetry has come out as a common way of tracking metrics in Elixir applications.

For this post, we're just going to go over the new Telemetry changes. See the older post for how to setup Prometheus.

Setting up Telemetry

We'll be going through the new Erlang version of Telemetry, which was transitioned from Elixir at version 0.3.

  {:telemetry, "~> 0.3"},

Once the package is installed, we need to update our instrumenter modules. Let's update the custom instrumenter from the last post, starting with the setup.

defmodule Metrics.CommandInstrumenter do  
  use Prometheus.Metric

  def setup() do
    Counter.declare(
      name: :myapp_command_total,
      help: "Command Count",
      labels: [:command]
    )

    events = [
      [:myapp, :command],
    ]

    :telemetry.attach_many("myapp-commands", events, &handle_event/4, nil)
  end
end

Telemetry uses an attach process to hook into events. With this we can attach to many events all at once using the same handle function. The last argument is any config you might wish to push to handle_event/4 when that gets called.

Calling Telemetry

To trigger a new Telemetry event, you execute the event with a count and metadata about the event. Update any previous calls directly to instrumenters to a telemetry call instead.

:telemetry.execute([:myapp, :command], 1, %{command: "loaded"})

This executes in the local process, and will call the function passes in directly. If anything fails with the function it is removed and will no longer be called. This way your metrics don't take down your application.

Handling Metrics

With metrics updated in your codebase, you can update the instrumenter handle_event/4 function. Inside this function you can do any logging and triggering Prometheus metrics. This keeps it all in a common spot, and out of the rest of your codebase scattered around.

defmodule Metrics.CommandInstrumenter do  
  def handle_event([:myapp, :command], _count, _metadata = %{command: command}, _config) do
    Counter.inc(name: :myapp_command_total, labels: [command])
  end
end  

Conclusion

The upgrade to Telemetry is pretty easy, and I have liked the change a lot. It's really nice to have a single pipeline for metrics and have whatever cares about those metrics work on them.

Another telemetry package you might be interested in is the poller, which is a GenServer that acts as a cron server to periodically trigger metrics. I was able to replace a custom GenServer in Gossip with this package.

If you wish to see telemetry in action in a real application, make sure to check out Gossip or ExVenture!

Header photo by Rachmat widyananda on Unsplash