Monitoring Your Elixir App with Prometheus

We previously monitored our applications with tools such as New Relic, but with the removal of the free server monitoring last year, we decided to take a look at other monitoring solutions. What we ended up with was a combination of Prometheus, Grafana, and the alert manager for Prometheus.

Prometheus and Grafana

Prometheus is a self hosted metrics platform that ingests data by scraping a web endpoint. This ends up working well in Elixir apps since we can keep common state in the whole application fairly easily. You can learn more about Prometheus at their site.

Grafana is a self hosted web application that handles the visualization for the metrics Prometheus is capturing. You set up a dashboard with graphs, charts, etc. and Grafana handles the querying for you. You can learn more about Grafana here.

Setting Up Instrumentation

Before we can get to nice graphs, we need to instrument our Elixir application. This is very simple with the prometheus.ex library. We'll also need a few other libraries in order for our metrics to be exported. Add the following to your mix.exs file:

[
  {:prometheus_ex, "~> 3.0"},
  {:prometheus_plugs, "~> 1.1"},
]

Next we set up the exporter with a new module and then adding it to your Endpoint.

defmodule Web.MetricsExporter do
  use Prometheus.PlugExporter
end

plug Web.MetricsExporter

Before this works though, we need to create a setup module that gets run during application boot. This is required because it sets up the metrics inside the prometheus elixir application structure. I like to bundle all of these setups into a single module to keep the Application module lean.

defmodule Metrics.Setup do
  def setup do
    Web.MetricsExporter.setup()
  end
end

Then call it in Application:

defmodule MyApp.Application do
  def start(_type, _args) do
    # ...

    Metrics.Setup.setup()

    # ...
    Supervisor.start_link(children, opts)
  end
end

With this you can now browse to http://localhost:4000/metrics to see what the base prometheus library gives you out the box.

Adding Custom Instrumentation

Let's create a new instrumentation. We'll keep it simple by adding a counter only. This is taken roughly from ExVenture's command instrumentation.

defmodule Metrics.CommandInstrumenter do
  use Prometheus.Metric

  def setup() do
    Counter.declare(
      name: :myapp_command_total,
      help: "Command Count",
      labels: [:command]
    )
  end
end

Add this to your Metrics.Setup module before the final exporter line to make glancing at metrics easier. The new metrics will be at the top of the file instead of the bottom this way.

def setup do
  Metrics.CommandInstrumenter.setup()
  Web.MetricsExporter.setup()
end

Finally, set up a function that increments the counter:

defmodule Metrics.CommandInstrumenter do
  def command(command_label) do
    Counter.inc(
      name: :myapp_command_total,
      labels: [command_label]
    )
  end
end

Now whenever a "command" is used in your app you can simply call this new function and prometheus will track everything for you.

Connecting Prometheus

To get Prometheus to scrape your application you need to set up a new scrape config. You can find more information available at Prometheus' documentation.

This is a simple scrape config for our application.

- job_name: 'myapp'
  scheme: 'http'
  static_configs:
    - targets: ['app.example.com:4000']

Viewing Your Metrics

The Granfana side is also pretty easy. Once you have your Grafana instance up and configured for your Prometheus instance you can create a new dashboard. Inside that dashboard make a new graph panel.

In the metrics tab while editing you can fill in your new metric to see a graph of it being called over time.

round(sum(increase(myapp_command_total[1m])) without (instance))

This will show a rough timeline of calls over 1 minute. I use without (instance) to get around prometheus scraping multiple servers and having them show up as different lines.

Conclusion

I hope this helps you get started on adding in Prometheus to your Elixir application. If you want to look over a real world application that implements all of this, I recommend looking at ExVenture, and in the lib/metrics folder.

Header photo by Mitchel Boot on Unsplash