Jump to content
From ArchWiki

Related articles

- Zabbix
- Munin
- Grafana
- TICK stack

Prometheus is an open-source metrics collection and processing tool. It consists primarily of a timeseries database and a query language to access and process the metrics it stores. Separate services perform metric exposure, from which the Prometheus server can pull. It provides a very minimal web UI out of the box. To get a functional dashboard system, third-party tools like Grafana can be used.

## Installation

Install the prometheus package. After that you can enable and start `prometheus.service` and access the application via HTTP on port 9090 by default.

The default configuration monitors the `prometheus` process itself, but not much beyond that. To perform system monitoring, you can install prometheus-node-exporter which performs metric scraping from the local system. You can start and enable the `prometheus-node-exporter` service. It will open port 9100 by default. Once the service is running, you will need to configure `prometheus` to scrape the exporter service periodically in order actually to collect the data. Do this by following the steps to add metrics as shown below.

Warning: The default configuration of `prometheus` listens on `*:9090` and `prometheus-node-exporter` listens on `*:9100`, so make sure to change the configuration or enable the relevant firewall rules. See also the Prometheus security model.

## Configuration

The Prometheus configuration is done through YAML files, the main one being located at `/etc/prometheus/prometheus.yml`.

### Adding metrics

You can add new places to scrape metrics from by adding them to the `scrape_configs` array. To add the local node exporter as a source, next to the prometheus process itself, the configuration would look like this:

     scrape_configs:
       - job_name: 'prometheus'
         static_configs:
           - targets: ['localhost:9090']
       - job_name: 'node'
         static_configs:
           - targets: ['localhost:9100']

## Exporters

The Arch Linux repository contains a subset of the available exporters:

- prometheus-node-exporter - system metrics
- prometheus-blackbox-exporter - blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP
- prometheus-memcached-exporter - memcached metrics
- prometheus-mysqld-exporter – MySQL server metrics

The exporters are implemented as services. For example to run the node exporter, enable and start `prometheus-node-exporter.service`.

## Using the UI

Prometheus comes with a very limited web UI to verify configuration, query and graph metrics. You can reach it at http://localhost:9090 by default. You can find an in-depth explanation of Prometheus' query language in the Prometheus documentation.

## Alerting

alertmanager can send out custom alerts when certain conditions are met configured in `/etc/prometheus/alert.rules.yml` and what alert to send out is configured in `/etc/alertmanager/alertmanager.yml`. Alertmanager supports various ways to notify users such as email, slack, and more. To configure email alerts add the following snippet:

    global:
      resolve_timeout: 5m
      smtp_smarthost: 'smtp.example.com:25'
      smtp_from: 'alertmanager@example.com'
    route:
      group_by: ['instance', 'severity']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 3h
      receiver: team-1
    receivers:
      - name: 'team-1'
        email_configs:
          - to: 'admin@example.com'

For prometheus to send alerts to alertmanager include the following snippet in `/etc/prometheus/prometheus.yml`:

    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          - localhost:9093

To configure an alert for when a systemd unit fails add the following snippet to `/etc/prometheus/alert.rules.yml`. For more rules read the alerting rules documentation.

    - name: systemd_unit
      interval: 15s
      rules:
      - alert: systemd_unit_failed
        expr: |
          node_systemd_unit_state{state="failed"} > 0
        for: 3m
        labels:
          severity: critical
        annotations:
          description: 'Instance : Service failed'
          summary: 'Systemd unit failed'

## Tips and tricks

### Telegraf instead of exporters

Telegraf can be used instead of multiple exporters when used with Prometheus Output Plugin. This reduces metrics collection into a single binary and offers more flexible configuration when compared to standard Prometheus exporters.

## See also

- Official homepage

Categories:

- Network monitoring
- Web applications