How To Set Up Alerts Using PMM

Percona Monitoring and Management allows you to set different alerts related to your infrastructure. Before we start setting up alerting, lets make sure this option is enabled on your PMM Server. To do it go to Configuration —> PMM Settings —> Advanced Settings —> Alerting

PMM Settings

Advanced Settings Alerting

PMM presents several types of alerts, depending on used datasources and complexity of needed alerts: Percona templated alerts, Grafana managed alerts, Mimir or Loki alerts, Mimir or Loki recording rules. As far as PMM Server gives you the most necessary alert templates out of the box, we will use them to set up our alerting system. And that is a good point to start. You may create rule from a template to fire alert about node high memory or cpu utilization, availability of your databases, and some most common parameters of different databases usage. To see necessary template go to Alerting —> Alert rule templates section

Alert rule templates

and chose one you need

alert templates

After that edit the name of alert rule, set up such parameters as threshold, alerting level, event duration and select a folder where your rule will be stored

Alert Rule Settings

If you need more complex or just different metrics alert, you may create your own rule template. Templates should be written in YAML format. Here is an example of such file

templates:
 - name: pmm_mongodb_high_memory_usage
   version: 1
   summary: Memory used by MongoDB
   expr: |-
      sum by (node_name) (mongodb_ss_mem_resident * 1024 * 1024)
      / on (node_name) (node_memory_MemTotal_bytes)
       * 100
       > [[ .threshold ]]
   params:
     - name: threshold
       summary: A percentage from configured maximum
       unit: "%"
       type: float
       range: [0, 100]
       value: 80
   for: 5m
   severity: warning
   labels:
      custom_label: demo
   annotations:
      summary: MongoDB high memory usage ({{ $labels.service_name }})
      description: |-
         {{ $value }}% of memory (more than [[ .threshold ]]%) is used
         by {{ $labels.service_name }} on {{ $labels.node_name }}

It should contain the next options:

name (required field): uniquely identifies template. Spaces and special characters are not allowed.
version (required): defines the template format version.
summary (required field): a template description.
expr (required field): a MetricsQL query string with parameter placeholders.
value (optional): default parameter value. Value strings must not include any of these special characters: < > ! @ # $ % ^ & * ( ) _ / \ ‘ + – = (space).
for (required): specifies the duration of time that the expression must be met before the alert will be fired.
severity (required): specifies default alert severity level.
labels (optional): are additional labels to be added to generated alerts.
annotations (optional): are additional annotations to be added to generated alerts.

After alert rules are created from build on or custom template, setup contact points. You may chose one or more notification channels like Slack, Email, Telegram and others.

Contact Points

You may also silence some of alerts on some timetable to avoid spamming your communication channels. There is special section maned Silences in Alerting menu section

Silences

You may setup much more complex alerts, using different data sources with Grafana managed alerts. But this is out of this article topic. Percona templated alerts will cover the majority of cases. So PMM server can help you not only to watch you databases server but also will notify you if something is going wrong.