How To Set Up Alerts Using PMM

Percona Monitoring and Management allows you to set different alerts related to your infrastructure. Before we start setting up alerting, lets make sure this option is enabled on your PMM Server. To do it go to Configuration —> PMM Settings —> Advanced Settings —> Alerting

PMM Settings

Advanced Settings Alerting

PMM presents several types of alerts, depending on used datasources and complexity of needed alerts: Percona templated alerts, Grafana managed alerts, Mimir or Loki alerts, Mimir or Loki recording rules. As far as PMM Server gives you the most necessary alert templates out of the box, we will use them to set up our alerting system. And that is a good point to start. You may create rule from a template to fire alert about node high memory or cpu utilization, availability of your databases, and some most common parameters of different databases usage. To see necessary template go to Alerting —> Alert rule templates section

Alert rule templates

and chose one you need

alert templates

After that edit the name of alert rule, set up such parameters as threshold, alerting level, event duration and select a folder where your rule will be stored

Alert Rule Settings

If you need more complex or just different metrics alert, you may create your own rule template. Templates should be written in YAML format. Here is an example of such file

templates:
 - name: pmm_mongodb_high_memory_usage
   version: 1
   summary: Memory used by MongoDB
   expr: |-
      sum by (node_name) (mongodb_ss_mem_resident * 1024 * 1024)
      / on (node_name) (node_memory_MemTotal_bytes)
       * 100
       > [[ .threshold ]]
   params:
     - name: threshold
       summary: A percentage from configured maximum
       unit: "%"
       type: float
       range: [0, 100]
       value: 80
   for: 5m
   severity: warning
   labels:
      custom_label: demo
   annotations:
      summary: MongoDB high memory usage ({{ $labels.service_name }})
      description: |-
         {{ $value }}% of memory (more than [[ .threshold ]]%) is used
         by {{ $labels.service_name }} on {{ $labels.node_name }}

It should contain the next options:

• name (required field): uniquely identifies template. Spaces and special characters are not allowed.

• version (required): defines the template format version.

• summary (required field): a template description.

• expr (required field): a MetricsQL query string with parameter placeholders.

• value (optional): default parameter value. Value strings must not include any of these special characters: < > ! @ # $ % ^ & * ( ) _ / \ ' + – = (space).

• for (required): specifies the duration of time that the expression must be met before the alert will be fired.

• severity (required): specifies default alert severity level.

• labels (optional): are additional labels to be added to generated alerts.

• annotations (optional): are additional annotations to be added to generated alerts.

After alert rules are created from build on or custom template, setup contact points. You may chose one or more notification channels like Slack, Email, Telegram and others.

Contact Points

You may also silence some of alerts on some timetable to avoid spamming your communication channels. There is special section maned Silences in Alerting menu section

Silences

You may setup much more complex alerts, using different data sources with Grafana managed alerts. But this is out of this article topic. Percona templated alerts will cover the majority of cases. So PMM server can help you not only to watch you databases server but also will notify you if something is going wrong.

---

Need help with database monitoring and performance? Our DevOps consulting team sets up monitoring for MySQL, PostgreSQL and MongoDB. The first conversation is free.

Frequently Asked Questions

How do I set up alerts in Percona PMM?

Enable alerting in PMM under Configuration → PMM Settings → Advanced Settings → Alerting. Then go to Alerting → Alert rule templates, choose a built-in template (e.g., high CPU, memory, or database availability), configure threshold, severity, and event duration, and save to a folder. PMM ships with the most common alert templates out of the box.

What notification channels does PMM alerting support?

PMM supports Slack, Email, Telegram, PagerDuty, and other channels via contact points. After creating alert rules, configure contact points in the Alerting section to route notifications to one or more channels. You can assign different contact points to different alert folders to control who gets notified for which alert.

How do I write a custom PMM alert template?

Custom PMM alert templates are written in YAML. Each template requires name, version, summary, expr (a MetricsQL query with [[ .threshold ]] placeholders), for (minimum duration before firing), and severity. Optional fields include params, labels, and annotations. Upload templates via the Alerting → Alert rule templates interface.

How do I silence PMM alerts during maintenance?

Use the Silences section in the PMM Alerting menu. Create a silence rule matching the affected service or node labels and set the start/end time for the maintenance window. During the silence period, PMM suppresses matching alerts and does not send notifications to configured contact points.

Related Reading

• DB Performance Troubleshooting Workflow Using Percona PMM
• How To Deploy PMM Using AWS
• How To Install PMM Using Linode