Restoring monitors and alerts from a snapshot
Introduced 3.10
OpenSearch stores alerting monitors and workflows in the .opendistro-alerting-config system index and alerts, findings, and comments in separate system indexes. When you restore these indexes from a snapshot using the Restore Snapshot API, for example, to another cluster, monitors continue to run in that cluster if you prepare the cluster first and restore the indexes as described in Restoring the alerting indexes.
This page uses original cluster for the cluster that took the snapshot and new cluster for the cluster that you restore to.
Indexes to restore
The following table lists the alerting indexes and whether to restore them.
| Index | Contents | Whether to restore |
|---|---|---|
.opendistro-alerting-config | Monitors and workflows | Yes. |
.opendistro-alerting-config-sharing | The users, roles, and backend roles that each monitor and workflow is shared with | Yes, if the original cluster used resource sharing. See Restoring monitor sharing. |
.opendistro-alerting-alerts | Active alerts | Yes, to retain active alerts. |
.opendistro-alerting-alert-history-* | Completed alerts | Yes, to retain alert history. |
.opensearch-alerting-finding-history-* | Findings | Yes, to retain findings. |
.opensearch-alerting-comments-history-* | Alert comments, if comments are enabled | Yes, to retain comments. |
.opensearch-alerting-queries* | The queries that document-level monitors run | No. OpenSearch rebuilds it from the monitors. |
.opensearch-alerting-config-lock | Temporary locks held while monitors run | No. |
Monitor actions send messages through notification channels, which OpenSearch stores in the .opensearch-notifications-config system index. If your monitors have actions, also restore that index so that the channels exist in the new cluster.
Preparing the new cluster
Restoring alerting indexes restores your monitors but not the settings and roles that control who can access them. Complete the following steps in the new cluster before you restore the indexes.
The original cluster might be unavailable when you need to restore its monitors and alerts. Record its settings and back up its security configuration as part of your recovery plan.
Copying settings from the original cluster
The following table lists the settings that control access to monitors and the retention of alert history. Configure these settings in the new cluster using the values from the original cluster.
| Setting | What it controls |
|---|---|
plugins.alerting.filter_by_backend_roles | Whether users can view only the monitors created by users who share a backend role. For more information, see Limit access by backend role. |
plugins.alerting.filter_by_backend_roles_access_strategy | How backend roles are compared when plugins.alerting.filter_by_backend_roles is true. |
plugins.security.resource_sharing.enabled | Whether resource sharing is enabled. |
plugins.security.resource_sharing.protected_types | The resource types that use resource sharing. For alerting, these are monitor and alerting-workflow. |
plugins.alerting.alert_history_retention_periodplugins.alerting.finding_history_retention_periodplugins.alerting.comments_history_retention_period | The amount of time for which history indexes are retained. |
Restored history indexes that are older than the retention period are deleted.
To view these values in the original cluster, including values set in opensearch.yml, send the following request:
GET /_cluster/settings?include_defaults=true&filter_path=*.plugins.alerting.filter_by_backend_roles*,*.plugins.alerting.*_retention_period,*.plugins.security.resource_sharing
If a setting in the new cluster doesn’t match its value in the original cluster, users’ access to monitors and alerts in the new cluster differs from their access in the original cluster and no error is reported. For example, if plugins.alerting.filter_by_backend_roles is true in the original cluster but false in the new cluster, every user with alerting permissions can view every monitor and its alerts.
Copying roles and role mappings from the original cluster
Monitors store the roles and backend roles of the user who created them and run with the permissions that those roles have in the new cluster. Copy the roles and role mappings, with the same permissions, from the original cluster to the new cluster. To do this, upload roles.yml and roles_mapping.yml from a backup of your security configuration using securityadmin.sh, or use the Security APIs. For more information about backing up your security configuration, see A word of caution. If a role grants more permissions in the new cluster, a monitor can read data that its creator couldn’t read in the original cluster.
If the original cluster used resource sharing, a restored monitor is shared with a user, role, or backend role only if a user, role, or backend role with the same name exists in the new cluster. Backend roles provided by an authentication backend, such as LDAP or SAML, require the new cluster to use the same backend.
Restoring the alerting indexes
Restore the alerting indexes to a cluster in which no monitors have been created. If the new cluster contains monitors, it already contains a .opendistro-alerting-config index. OpenSearch can’t restore an index when an open index with the same name exists, so the restore operation fails. To restore monitors and alerts, follow these steps:
- Ensure that you have permission to restore system indexes. By default, a REST admin (a user mapped to a role in
plugins.security.restapi.roles_enabled) can restore every alerting index that you need to restore, as listed in Indexes to restore. To restore notification channels, a REST admin must add.opensearch-notifications-configto theplugins.security.system_indices.restore.indicessetting. For more information, see Who can restore system indexes. -
Find the names of the history indexes in the snapshot. History index names include a date and a number, such as
.opendistro-alerting-alert-history-2026.10.27-1. To restore all of them, use a pattern such as.opendistro-alerting-alert-history-*. A REST admin can use a pattern only if it is no broader than an entry in theplugins.security.system_indices.restore.indicessetting. Otherwise, name each index. To list the indexes in the snapshot, use the Get Snapshot API:GET /_snapshot/my-repository/snapshot-1 -
Use one request to restore the alerting indexes and the data indexes that your monitors query. Name each alerting system index or match it using a pattern, as described in the previous step:
POST /_snapshot/my-repository/snapshot-1/_restore { "indices": "my-logs,.opendistro-alerting-config,.opendistro-alerting-alerts,.opendistro-alerting-alert-history-*,.opensearch-alerting-finding-history-*", "include_global_state": false }Add any of the following indexes that apply to your cluster to the same request:
.opendistro-alerting-config-sharing, if the original cluster used resource sharing.opensearch-alerting-comments-history-*, to retain alert comments.opensearch-notifications-config, if your monitors have actions
Monitors start running as soon as
.opendistro-alerting-configis restored. If you restore the other alerting indexes in a later request, the restore might fail or alerts might be lost. A monitor can’t query a data index that doesn’t exist in the new cluster. Restore the data indexes from the same snapshot as.opendistro-alerting-configso that each document-level monitor resumes from the last document that it evaluated in the original cluster, as described in Document-level monitors. -
Verify that the monitors were restored by using the Search Monitors API:
GET /_plugins/_alerting/monitors/_search { "query": { "match_all": {} } }The response lists the restored monitors.
Restoring monitor sharing
If you restore .opendistro-alerting-config without .opendistro-alerting-config-sharing, users other than a super admin lose access to the restored monitors, including the users who created them. To restore their access, restore .opendistro-alerting-config-sharing from the same snapshot. Its records refer to monitors and workflows by ID, and a restore retains the IDs. Don’t use the Migrate API for this, because it shares each monitor with its creator’s backend roles and doesn’t restore the original sharing.
Document-level monitors
A document-level monitor records the last document that it evaluated in each shard of its data index and resumes from that document after a restore. On its first run in the new cluster, the monitor might not evaluate documents that were written to its data indexes before that run. After the first run, the monitor evaluates every new document. To ensure that every document is evaluated, wait until each document-level monitor has run once in the new cluster before writing new documents to its data indexes.