System indexes

System indexes store state that is managed by OpenSearch or an OpenSearch plugin, such as security configuration, asynchronous task results, and plugin metadata. A system index usually starts with a period (.), but a period-prefixed index is not necessarily a system index. OpenSearch identifies system indexes from descriptors registered by OpenSearch and its plugins.

System indexes differ from regular indexes in the following ways:

• OpenSearch permits a registered system index to be automatically created even when action.auto_create_index is false or its pattern would otherwise exclude the index. Registering a system index does not create it immediately. The owning component typically creates it when the component first stores data.
• Operations that target only system indexes use the system_read and system_write thread pools for supported read and write paths. This isolates internal component work from regular search and write traffic. A request that mixes system and regular indexes might use a regular thread pool.
• System indexes are intended to be accessed through their owning component’s API. When the Security plugin is enabled, it adds the access controls described in Security plugin protection.

Do not modify a system index directly. Use the API provided by the component that owns the index. Direct changes can corrupt component state or become incompatible after an upgrade.

System index patterns in the standard distribution

The following table lists the descriptors registered at startup by the components in the standard OpenSearch distribution. The owning component usually creates an index only when its feature first stores data, so an index does not need to exist to be registered as a system index. The exact set can change between OpenSearch versions and can also depend on plugin settings.

An index created or managed by a plugin is not necessarily a system index. Some plugin-created indexes are regular indexes because their contents are intended to be searched by users. For example, the Forecasting plugin result indexes are regular indexes even though the plugin’s configuration, checkpoint, and state indexes are system indexes. Similarly, when the Security plugin audit logs are stored in OpenSearch, their index or data stream is user searchable rather than registered as a system index.

Security plugin protection

The Security plugin always protects its configuration index, .opendistro_security, by default. When system index protection is enabled, the plugin also protects indexes registered with OpenSearch and any legacy patterns configured in plugins.security.system_indices.indices.

The demo security configuration enables system index protection:

plugins.security.system_indices.enabled: true

Write protection

Regular index permissions, including broad permissions for *, do not grant write access to a protected system index. Writes are permitted only in one of the following contexts:

• The plugin that registered the system index performs the operation using its plugin identity.
• A super admin authenticates using an admin certificate.
• System index permissions are enabled and a role explicitly grants system:admin/system_index for the index pattern. This option does not grant access to the Security plugin configuration index.

Use the owning plugin’s API whenever possible, including when you authenticate as a super admin. For example, use the Security REST API or securityadmin.sh to change Security plugin configuration instead of indexing documents directly into .opendistro_security.

Read protection and scrubbed results

Read protection does not produce the same response for every API. For a user without system index access, the Security plugin can replace the underlying index reader with an empty reader. As a result:

• A search can return 200 OK with zero hits even though matching documents exist.
• A get request can behave as though the document does not exist.
• An operation that cannot be safely filtered can return 403 Forbidden.

Do not interpret a successful response or an empty result as proof that the caller has access to a system index. When plugins.security.system_indices.permission.enabled is enabled, an explicit request from a user without the required system index permission is generally rejected instead of returning scrubbed search results.

The .tasks* family is an exception. The Security plugin permits reads so that authorized users can use the task APIs and read stored task results. OpenSearch still protects writes to the index.

To read the Security configuration index directly, authenticate with an admin certificate:

curl -k --cert ./kirk.pem --key ./kirk-key.pem \
  -X GET 'https://localhost:9200/.opendistro_security/_search'

Audit logging

When Security audit logging is enabled, rejected attempts to access the Security plugin configuration index or another protected system index are recorded in the OPENDISTRO_SECURITY_INDEX_ATTEMPT audit category. This check is separate from ordinary index permissions and document-level or field-level security, so granting broad index permissions does not bypass the protection or its audit event. For information about audit configuration and excluded categories, see Audit logs.

Configuring additional system indexes

The plugins.security.system_indices.indices setting can protect additional index patterns, but it is deprecated. Existing deployments can continue to use it while migrating plugin-owned indexes to system index descriptors. Because it is a static node setting, every node must use the same value and must be restarted after a change:

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices:
  - ".example-plugin-*"

For information about granting users explicit access, see System index permissions.

System indexes for plugin developers

A plugin that owns internal indexes should register them with OpenSearch rather than requiring an administrator to add patterns to plugins.security.system_indices.indices. To register a plugin’s system indexes with OpenSearch, follow these steps:

1. Implement SystemIndexPlugin and return a SystemIndexDescriptor for each index pattern. Patterns must start with . and must not overlap descriptors registered by another plugin.
2. Expose plugin-specific actions and REST APIs for accessing the data instead of requiring callers to use the standard index APIs.
3. Implement IdentityAwarePlugin and retain the assigned PluginSubject.
4. Run internal client operations as the assigned subject. A FilterClient wrapper can apply the PluginSubject consistently and restore the caller’s thread context before invoking an asynchronous listener.

Calling ThreadContext.stashContext() by itself is not sufficient when the Security plugin is installed: stashing removes the caller’s context but does not establish the plugin identity that authorizes access to the plugin’s registered system indexes.

For an implementation example, see the Security plugin’s sample PluginClient and SampleResourcePlugin.