You're viewing version 3.8 of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.

Audit logs

Audit logs let you track access to your OpenSearch cluster and are useful for compliance purposes or in the aftermath of a security breach. You can configure the categories to be logged, the detail level of the logged messages, and where to store the logs.

OpenSearch supports two audit logging modes. The following table describes each mode and how it is configured.

For more information about standalone mode, see Standalone audit logging.

Enabling audit logging (standard mode)

Audit logging is disabled by default. To enable audit logging:

1. Add the following line to opensearch.yml on each node:

   plugins.security.audit.type: internal_opensearch
   

   This setting stores audit logs on the current cluster. For other storage options, see Audit Log Storage Types.

2. Restart each node.

Audit logging requires two settings: a storage type (plugins.security.audit.type) in opensearch.yml and config.enabled: true in audit.yml. The audit.yml file provided by the Security plugin sets config.enabled to true by default, so audit logging can appear to be enabled in a new cluster. Until you specify a storage type, the Security plugin cannot create a storage endpoint for the audit log, so it does not record any events and logs a warning at startup that no default storage is available.

After this initial setup, you can use OpenSearch Dashboards to manage your audit log categories and other settings. In OpenSearch Dashboards, select Security and then Audit logs. Alternatively, you can specify settings for audit logging in the audit.yml and opensearch.yml files (which file depends on the setting—see Audit log settings). You can also use the Audit log APIs to manage and update settings.

Tracked events

Audit logging records events in two ways: HTTP requests (REST) and the transport layer. The following table provides descriptions of tracked events and whether or not they are logged on the REST or transport layer.

REQUEST_AUDIT originates from a REST request (audit_request_origin: REST), but the event itself is recorded with audit_request_layer: TRANSPORT. As a result, it can be suppressed by any of disabled_categories, disabled_transport_categories, or disabled_rest_categories—adding REQUEST_AUDIT to any one of these settings suppresses the event.

Audit log settings

The following default log settings work well for most use cases. However, you can change settings to save storage space or adapt the information to your exact needs.

Settings in audit.yml

The following settings are stored in the audit.yml file.

Exclude categories

To exclude categories, list them in the following setting:

config:
  audit:
    disabled_rest_categories: <disabled categories>
    disabled_transport_categories: <disabled categories>

For example:

config:
  audit:
    disabled_rest_categories:
      - AUTHENTICATED
      - GRANTED_PRIVILEGES
    disabled_transport_categories: [ GRANTED_PRIVILEGES ]

Alternatively, you can use the unified disabled_categories setting to disable categories on both layers simultaneously:

config:
  audit:
    disabled_categories:
      - AUTHENTICATED
      - GRANTED_PRIVILEGES

When disabled_categories is configured alongside disabled_rest_categories or disabled_transport_categories, a category is disabled on a given layer if it appears in either the unified setting or the layer-specific setting.

A deprecation warning is logged when disabled_categories is configured alongside layer-specific settings, encouraging migration to disabled_categories only.

For example, the following configuration disables AUTHENTICATED on both layers (using disabled_categories) and disables SSL_EXCEPTION on the REST layer only:

config:
  audit:
    disabled_categories:
      - AUTHENTICATED
    disabled_rest_categories:
      - SSL_EXCEPTION

By default, the CLUSTER_SETTINGS_CHANGED and INDEX_SETTINGS_CHANGED categories are disabled on the transport layer. To enable them, remove them from disabled_transport_categories:

config:
  audit:
    disabled_transport_categories:
      - AUTHENTICATED
      - GRANTED_PRIVILEGES

If you want to log events in all categories, use NONE:

config:
  audit:
    disabled_rest_categories: NONE
    disabled_transport_categories: NONE

Disable REST or the transport layer

By default, the Security plugin logs events on both REST and the transport layer. You can disable either type:

config:
  audit:
    enable_rest: false
    enable_transport: false

Disable request body logging

By default, the Security plugin includes the body of the request (if available) for both REST and the transport layer. If you do not want or need the request body, you can disable it:

config:
  audit:
    log_request_body: false

Log index names

By default, the Security plugin logs all indexes affected by a request. Because index names can be aliases and contain wildcards/date patterns, the Security plugin logs the index name that the user submitted and the actual index name to which it resolves.

For example, if you use an alias or a wildcard, the audit event might look like:

audit_trace_indices: [
  "human*"
],
audit_trace_resolved_indices: [
  "humanresources"
]

You can disable this feature by setting:

config:
  audit:
    resolve_indices: false

This feature is only disabled if config.audit.log_request_body is also set to false.

Configure bulk request handling

Bulk requests can contain many indexing operations. By default, the Security plugin only logs the single bulk request, not each individual operation.

The Security plugin can be configured to log each indexing operation as a separate event:

config:
  audit:
    resolve_bulk_requests: true

This change can create an extremely large number of events in the audit logs, so we don’t recommend enabling this setting if you frequently use the _bulk API.

Exclude requests

To exclude certain requests from being logged, configure actions for transport requests, HTTP request paths (REST), or both:

config:
  audit:
    ignore_requests: ["indices:data/read/*", "SearchRequest"]

Exclude users

By default, the Security plugin logs events from all users but excludes the internal OpenSearch Dashboards server user kibanaserver. You can exclude other users:

config:
  audit:
    ignore_users:
      - kibanaserver
      - admin

If requests from all users should be logged, use NONE:

config:
  audit:
    ignore_users: NONE

Exclude headers

You can exclude sensitive headers from being included in the logs—for example, the Authorization: header:

config:
  audit:
    exclude_sensitive_headers: true

Settings in opensearch.yml

The following settings are stored in the opensearch.yml file. How they are applied at runtime depends on the audit logging mode:

• In standalone mode (SSL-only or security-disabled), most audit filter and compliance settings can be changed at runtime using the Cluster Settings API without a node restart. These settings include log_request_body, resolve_indices, disabled_categories, enable_rest, enable_transport, ignore_users, and ignore_requests.

• In standard mode with FGAC, the audit configuration is stored in audit.yml, and the corresponding plugins.security.audit.config.* and plugins.security.audit.compliance.* cluster settings do not update it. A PUT _cluster/settings request can succeed without changing the audit configuration. The body_logging_exclusions setting is the exception: in FGAC mode, you can update it using the Cluster Settings API.

Some audit settings are static and require a node restart: action_groups.<NAME>, log4j.enable_mdc_routing, config.index, the thread pool settings, and the sink connection settings.

Updating a dynamic audit setting requires a role listed in plugins.security.restapi.roles_enabled when FGAC is enabled. This restriction is not enforced in SSL-only or security-disabled mode. The body_logging_exclusions setting can be updated without the elevated role.

The following table describes the audit settings that a caller can read using GET _cluster/settings in each mode.

The plugins.security.audit.enabled runtime toggle applies only to standalone audit logging. See Standalone audit logging for instructions.

Exclude categories

You can configure disabled categories in opensearch.yml using the plugins.security.audit.config prefix. This is useful for non-FGAC modes (SSL-only or security-disabled), in which the audit.yml security index is not available:

plugins.security.audit.config.disabled_categories:
  - AUTHENTICATED
  - GRANTED_PRIVILEGES

The layer-specific settings disabled_rest_categories and disabled_transport_categories are deprecated. Use the unified disabled_categories setting instead.

The layer-specific settings are also available:

plugins.security.audit.config.disabled_rest_categories:
  - AUTHENTICATED
  - GRANTED_PRIVILEGES
plugins.security.audit.config.disabled_transport_categories:
  - AUTHENTICATED
  - GRANTED_PRIVILEGES

When both disabled_categories and the layer-specific settings are configured, a category is disabled on a given layer if it appears in either setting. A deprecation warning is also logged, recommending that you use disabled_categories only.

The disabled_categories settings suppress only REST and transport categories. They do not affect COMPLIANCE_* categories, which are governed solely by the compliance settings (compliance.enabled, the watched indexes and fields, and the compliance ignore-users settings).

Body logging exclusions

Use body_logging_exclusions to suppress request bodies for specific actions or REST paths while continuing to log bodies for all other requests. The log_request_body setting applies to every request, so it cannot exclude high-volume operations such as bulk ingestion selectively.

All expanded action-group patterns and raw exclusion patterns form a single combined wildcard matcher. For each audit layer, this matcher is tested against the corresponding identifier described in the following table. When a match is found, the request body field is omitted from the audit event. All other fields, such as the user, IP address, indexes, and timestamp, are preserved.

Both identifiers support wildcard patterns using * (for example, indices:data/write/bulk* matches indices:data/write/bulk[s][p]).

Configuring action groups

Action groups are named collections of action patterns, REST paths, or both, defined statically in opensearch.yml. You choose the group name, so use any name that is meaningful for your operations:

plugins.security.audit.config.action_groups.BULK: "indices:data/write/bulk*,/_bulk"
plugins.security.audit.config.action_groups.SEARCH: "indices:data/read/search*,/_search"
plugins.security.audit.config.action_groups.INDEX_ADMIN: "indices:admin/*"

Each action group maps a name to a comma-separated list of literal or wildcard patterns, such as indices:data/write/bulk*, /_bulk, or indices:data/write/*. All patterns use the same matcher; characters such as :, /, and * do not assign a pattern to a particular audit layer.

Action groups are static settings and require a node restart to change. The group names themselves are case-sensitive.

Configuring body logging exclusions

The body_logging_exclusions setting references action group names or raw patterns. To set the initial exclusions, configure the setting in opensearch.yml:

plugins.security.audit.config.body_logging_exclusions:
  - BULK

The setting is dynamic, so you can also update the exclusions at runtime without restarting the cluster:

PUT _cluster/settings
{
  "persistent": {
    "plugins.security.audit.config.body_logging_exclusions": ["BULK", "SEARCH"]
  }
}

Each entry in the list is processed as follows:

1. If the entry matches a defined action group name, that group’s patterns are expanded.
2. If it does not match any group name, the entry is treated as a raw pattern (literal or wildcard).

For entries that do not match a defined group name, a warning is logged if the entry contains no :, does not start with /, and contains no *, since it is unlikely to match an action or REST path. Note that this check looks for a leading / only, not a / anywhere in the entry: an entry such as _bulk/items contains a / but does not start with one, so the warning is still logged for it. This check only determines whether to log a warning; the entry is still included in the combined matcher.

Bulk requests

When resolve_bulk_requests: true is configured (logging individual bulk sub-items), the exclusion check uses the parent bulk action string. As a result, excluding the BULK group suppresses bodies for every sub-item (index, update, and delete) within that bulk request. You cannot keep index-item bodies while dropping delete-item bodies within the same bulk request.

Interaction with log_request_body

Body logging exclusions only apply when log_request_body is true. If log_request_body is false, no bodies are logged regardless of the exclusion configuration.

Example configuration

The following configuration defines three action groups and excludes the request bodies of bulk requests:

# opensearch.yml

# Define action groups (static, requires restart)
plugins.security.audit.config.action_groups.BULK: "indices:data/write/bulk*,/_bulk"
plugins.security.audit.config.action_groups.SEARCH: "indices:data/read/search*,/_search"
plugins.security.audit.config.action_groups.MONITORING: "cluster:monitor/*,indices:monitor/*"

# Initial exclusions (can be updated at runtime using _cluster/settings)
plugins.security.audit.config.body_logging_exclusions:
  - BULK

The following table describes whether the request body is logged for each request type when you use this configuration.

To add search exclusions at runtime, send the following request:

PUT _cluster/settings
{
  "persistent": {
    "plugins.security.audit.config.body_logging_exclusions": ["BULK", "SEARCH"]
  }
}

To clear all exclusions (resume logging all bodies):

PUT _cluster/settings
{
  "persistent": {
    "plugins.security.audit.config.body_logging_exclusions": []
  }
}

Log4j MDC routing

When using the log4j audit sink, enable Mapped Diagnostic Context (MDC) routing so that Log4j routes audit events to different appenders based on event properties:

plugins.security.audit.config.log4j.enable_mdc_routing: true

When enabled, the following MDC keys are set on each audit event:

• audit_category — The audit event category (for example, REQUEST_AUDIT or GRANTED_PRIVILEGES)
• audit_action — The action name
• audit_user — The effective user
• audit_request_type — The request type

Use Log4j routing appenders in log4j2.properties to split audit logs by category, user, or any other MDC key.

This is a static setting and requires a node restart.

Configure the audit log index name

By default, the Security plugin stores audit events in a daily rolling index named security-auditlog-YYYY.MM.dd:

plugins.security.audit.config.index: myauditlogindex

Use a date pattern in the index name to configure daily, weekly, or monthly rolling indexes:

plugins.security.audit.config.index: "'auditlog-'YYYY.MM.dd"

For a reference on the date pattern format, see the Joda DateTimeFormat documentation.

(Advanced) Tune the thread pool

The Search plugin logs events asynchronously, which minimizes the performance impact on your cluster. The plugin uses a fixed thread pool to log events:

plugins.security.audit.config.threadpool.size: <integer>

The default setting is 10. Setting this value to 0 disables the thread pool, which means the plugin logs events synchronously. To set the maximum queue length per thread:

plugins.security.audit.config.threadpool.max_queue_len: 100000

Disabling audit logs

To disable audit logs after they’ve been enabled, remove the plugins.security.audit.type: internal_opensearch setting from opensearch.yml, or switch off the Enable audit logging check box in OpenSearch Dashboards. The plugins.security.audit.enabled runtime toggle, set using the Cluster Settings API, applies only to standalone audit logging; see Standalone audit logging for instructions.

Audit user account manipulation

To enable audit logging on changes to a security index, such as changes to roles mappings and role creation or deletion, use the following settings in the compliance: portion of the audit log configuration, as shown in the following example:

_meta:
  type: "audit"
  config_version: 2

config:
  # enable/disable audit logging
  enabled: true

  ...


  compliance:
    # enable/disable compliance
    enabled: true

    # Log updates to internal security changes
    internal_config: true

    # Log only metadata of the document for write events
    write_metadata_only: false

    # Log only diffs for document updates
    write_log_diffs: true

    # List of indices to watch for write events. Wildcard patterns are supported
    # write_watched_indices: ["twitter", "logs-*"]
    write_watched_indices: [".opendistro_security"]