You're viewing the current version of Migration Assistant documentation (Kubernetes/EKS-based). For the classic ECS/CDK-based version, see the classic documentation.

Tracking and remediating failed documents

During a backfill, Reindex-from-Snapshot (RFS) retries most document errors automatically. A document counts as failed only when the error is terminal, either because the error is non-retryable or because RFS exhausted the retry limit. You can list which documents failed, determine why they failed, and remediate them.

Where failures are recorded

Terminal document failures are recorded in two places:

• The failed document stream is a durable inventory of terminal failures written to Amazon S3 as gzip-compressed NDJSON. Each record identifies the document and captures enough detail to diagnose or resubmit it without returning to the source cluster.
• The RFS worker logs contain lower-level diagnostic detail. The workers log each failed bulk request, including the target index, failed item count, root cause, and the request and response bodies.

The failed document stream is off by default. If you do not enable it before running the backfill, a failed migration leaves no durable inventory of which documents did not land; you would be limited to the worker logs. Enable it as described in Enabling the failed document stream before you start the backfill.

Enabling the failed document stream

Setting an S3 bucket in the migration’s documentBackfillConfig enables the stream. There is no separate enable flag, and the stream does not fall back to the deployment’s default bucket, so you must name a bucket explicitly. Complete these steps from a Migration Console shell before you start the backfill:

1. Choose a bucket. On Amazon EKS, you can use the deployment’s default bucket, migrations-default-<ACCOUNT_ID>-<STAGE>-<REGION>, to which Migration Assistant can write. The following command lists the default bucket for every Migration Assistant deployment in the account, so choose the one that matches your stage and Region:

   aws s3 ls | grep migrations-default
   

   Alternatively, create a bucket in the same AWS account:

   aws s3 mb s3://<BUCKET_NAME> --region <REGION>
   

2. Open the workflow configuration:

   workflow configure edit
   

3. Add the bucket to each migration’s documentBackfillConfig and then save:

   snapshotMigrationConfigs:
     - ...
       perSnapshotConfig:
         snap1:
           - documentBackfillConfig:
               failedDocumentStreamS3Bucket: <BUCKET_NAME>
   

4. Submit the workflow:

   workflow submit
   

5. After the backfill starts, confirm the stream location:

   console failed-document-stream location
   

To use a bucket in another AWS account or a bucket encrypted with a KMS key that you manage, also grant the Migration Assistant pod role access in the bucket policy or KMS key policy. On Amazon EKS, uninstalling Migration Assistant empties and deletes the default bucket by default. Copy any records that you want to keep before uninstalling Migration Assistant.

The following table lists the options that configure the failed document stream.

The console reports the session root:

s3://<bucket>/<prefix>session=<migration-UID>/

The individual gzip-compressed NDJSON objects are stored beneath that root:

s3://<bucket>/<prefix>session=<migration-UID>/index=<targetIndex>/worker=<workerId>/failed-document-stream-<timestamp>-<sequence>.ndjson.gz

Checking whether any documents failed

After the backfill finishes, run the following commands from a Migration Console shell:

workflow status

console failed-document-stream count

A count greater than 0 means that documents failed. workflow status can report the backfill as completed even when documents failed, so check the count rather than relying on the workflow status alone.

If the stream is configured but cannot be read, for example, because of missing S3 permissions, the console command fails rather than reporting no failures.

Inspecting failed documents

Run the following commands from a Migration Console shell. When more than one migration exists, add --migration <name> to select one:

# S3 location for the current session
console failed-document-stream location

# Count of distinct failed documents
console failed-document-stream count

# List failures as tab-separated rows without a header
# (columns: timestamp, targetIndex, documentId, failureClass, failureType)
console failed-document-stream list --limit 100

# Full records as JSON, including the captured request item and the OpenSearch response
console --json failed-document-stream list --limit 100

The following table lists the fields included in each record.

A document can appear in the stream more than once, so the console deduplicates records on read by targetIndex and documentId. Counts therefore reflect the number of distinct failed documents.

Reading the RFS worker logs

For lower-level detail, inspect the RFS worker logs. Each failed bulk request produces an error entry with the target index, failed item count, root cause, and the OpenSearch response body, plus a structured entry under the FailedRequestsLogger category containing the request and response bodies.

Individual failed-item bodies are deliberately omitted from the general worker log to avoid leaking document data; the full request body is emitted only to the dedicated FailedRequestsLogger category.

Remediating failures

Use the failed document stream to identify the main cause before retrying any documents.

Identifying the failure type

Group the failures returned by list by failureType to identify the cause, and use failureClass to determine whether the error was non-retryable or became terminal only after retries were exhausted. The following table lists common failure types and the recommended remediation for each.

Resubmitting the failed documents

The console --json failed-document-stream list output contains each failed document’s requestItem, so you can correct the root cause, such as a mapping or a transform, and then resubmit those documents to the target. When the original source document was available, requestItem holds that source content. It is not guaranteed to hold the exact transformed payload that was sent on the failed write.

Deleting failed document records

The Migration Console doesn’t provide a command to delete failed document records. To delete the current session’s records, remove the session location returned by the console failed-document-stream location command. This deletion is irreversible:

aws s3 rm --recursive s3://<BUCKET>/<PREFIX>session=<migration-UID>/

Related documentation

• Backfill
• Troubleshooting