Faiss scalar quantization

OpenSearch supports built-in scalar quantization for the Faiss engine. The Faiss scalar quantizer converts 32-bit floating-point input vectors into lower-bit representations during ingestion and stores the quantized vectors in a vector index. OpenSearch supports 1-, 2-, 4-, and 16-bit Faiss scalar quantization.

Quantization can decrease the memory footprint in exchange for some loss in recall. When used with SIMD optimization, Faiss scalar quantization can also significantly reduce search latencies and improve indexing throughput.

SIMD optimization is not supported on Windows. Using Faiss scalar quantization on Windows can lead to a significant drop in performance, including decreased indexing throughput and increased search latencies.

Using Faiss scalar quantization

To use Faiss scalar quantization, set the k-NN vector field’s method.parameters.encoder.name to sq when creating a vector index. You must specify the bits parameter in the method.parameters.encoder.parameters object:

PUT /test-index
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector1": {
        "type": "knn_vector",
        "dimension": 3,
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "parameters": {
            "encoder": {
              "name": "sq",
              "parameters": {
                "bits": 16
              }
            },
            "ef_construction": 256,
            "m": 8
          }
        }
      }
    }
  }
}

The Faiss sq encoder supports the following parameters.

The type and clip parameters are supported only for 16-bit quantization. If you set bits to any other value and specify type or clip, the request is rejected.

1-bit, 2-bit, and 4-bit quantization

For the lowest memory footprint, quantize each vector dimension to 1, 2, or 4 bits. Each bit width corresponds to a compression_level.

Fewer bits per dimension produce a smaller index at the cost of recall. These bit widths are supported only for the HNSW method; IVF requires 16-bit quantization. 1-bit quantization uses memory-optimized search.

The following example enables 2-bit quantization for float fields by setting compression_level to 16x in the knn_vector mapping. To use 1-bit or 4-bit quantization, set compression_level to 32x or 8x:

PUT /test-index
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector1": {
        "type": "knn_vector",
        "dimension": 8,
        "space_type": "l2",
        "compression_level": "16x"
      }
    }
  }
}

Alternatively, specify the encoder explicitly by setting bits in the sq encoder:

PUT /test-index
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector1": {
        "type": "knn_vector",
        "dimension": 8,
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "parameters": {
            "encoder": {
              "name": "sq",
              "parameters": {
                "bits": 2
              }
            },
            "ef_construction": 256,
            "m": 8
          }
        }
      }
    }
  }
}

1-bit quantization is also supported for half_float vectors. Because half_float fields do not accept an encoder in the method mapping, set compression_level to 16x instead. This level is measured against the 16-bit baseline of half_float vectors and therefore maps each dimension to a single bit.

16-bit quantization

With 16-bit quantization, the Faiss scalar quantizer converts 32-bit floating-point vectors into 16-bit vectors and stores them in the vector index. At search time, the stored 16-bit values are converted back to 32-bit floating-point values for distance computation. On Intel Sapphire Rapids or newer-generation processors, OpenSearch computes distances directly on the 16-bit values, using AVX-512 BF16 instructions for bf16 inner product and AVX-512 FP16 instructions for fp16 cosine similarity. For more information, see SIMD optimization.

OpenSearch supports two 16-bit encoder types, specified in the type parameter:

• fp16 (Default): The IEEE 754 half-precision format (FP16), which uses 5 exponent bits and 10 mantissa bits. This format provides the highest 16-bit precision but a narrower value range of [-65504.0, 65504.0]. Use fp16 when all of your vector values fall within that range.
• bf16: The bfloat16 format (BF16), which uses 8 exponent bits and 7 mantissa bits. It spans the same value range as 32-bit floating-point numbers but with lower precision, so it can cause a slightly larger drop in recall. Use bf16 when your vectors may contain values outside of the fp16 range or when you want to avoid fp16 range validation and clipping.

Both encoder types store 2 bytes per vector dimension, so both reduce the memory footprint by a factor of 2 with minimal loss in recall.

The fp16 encoder

The fp16 encoder converts 32-bit vectors into their 16-bit counterparts. For this encoder type, the vector values must be in the [-65504.0, 65504.0] range. To define how to handle out-of-range values, you can specify the clip parameter. By default, this parameter is false, and any vectors containing out-of-range values are rejected.

When clip is set to true, out-of-range vector values are rounded up or down so that they are in the supported range. For example, if the original 32-bit vector is [65510.82, -65504.1], the vector will be indexed as a 16-bit vector [65504.0, -65504.0].

We recommend setting clip to true only if very few vector dimensions lie outside of the supported range. Rounding the values may cause a drop in recall.

The following example specifies the Faiss fp16 encoder with 16-bit quantization, which rejects any indexing request that contains out-of-range vector values (because the clip parameter is false by default):

PUT /test-index
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector1": {
        "type": "knn_vector",
        "dimension": 3,
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "parameters": {
            "encoder": {
              "name": "sq",
              "parameters": {
                "bits": 16
              }
            },
            "ef_construction": 256,
            "m": 8
          }
        }
      }
    }
  }
}

When indexing vectors, ensure that each vector dimension is in the supported range:

PUT test-index/_doc/1
{
  "my_vector1": [-65504.0, 65503.845, 55.82]
}

When querying vectors, the query vector has no range limitation:

GET test-index/_search
{
  "size": 2,
  "query": {
    "knn": {
      "my_vector1": {
        "vector": [265436.876, -120906.256, 99.84],
        "k": 2
      }
    }
  }
}

The bf16 encoder

Introduced 3.9

The bf16 encoder converts 32-bit vectors into bfloat16 vectors. Because bfloat16 uses the same number of exponent bits as a 32-bit floating-point number, it spans the same value range. Any finite 32-bit floating-point value can therefore be indexed without range-based rejection or clipping. During quantization, each value is rounded to the nearest representable bfloat16 value, reducing the mantissa from 23 bits to 7 bits. As a result, bf16 trades precision for range compared to fp16.

The following example specifies the Faiss bf16 encoder with 16-bit quantization:

PUT /test-index-bf16
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector1": {
        "type": "knn_vector",
        "dimension": 3,
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "parameters": {
            "encoder": {
              "name": "sq",
              "parameters": {
                "bits": 16,
                "type": "bf16"
              }
            },
            "ef_construction": 256,
            "m": 8
          }
        }
      }
    }
  }
}

Vector values are not restricted to a range, so no dimension is rejected:

PUT test-index-bf16/_doc/1
{
  "my_vector1": [-150000.5, 123456.75, 55.82]
}

Only finite values are accepted. Indexing requests containing NaN or infinite values are rejected.

Note the following limitations of the bf16 encoder:

• Setting clip to true is not supported and causes the request to be rejected. Because bf16 spans the full 32-bit floating-point value range, clipping has no effect.
• Remote index build is not supported. Indexes using the bf16 encoder are always built locally.

Memory estimation

In the best-case scenario, quantized vectors require the following percentage of the memory that 32-bit vectors require.

HNSW memory estimation

The memory required for Hierarchical Navigable Small Worlds (HNSW) is estimated to be 1.1 * (dimension * bits_per_dimension / 8 + 8 * m) bytes per vector, where m is the maximum number of bidirectional links created for each element during the construction of the graph.

For example, assume that you have 1 million vectors with a dimension of 256 and an m of 16. The memory requirement for each bit width can be estimated as follows.

IVF memory estimation

The memory required for IVF is estimated to be 1.1 * (((bytes_per_dimension * dimension) * num_vectors) + (4 * nlist * dimension)) bytes, where nlist is the number of buckets to partition vectors into.

As an example, assume that you have 1 million vectors with a dimension of 256 and an nlist of 128.

IVF is only supported for 16-bit Faiss scalar quantization. 1-bit, 2-bit, and 4-bit quantization are supported only for the HNSW method.

For 16-bit quantization, the memory requirement can be estimated as follows:

1.1 * (((2 * 256) * 1,000,000) + (4 * 128 * 256))  ~= 0.525 GB

Next steps

• Memory-optimized vectors
• k-NN query