Elastic

Elasticsearch is the most widely deployed search engine, and it's the most flexible option in this set of guides: you control the mappings, the analyzers and every part of the query. It covers keyword search, vector search and hybrid search in one index, and it can also serve as the retrieval layer for an AI assistant.

This guide builds on Indexing Agility Content for Search. Read that first. It covers the webhook, signature verification, re-fetching from the Fetch API, deletes and the full reindex, which are the same for every provider. This guide adds the Elastic-specific parts.

Seen an older Agility + Elastic guide that uses App Search or the Elastic web crawler in Kibana? Those products aren't part of Elasticsearch 9 or Elastic Cloud Serverless. This guide uses Elasticsearch's own APIs, which work everywhere.

Choose where it runs

• Elastic Cloud Serverless — an "Elasticsearch" project (recommended). There are no clusters to size, it's billed on usage, and semantic and hybrid search work out of the box. Note that it has a small baseline cost even when idle.
• Elastic Cloud Hosted or self-managed. You get full control, but some features depend on your subscription: hybrid search with RRF, the Elastic Inference Service and Agent Builder need the Enterprise tier. Keyword search and everything in the first half of this guide work on any tier.

npm install @elastic/elasticsearch

Version 9.5 of the client needs Node.js 22 or later, and it runs on the Node.js runtime, not an Edge runtime.

# .env.local
ELASTIC_URL=https://<your-project>.es.<region>.elastic.cloud   # from your project's connection details
ELASTIC_INDEX=agility-content                                   # an alias; see "Rebuilding"
ELASTIC_WRITE_API_KEY=...    # server only: webhook + reindex
ELASTIC_SEARCH_API_KEY=...   # server only: the search route

Create two API keys

Create keys in Kibana (Stack Management > API keys) or with the API, each scoped to your index:

// Read-only key for the search route
await es.security.createApiKey({
  name: "agility-search",
  role_descriptors: {
    search: { indices: [{ names: ["agility-content*"], privileges: ["read", "view_index_metadata"] }] },
  },
})

// Write key for the webhook and reindex (manage is for creating indices and moving the alias)
await es.security.createApiKey({
  name: "agility-indexer",
  role_descriptors: {
    indexer: { indices: [{ names: ["agility-content*"], privileges: ["create_index", "index", "delete", "manage"] }] },
  },
})

Use the encoded value from each response. Never send either key to the browser. Serverless doesn't allow cross-origin requests anyway, so all searches go through a route on your server.

Create the index

The mapping mirrors the SearchRecord shape from the indexing guide. Create a versioned index, and point an alias at it, so that later you can rebuild without downtime:

// scripts/create-index.ts
import { Client } from "@elastic/elasticsearch"

const es = new Client({ node: process.env.ELASTIC_URL!, auth: { apiKey: process.env.ELASTIC_WRITE_API_KEY! } })

const index = `agility-content-${Date.now()}`

await es.indices.create({
  index,
  mappings: {
    dynamic: false,
    properties: {
      kind: { type: "keyword" },
      agilityId: { type: "integer" },
      locale: { type: "keyword" },
      referenceName: { type: "keyword" },
      title: { type: "text", analyzer: "english" },
      description: { type: "text", analyzer: "english" },
      body: { type: "text", analyzer: "english" },
      url: { type: "keyword", index: false },
      updatedAt: { type: "date" },
    },
  },
})

await es.indices.putAlias({ index, name: "agility-content" })

dynamic: false stops unexpected fields from being added to the mapping. The english analyzer handles stemming and stop words. For other locales, use one index per locale with the matching language analyzer.

Write to the index

This is the searchIndex object the webhook route and the reindex script import:

// lib/search/provider.ts
import { Client } from "@elastic/elasticsearch"
import type { SearchRecord } from "./types"

const es = new Client({
  node: process.env.ELASTIC_URL!,
  auth: { apiKey: process.env.ELASTIC_WRITE_API_KEY! },
  serverMode: process.env.ELASTIC_SERVERLESS === "false" ? "stack" : "serverless",
})
const index = process.env.ELASTIC_INDEX ?? "agility-content"

export const searchIndex = {
  async upsert(record: SearchRecord) {
    await es.index({ index, id: record.id, document: record })
  },

  async upsertMany(records: SearchRecord[]) {
    if (!records.length) return
    const result = await es.helpers.bulk<SearchRecord>({
      datasource: records,
      onDocument: (doc) => ({ index: { _index: index, _id: doc.id } }),
    })
    if (result.failed) throw new Error(`${result.failed} of ${result.total} documents failed to index`)
  },

  async remove(id: string) {
    await es.delete({ index, id }, { ignore: [404] })
  },
}

• index with an explicit id is an upsert: it replaces the whole document.
• ignore: [404] makes a delete of a document that's already gone succeed, so duplicate Deleted webhooks are harmless.
• Don't pass refresh: true on webhook writes. Documents are searchable within about a second anyway, and forcing a refresh on every write slows the cluster down.

Plug this into the route handler and reindex script from the indexing guide, and publishing in Agility now updates Elasticsearch.

Query from your site

// app/api/search/route.ts
import { Client } from "@elastic/elasticsearch"
import type { SearchRecord } from "@/lib/search/types"

const es = new Client({ node: process.env.ELASTIC_URL!, auth: { apiKey: process.env.ELASTIC_SEARCH_API_KEY! } })

export async function GET(req: Request) {
  const url = new URL(req.url)
  const q = (url.searchParams.get("q") ?? "").slice(0, 200)
  const locale = url.searchParams.get("locale") ?? "en-us"
  if (!q) return Response.json({ count: 0, hits: [] })

  const res = await es.search<SearchRecord>({
    index: process.env.ELASTIC_INDEX ?? "agility-content",
    size: 10,
    query: {
      bool: {
        must: [{ multi_match: { query: q, fields: ["title^3", "description^2", "body"], fuzziness: "AUTO" } }],
        filter: [{ term: { locale } }],
      },
    },
    highlight: {
      pre_tags: ["<mark>"],
      post_tags: ["</mark>"],
      fields: { body: { fragment_size: 160, number_of_fragments: 1 } },
    },
    _source: ["title", "description", "url", "kind"],
  })

  const hits = res.hits.hits.map((h) => ({ id: h._id, ...h._source, caption: h.highlight?.body?.[0] }))
  const total = typeof res.hits.total === "number" ? res.hits.total : res.hits.total?.value

  return Response.json({ count: total, hits }, { headers: { "Cache-Control": "s-maxage=60, stale-while-revalidate=300" } })
}

title^3 weights a title match three times as heavily as a body match, and fuzziness: "AUTO" tolerates small typos. The locale filter is applied on the server, so visitors can't remove it.

Search UI

The route returns { count, hits }, with each hit carrying title, url and a highlighted caption. That's the shape the minimal search box in the indexing guide expects. If you'd rather use a component library:

• Search UI — Elastic's React library. Use @elastic/search-ui-elasticsearch-connector through its API proxy connector, so the key stays on your server. Don't use the App Search connector; it's deprecated.
• Searchkit — a community project that lets Algolia's InstantSearch widgets run against Elasticsearch.

Add semantic and hybrid search

Semantic search matches on meaning, so "cancel my subscription" can find a page titled "Ending your plan". Hybrid search runs keyword and semantic retrieval together and merges the results, and it's usually better than either on its own.

With a semantic_text field, Elasticsearch generates the embeddings for you, and splits long text into passages automatically. There's no embedding code in your webhook. This works out of the box on Serverless and on Hosted with the Enterprise tier.

1. Find the embedding endpoint available to you, and pin it. The default for semantic_text has changed between versions, and an unpinned field can end up on a different model after an upgrade:

   GET _inference/_all
   

2. Add a semantic_text field and copy the text fields into it. Put this in a new index, because the mapping of an existing field can't be changed:

   properties: {
     // …existing fields…
     title: { type: "text", analyzer: "english", copy_to: "semantic" },
     description: { type: "text", analyzer: "english", copy_to: "semantic" },
     body: { type: "text", analyzer: "english", copy_to: "semantic" },
     semantic: { type: "semantic_text", inference_id: ".jina-embeddings-v5-text-small" }, // the ID from step 1
   }
   

3. Query with an RRF retriever, which searches the keyword fields and the semantic field and fuses the results:

   const res = await es.search<SearchRecord>({
     index: process.env.ELASTIC_INDEX ?? "agility-content",
     size: 10,
     retriever: {
       rrf: {
         query: q,
         fields: ["title^3", "description^2", "body", "semantic"],
         filter: [{ term: { locale } }],
         rank_window_size: 50,
       },
     },
     highlight: { fields: { semantic: { type: "semantic", number_of_fragments: 1 } } },
     _source: ["title", "description", "url", "kind"],
   })
   

   The semantic highlighter returns the passage that best matches the question, which makes a good result snippet. RRF results can't be sorted, and paging only reaches as far as rank_window_size.

Things to know:

• Every index write re-embeds the document, even if its text hasn't changed. That's fine at the rate editors publish, but a full reindex re-embeds everything, so budget for it. If you reindex often, store a hash of the text and skip unchanged items.
• Reranking — for the highest relevance, wrap the retriever in a text_similarity_reranker, which re-scores the top results with a reranking model. See semantic reranking.
• Self-managed clusters have no default embedding endpoint. Deploy a model to a machine learning node, or connect to the Elastic Inference Service, before semantic_text will work.

Ground an AI assistant on your content

Agent Builder (GA) lets you build agents in Kibana that search your indices and answer questions from them. Every deployment on 9.2 or later, and every Serverless project, exposes those tools through an MCP endpoint:

{KIBANA_URL}/api/agent_builder/mcp

Point any MCP client (Claude, Cursor, VS Code and others) at it with an API key, and your AI assistant can search your Agility content directly. The older standalone mcp-server-elasticsearch package is deprecated in favour of this endpoint. On Serverless, the first 1,000 Agent Builder executions each month are free. On Hosted, Agent Builder needs the Enterprise tier.

Rebuilding without downtime

Because the app reads and writes through the agility-content alias, you can rebuild into a fresh index and switch over in one atomic step:

const next = `agility-content-${Date.now()}`
// 1. Create `next` with the mapping above
// 2. Run the full reindex, writing to `next`
// 3. Swap the alias and drop the old index
const old = Object.keys(await es.indices.getAlias({ name: "agility-content" }))
await es.indices.updateAliases({
  actions: [
    ...old.map((i) => ({ remove: { index: i, alias: "agility-content" } })),
    { add: { index: next, alias: "agility-content" } },
  ],
})
for (const i of old) await es.indices.delete({ index: i })

Use the same approach to change a mapping, add the semantic field, or change the embedding model.

OpenSearch

OpenSearch is an open-source fork of Elasticsearch 7.10. It's the usual choice on AWS (Amazon OpenSearch Service and OpenSearch Serverless). The overall integration is the same — the webhook, the record and the reindex — but the code in this guide won't run on it unchanged:

• Use the @opensearch-project/opensearch client. The Elasticsearch client refuses to connect to non-Elastic servers.
• Semantic and hybrid search use OpenSearch's own neural search, hybrid query and search pipelines rather than semantic_text and retrievers. You deploy and register the embedding model yourself.

Troubleshooting

• security_exception on write. The write key lacks index or delete on the index behind the alias. Check the names pattern covers agility-content-*.
• version_conflict_engine_exception. Two deliveries for the same item raced each other. It's harmless because the next delivery or the reindex corrects it, but it's also a sign your endpoint is slow. Check the webhook's History for timeouts.
• Semantic queries fail with an inference error. The inference_id doesn't exist on this deployment. Run GET _inference/_all.
• A result is stale. Check the webhook's History in Agility, then run the reindex. See Debugging.

Learn more

• Elasticsearch documentation
• Elasticsearch JavaScript client
• Hybrid search and semantic_text
• Elastic Cloud Serverless