> ## Content Index
> Fetch the complete content index at: https://tomgersic.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Salesforce Vector Search - External Query API
- URL: https://tomgersic.com/salesforce-vector-search-external-api/
- Published: 2025-01-13T21:18:56.000Z
- Updated: 2025-01-14T13:52:42.000Z
- Description: This is a Salesforce Data Cloud step-by-step guide that shows how to create a Vector Search Index from Knowledge Base Articles in a Core Salesforce Org and then Query that Vector Search Index from the external Query API.
- Author: Tom Gersic
- Tags: Tech Stuff, Humans and AI

This is a Salesforce Data Cloud step-by-step guide that shows how to create a Vector Search Index from Knowledge Base Articles in a Core Salesforce Org and then Query that Vector Search Index from the external Query API. This post will be informative for any tech team looking to use the Salesforce Vector Database with external platforms such as ChatGPT Enterprise, Slack, or any preferred agentic middleware solution (n8n, Langchain, Flowise, Zapier, [Make.com](http://make.com/?ref=tomgersic.com), etc). Here's the overall idea:

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Vector-Diagram-Example---Page-3--1-.png)

I aim to show each step needed from start to finish. There are a lot of steps involved, so if you're already familiar with how to set up Data Cloud, you may want to skip to the API section at the end.

**Prerequisite:** You will need access to a Salesforce Org and Salesforce Data Cloud in order to use the Salesforce Vector Database (Search Index). There are several free ways to do this. As an example, you could:

- [Sign up for a Data Cloud Developer Edition](https://trailhead.salesforce.com/content/learn/projects/create-a-data-stream-in-data-cloud/get-started-with-a-data-cloud-developer-edition?ref=tomgersic.com)
- [Enable Data Cloud in your Enterprise Edition (EE) or above Production Org](https://help.salesforce.com/s/articleView?id=000396380&type=1&ref=tomgersic.com)
- [If you are an SI or ISV partner, you could create a Simple Demo Org (SDO)](https://partnerlearningcamp.salesforce.com/s/demo-org?ref=tomgersic.com)

I will use a Simple Demo Org (SDO) for this demonstration.

## **Step 1: Set up Data Cloud**

First, you'll need to set up Data Cloud in your org. Assuming it has already been provisioned, you'll find the Data Cloud Setup link to the admin interface at the top right, under the standard Salesforce setup link.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-11-at-5.55.28-PM.png)

### 1.1: Set Up Your Data Cloud Instance

Once there, you can enable data cloud from the Data Cloud Setup Home.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-3.00.51-PM.png)

First Step: Data Cloud Setup

### 1.2: Salesforce CRM Integration

Next, you will need to connect your Core Org to Data Cloud. This is a little confusing if you're using Data Cloud from within your Core Org, but it works the same way as you connect other external Orgs (or other Data Sources). You need to set up the connection using Salesforce CRM Setup under the Integrations heading.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.25.21-PM.png)

Setup Salesforce CRM: Data Bundles

In this instance, you can see I've installed **Sales Cloud** **(1.3)** and am installing the **Service Cloud (5.0)** Data Bundle (bottom right). The CDP bundles listed haven't been installed. This brings up a standard Package Install page, and I'm choosing to install it for all users.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.26.36-PM.png)

### 1.3: Permission Set Setup

Lastly, since we will be setting up a Data Stream for Knowledge Articles, Data Cloud will need *Read All* and *Modify All* permissions on your Core Org Knowledge Article Objects. Data Cloud uses an integration user named *Platform Integration User* with a Permission Set named *Data Cloud Salesforce Connector.* 

To provide access to Knowledge, you need to give access to the *Data Cloud Salesforce Connector* Permission Set for these Objects under the *Permission Set Object Settings*:

- CaseArticle\_\_c
- KCSArticle\_DataCategorySelection\_\_c
- KCSArticle\_ka\_\_c
- KCSArticle\_kav\_\_c
- Knowledge

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-11-at-6.13.09-PM.png)

Example Permission Set Setting for Knowledge Object

You also need to provide access to Knowledge Management in the *Permission Set App Settings*.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-3.10.49-PM.png)

Permission Set: App Settings

## **Step 2: Data Streams Setup**

A **Data Stream** is a pipeline that brings data into Data Cloud for unification, analysis, and activation. In this case, you'll be bringing that Data into the Data Cloud environment in the Core Org that you're already in. But it could also be a separate Org, or multiple other Core Orgs, or other services entirely like Amazon S3, Airtable, Snowflake, or CosmosDB.

For these next steps, you will want to exit *Data Cloud Setup*, and select just regular old *Data Cloud* from the App Launcher for further... ah... well, setup. The first tab next to Home is Data Streams, which is where you should go next.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-11-at-6.17.09-PM.png)

Finally done with Setup! Now it's time for more Setup!

Next, we're going to select the Salesforce CRM org that we connected back in Step 1.2

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-11-at-6.20.36-PM.png)

New Data Stream for Salesforce CRM (Our Core Org)

After clicking Next, we'll be able to select the Knowledge *Standard Data Bundle.*

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-03-at-5.24.56-PM.png)

Select Knowledge Standard Data Bundle

You can see the Knowledge bundle includes 7 Objects

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-05-at-5.37.55-PM.png)

Standard Data Bundle: Knowledge

This is where you may get this error if you didn't get all of the Permission Set settings right above:

> **Insufficient Permissions:* You do not have permission to import records for 1 object. Please check with your Salesforce Admin to make the following object available for use.* 
> **Knowledge:* 7 Fields unavailable: VersionNumber, IsVisibleInPkb, IsLatestVersion, IsVisibleInPrm, ArticleMasterLanguage, IsVisibleInCsp, IsVisibleInApp.*

If you get this error, make sure you've updated both the *App Settings* and the *Object Settings* for the *Data Cloud Salesforce Connector Permission Set*. See above for more info.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-03-at-5.45.58-PM.png)

Insufficient Permissions Error.

When you go to create your Data Stream, it won't include the actual Knowledge Article text by default. Since our goal is to create a vector search on the Knowledge Article text, we'll need to add that to the Data Stream by checking the box next to *FAQ\_Question\_c* and *FAQ\_Answer\_c*:

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.33.56-PM.png)

Add FAQ\_Question\_c and FAQ\_Answer\_c to the Data Stream

We'll leave these settings as their defaults before Deploying. The [data model for Salesforce Knowledge](https://developer.salesforce.com/docs/atlas.en-us.knowledge%5Fdev.meta/knowledge%5Fdev/knowledge%5Fdevelopment%5Fobject%5Fmodel.htm?ref=tomgersic.com) is a little complicated, but the main thing to know for the purposes of this exercise is that the actual Knowledge Article text is contained in the *KnowledgeArticleVersion* Object, which is shown as *Knowledge\_\_kav\_Home* here ("*kav"* is short for Knowledge Article Version).

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.34.49-PM.png)

Defaults are fine

Once deployed, you'll see that you have some new Data Streams that have not yet been refreshed, meaning they don't have any data ingested yet.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.37.12-PM.png)

Data Streams

This process has also created corresponding Data Lake Objects and Data Model Objects for you:

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.38.37-PM.png)

Data Lake Objects are Auto-Created

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.39.15-PM.png)

Data Model Objects are Auto-Created

Data Streams need to be refreshed before you can do anything with them, so we'll go ahead and refresh each one:

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.40.03-PM.png)

Refresh the Data Streams

Lastly, the default Knowledge Data Bundle doesn't map the KB article content *FAQ\_Question\_\_c* or *FAQ\_Answer\_\_c* to the Data Model Object (DMO), so that's the last step we need to do. In the *Knowledge\_\_kav\_Home* Data Stream, click the Review button in the Data Mapping section.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.31.33-AM.png)

The field label for *FAQ\_Question\_\_c* is Question" and the field label for *FAQ\_Answer\_\_c* is "Details":

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.35.56-AM.png)

These fields aren't mapped by default, so we'll map *Details* to *Article Content,* and Question to a Custom Field (click *Add New Field*) that we'll name *Article Question*. Click the *Save and Close* button at the top of the page, and then the *Refresh Now* button from the *Knowledge\_\_kav\_Home* Data Stream page.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-4.47.25-PM.png)

## Step 3: Search Index Setup

The *Search Index*tab is where you will set up a Vector Search index for your Unstructured Data Lake Object (DLO). In the *Data Cloud* App, click the *More* tab to see a tab named *Search Index*. Click *New,* and you'll be presented with a dialog box like this. We will choose *Advanced Setup*.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-3.25.17-PM.png)

There are two types of Search Index available today. *Vector Search* and *Hybrid Search.* 

### Step 3.0: What is a Vector Database?

Either will work for the purposes of our demonstration, but here's a brief description of the difference:

A **Vector Database** takes long strings of text (a.k.a. "unstructured data"), breaks it up into smaller chunks (usually half a page to a page worth of text), and converts it into "Vectors" (a.k.a. an ordered list of numbers). Here's what a Vector looks like in [Pinecone.io](https://pinecone.io/?ref=tomgersic.com), a popular Vector Database.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-12-at-3.33.52-PM.png)

**A Vector Database** makes it possible to do what's called "Similarity Search" or "Semantic Search", which is much more powerful for Generative AI Q&A than a typical Keyword search. 

*Here's an example:* let's imagine you are searching through a database of clipart metadata looking for a picture of a pony.

- A *Keyword Search* will match images where the word "pony" is used exactly, but it won't find pictures of small horses, or other 4-legged barnyard animals.
- A *Vector Search* will correctly identify that a "pony" is similar to a "horse", which is also more similar to a "cow" than it is to an "apple". This sort of semantic understanding is what makes a Vector Database so powerful.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Vector-Diagram-Example.png)

Vector Database Illustration

Vector Search is extremely powerful for this sort of semantic matching, but it's not great at other types of basic queries that computers are normally pretty good at like *"how many apples are there?"* or *"what time is it"*. That's why the best option for any application is often a hybrid between Vector Search and a Keyword Search of additional metadata or data.

### Step 3.1: Select the Search Type

Back to *Salesforce Data Cloud*,the *Vector Search* type does exactly what is illustrated above. We'll select *Hybrid Search because it* includes a *Vector Search,* but it also adds on some elements of *Keyword Search* in order to round out those shortcomings of a *Vector Search.*

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.46.26-AM.png)

### Step 3.2: Chunking

Vector Databases ingest data through a process called "Chunking and Embedding". If you think of vectorizing a long document, like a book, the first step will be to chunk it into smaller sections. There are a variety of different chunking strategies that try to keep contextually similar things within the chunk, but for the most part the idea is to create more manageable pieces of information before generating the vector embeddings.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Vector-Diagram-Example---Page-2.png)

Chunking and Embedding

In this step, we'll select the fields that contain the most text for the Knowledge Article in order to use those for our Vector Embeddings

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.47.30-AM.png)

Select fields for chunking that contain the most text.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.48.00-AM.png)

Selected Fields for Chunking

### Step 3.3: Embedding

The next step is to pass these Chunks to an Embedding Model. This uses a Large Language Model (LLM) to generate vector "embeddings" to store in the Vector Database. Each of the LLM vendors have options for vector embeddings (OpenAI, Anthropic, etc).

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.48.14-AM.png)

### Step 3.4: Filtering and Ranking

In this step you can control the search results a bit by adding filters and ranking the results.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.48.48-AM.png)

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.49.38-AM.png)

### Step 3.5: Review and Build

That's it. Just click Save at this point and let it run.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-7.49.53-AM.png)

## Step 4: Query API

Now that you've built a Vector Search Index from Data Cloud, we've finally gotten to the goal of this article! In this step, we will show how to query Vector Index from an external source using the Query API. We'll be using Postman in order to do this, so start off by signing up for a free account at [https://www.postman.com/](https://www.postman.com/?ref=tomgersic.com)

### Step 4.1: Postman Setup

To show how to use the Query API, we'll start with the [Postman Data Cloud API](https://www.postman.com/salesforce-developers/salesforce-developers/collection/vkln4gx/salesforce-data-cloud-apis?ref=tomgersic.com) interface published by *salesforce-developers*. Just follow the instructions on Postman to fork it into your own environment with the "Fork the collection using this button".

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-8.44.01-AM.png)

### Step 4.2: Postman Authentication with Data Cloud

Postman makes this step very easy. From the *Overview* tab of the *Salesforce Data Cloud APIs* page*,* click on the *Authentication* tab, scroll down and click the *"Get new Access Token"* button, and then *"Use Token"* from the dialog box that comes up next.

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-8.47.14-AM.png)

You can then Complete the authentication by clicking Send in the script under the Auth folder:

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-8.49.08-AM.png)

### Step 4.3: Postman Authentication with Data Cloud

Next we will complete a Vector Search with the *Query API V2*. Here's an example SQL query for: *"What is our return policy"*.

> SELECT c.Chunk\_\_c, v.score\_\_c, v.ArticleNumber\_\_c from vector\_search(TABLE(Knowledge\_Article\_Version\_index\_\_dlm), 'What is our return policy', '', 100) as v join Knowledge\_Article\_Version\_chunk\_\_dlm as c on v.SourceRecordId\_\_c=c.RecordId\_\_c ORDER by v.score\_\_c desc LIMIT 10

With this, you can see the vector search results from the API

![](https://storage.ghost.io/c/51/6d/516de838-8dcf-4db9-8bbd-9eef82013d82/content/images/2025/01/Screenshot-2025-01-13-at-8.53.42-AM.png)

Vector Search with Query API V2

## Summary and Next Steps

This article was a step-by-step guide that shows how to set up Data Cloud from scratch, ingest Knowledge Articles into an Unstructured *Data Model Object (DMO)* using a *Data Stream*, and then create a Vector *Search Index* from that unstructured data. From that point, we've shown how to query this from an external source using PostMan in order to authenticate and query the *Data Cloud Query API V2*.

The next step from this point will be to show how to build a middleware that can connect a User Experience like ChatGPT Enterprise with this Vector Search API. Stay tuned for more.