Connect an OpenAI chat bot to the internet using Bing Search API

In the previous post, we saw what is OpenAI function calling and how to use it to chat with your organization's user directory using Microsoft Graph. Please have a look at the article here: Chat with your user directory using OpenAI functions and Microsoft Graph

In this post, we will implement function calling for a very common scenario of augmenting the large language model's responses with data fetched from internet search.

Since the Large Language model (LLM) was trained with data only up to a certain date, we cannot talk to it about events which happened after that date. To solve this, we will use OpenAI function calling to call out the Bing Search API and then augment the LLM's responses with the data returned via internet search.

This pattern is called Retrieval Augmented Generation or RAG. 

Let's look at the code now on how to achieve this. In this code sample I have used the following nuget packages:

https://www.nuget.org/packages/Azure.AI.OpenAI/1.0.0-beta.6/

https://www.nuget.org/packages/Azure.Identity/1.10.2/

The very first thing we will look at is our function definition for informing the model that it can call out to external search API to search information:

	"functions": [
	{
	"name": "search_online",
	"description": "Search online if the requested information cannot be found in the language model or the information could be present in a time after the language model was trained",
	"parameters": {
	"type": "object",
	"required": [
	"searchQuery"
	],
	"properties": {
	"searchQuery": {
	"type": "string",
	"description": "the text to search online to get the necessary information"
	}
	}
	}
	}
	],

view raw searchonlineopenaifunction.json hosted with ❤ by GitHub

In this function we are informing the LLM that if it needs to search the internet as part of providing the responses, it can call this function. The function name will be returned in the response and the relevant parameters will be provided as well.

Next, let's see how our orchestrator looks. I have added comments to each line where relevant:

	static async Task Main(string[] args)
	{
	var openaiApiKey = "<azure-openi-key>";
	var openaiEndpoint = "<azure-openi-endpoint>";
	var modelDeploymentName = "gpt-35-turbo"; //azure deployment name
	var bingAPIKey = "<bing-api-key>";
	var credential = new AzureKeyCredential(openaiApiKey);
	var openAIClient = new OpenAIClient(new Uri(openaiEndpoint), credential);
	//get userQuestion from the console
	Console.ForegroundColor = ConsoleColor.Yellow;
	Console.WriteLine("Ask a question to the internet: ");
	Console.ResetColor();
	string userQuestion = Console.ReadLine();
	//1. Call Open AI Chat API with the user's question.
	Response<ChatCompletions> result = await CallChatGPT(userQuestion, modelDeploymentName, openAIClient);
	//2. Check if the Chat API decided that for answering the question, a function call to Bing needs to be made.
	var functionCall = result.Value.Choices[0].Message.FunctionCall;
	if (functionCall != null)
	{
	Console.ForegroundColor = ConsoleColor.Green;
	Console.WriteLine($"Function Name: {functionCall.Name}, Params: {functionCall.Arguments}");
	Console.ResetColor();
	if (functionCall.Name == "search_online")
	{
	//3. If the Bing function call needs to be made, the Chat API will also provide which parameters need to be passed to the function.
	var searchOnlineParams = JsonSerializer.Deserialize<SearchOnlineParams>(functionCall.Arguments);
	Console.WriteLine($"Search Query: {searchOnlineParams.searchQuery}");
	//4. Call Bing with the parameters provided by the Chat API
	var searchResults = await SearchAsync(bingAPIKey, searchOnlineParams.searchQuery, 3);
	//5. Call the Chat API again with the function response.
	var functionMessages = new List<ChatMessage>();
	functionMessages.Add(new ChatMessage(ChatRole.Assistant, functionCall.Arguments) { Name = functionCall.Name });
	functionMessages.Add(new ChatMessage(ChatRole.Function, searchResults) { Name = functionCall.Name });
	result = await CallChatGPT(userQuestion, modelDeploymentName, openAIClient, functionMessages);
	//6. Print the final response from the Chat API.
	Console.WriteLine("------------------");
	//Write in yellow color
	Console.ForegroundColor = ConsoleColor.Yellow;
	Console.WriteLine(result.Value.Choices[0].Message.Content);
	Console.ResetColor();
	}
	}
	else
	{
	Console.WriteLine(result.Value.Choices[0].Message.Content);
	}
	}

view raw openaifuncsearchmain.cs hosted with ❤ by GitHub

This code is responsible for handling the chat with OpenAI, calling the Bing API and also responding back to the user based on the response from internet search. 

Next, let's have a look at the code which calls the Bing API based on the parameters provided by the LLM. Before executing this code, you will need to have created Bing Web Search API resource in Azure. Here is more information on it: https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/overview

The Bing Web Search API key can be found in the "Keys and Endpoint" section on the Azure resource:

Here is the code for calling the Bing Search API:

	public static async Task<string> SearchAsync(string apiKey, string query, int count = 1, int offset = 0, CancellationToken cancellationToken = default)
	{
	if (count <= 0) { throw new ArgumentOutOfRangeException(nameof(count)); }
	if (count >= 50) { throw new ArgumentOutOfRangeException(nameof(count), $"{nameof(count)} value must be less than 50."); }
	if (offset < 0) { throw new ArgumentOutOfRangeException(nameof(offset)); }
	Uri uri = new($"https://api.bing.microsoft.com/v7.0/search?q={Uri.EscapeDataString(query)}&count={count}&offset={offset}");
	using HttpResponseMessage response = await SendGetRequest(apiKey, uri, cancellationToken).ConfigureAwait(false);
	response.EnsureSuccessStatusCode();
	string json = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
	BingSearchResponse? data = JsonSerializer.Deserialize<BingSearchResponse>(json);
	WebPage[]? results = data?.WebPages?.Value;
	string pagesContent = string.Empty;
	foreach (var result in results)
	{
	//To keep this simple for the sample code, we are only going to return the search results snippet.
	//In production, ideally you will need to get the result.Url property of the search result and then get the content of the page using the Url.
	pagesContent += result.Snippet;
	}
	return pagesContent;
	}
	//For production scenarious, please re-use httpclient as shown here: https://www.aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
	private static async Task<HttpResponseMessage> SendGetRequest(string apiKey, Uri uri, CancellationToken cancellationToken = default)
	{
	using var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, uri);
	httpRequestMessage.Headers.Add("Ocp-Apim-Subscription-Key", apiKey);
	return await new HttpClient().SendAsync(httpRequestMessage, cancellationToken).ConfigureAwait(false);
	}

view raw searchonline.cs hosted with ❤ by GitHub

In this code, we are calling the Bing Web Search REST API to get results based on the search query created by the LLM. Once the top 3 results are fetched we are getting the text snippets of those results, combining them and sending it back the LLM. 

We are only getting the search result snippets to keep this demo simple. In production, ideally you will need to get the Url of each search result and then get the content of the page using the Url.

Finally, lets have a look at our CallChatGPT function which is responsible for talking to the Open AI chat API:

	private static async Task<Response<ChatCompletions>> CallChatGPT(string userQuestion, string modelDeploymentName, OpenAIClient openAIClient, IList<ChatMessage> functionMessages = null)
	{
	var chatCompletionOptions = new ChatCompletionsOptions();
	chatCompletionOptions.Messages.Add(new ChatMessage(ChatRole.System, $"You are a helpful assistant. If you don't know the answer to a question, you can look it up online. Today's date in UTC is {DateTime.UtcNow}"));
	chatCompletionOptions.Messages.Add(new ChatMessage(ChatRole.User, userQuestion));
	if (functionMessages != null)
	{
	foreach (var functionMessage in functionMessages)
	{
	chatCompletionOptions.Messages.Add(functionMessage);
	}
	}
	chatCompletionOptions.Functions.Add(new FunctionDefinition()
	{
	Name = "search_online",
	Description = "Search online if the requested information cannot be found in the language model or the information could be present in a time after the language model was trained",
	Parameters = BinaryData.FromString("{\"type\": \"object\",\"required\": [\"searchQuery\"],\"properties\": {\"searchQuery\": {\"type\": \"string\",\"description\": \"the text to search online to get the necessary information\"}}}\r\n")
	});
	chatCompletionOptions.Temperature = 0;
	var result = await openAIClient.GetChatCompletionsAsync(modelDeploymentName, chatCompletionOptions);
	return result;
	}

view raw openaisearchcallchatgpt.cs hosted with ❤ by GitHub

This code defines the OpenAI function which will be included in our Chat API calls. Also, the user's question is sent to the Chat API to determine if the function needs to be called. This function is also called again after the response from the Bing Web Search API is fetched. At that time, this function contains the search results and uses them to generate an output in natural language.

This way, we can use Open AI function calling together with Bing Web Search API to connect our chat bot to the internet!

Chat with your user directory using OpenAI functions and Microsoft Graph

Ever since OpenAI function calling was released, I have been incredibly fascinated by it. To me, it is as big a game changer as ChatGPT itself. 

With function calling, we are no longer limited by the data which was used to train the Large Language Model (LLM). We can call out to external APIs, protected company data and other business specific APIs and use the data to supplement the responses from the LLM. 

To know more about function calling specifically with the Azure OpenAI service, check out the Microsoft docs: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/function-calling

In this post, let's have a look at how we can leverage OpenAI function calling to chat with our user directory and search for users in natural language. To make this possible we will use the Microsoft Graph to do the heavy lifting. 

This is what we want to achieve: The user asks a question about the people directory in natural language, the LLM is able to transform the question to code which the Microsoft Graph understands and the LLM is again able to transform the response from the Microsoft Graph back to natural language.

On a high level, our approach can be summarised as follows:

1. Define the OpenAI functions and make them available to the LLM 

2. During the course of the chat, if the LLM thinks that to respond to the user, it needs to call our function, it will respond with the function name along with the parameters to be sent to function.

3. Call the Microsoft Graph user search API based on the parameters provided by the LLM.

4. Send the results returned from the Microsoft Graph back to the LLM to generate a response in natural language.

Alright, let's now look at the code. In this code sample I have used the following nuget packages:

https://www.nuget.org/packages/Azure.AI.OpenAI/1.0.0-beta.6/

https://www.nuget.org/packages/Microsoft.Graph/5.30.0/

https://www.nuget.org/packages/Azure.Identity/1.10.2/

The very first thing we will look at is our function definition:

	"functions": [
	{
	"name": "msgraph_search_users",
	"description": "Call the Microsoft Graph API to search users based on user attributes",
	"parameters": {
	"type": "object",
	"required": [
	"officeLocation", "department", "jobTitle"
	],
	"properties": {
	"officeLocation": {
	"type": "string",
	"enum": ["London", "Mumbai", "New York"],
	"description": "The user's location"
	},
	"department": {
	"type": "string",
	"enum": ["Engineering", "Product Management", "Executive"],
	"description": "The user's department"
	},
	"jobTitle": {
	"type": "string",
	"description": "The user's job title"
	}
	}
	}
	}
	],

view raw msgraphopenaifunction.json hosted with ❤ by GitHub

In this function we are informing the LLM that if needs to search any users as part of providing the responses, it can call this function. The function name will be returned in the response and the relevant parameters will be provided as well. 

The enums in the officeLocation and department parameter will instruct the LLM to only return those values even if user asks a slightly different variation in their question. We can see an example of this in the gif above. Even if the question asked contains words like "devs" and "NY", the LLM is able to determine and use the terms "Engineering" and "New York" instead.

Next, let's see how our orchestrator looks. I have added comments to each line where relevant:

	static async Task Main(string[] args)
	{
	var openaiApiKey = "<azure-openi-key>";
	var openaiEndpoint = "<azure-openi-endpoint>";
	var modelDeploymentName = "gpt-35-turbo"; //azure deployment name
	var credential = new AzureKeyCredential(openaiApiKey);
	var openAIClient = new OpenAIClient(new Uri(openaiEndpoint), credential);
	//get userQuestion from the console
	Console.WriteLine("Ask a question to your user directory: ");
	string userQuestion = Console.ReadLine();
	//1. Call Open AI Chat API with the user's question.
	Response<ChatCompletions> result = await CallChatGPT(userQuestion, modelDeploymentName, openAIClient);
	//2. Check if the Chat API decided that for answering the question, a function call to the MS Graph needs to be made.
	var functionCall = result.Value.Choices[0].Message.FunctionCall;
	if (functionCall != null)
	{
	Console.WriteLine($"Function Name: {functionCall.Name}, Params: {functionCall.Arguments}");
	if (functionCall.Name == "msgraph_search_users")
	{
	//3. If the MS Graph function call needs to be made, the Chat API will also provide which parameters need to be passed to the function.
	var userSearchParams = JsonSerializer.Deserialize<UserSearchParams>(functionCall.Arguments);
	//3. Call the MS Graph with the parameters provided by the Chat API
	var functionResponse = await CallMSGraph(userSearchParams);
	Console.WriteLine($"Graph Response: {functionResponse}");
	//4. Call the Chat API again with the function response.
	var functionMessages = new List<ChatMessage>();
	functionMessages.Add(new ChatMessage(ChatRole.Assistant, functionCall.Arguments) { Name = functionCall.Name });
	functionMessages.Add(new ChatMessage(ChatRole.Function, functionResponse) { Name = functionCall.Name });
	result = await CallChatGPT(userQuestion, modelDeploymentName, openAIClient, functionMessages);
	//5. Print the final response from the Chat API.
	Console.WriteLine("------------------");
	Console.WriteLine(result.Value.Choices[0].Message.Content);
	}
	}
	else
	{
	//If the LLM decided that a function call is not needed, print the final response from the Chat API.
	Console.WriteLine(result.Value.Choices[0].Message.Content);
	}
	}

view raw openaifuncmain.cs hosted with ❤ by GitHub

There is a lot to unpack here as this function is the one which does the heavy lifting. This code is responsible for handling the chat with OpenAI, calling the MS Graph and also responding back to the user based on the response from the Graph.

Next, let's have a look at the code which calls the Microsoft Graph based on the parameters provided by the LLM. 

Before executing this code, you will need to have created an App registration with a clientId and clientSecret. Here is how to do that: https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app

Since we are calling the Microsoft Graph /users endpoint with application permissions, the app registration will need a minimum of the User.Read.All application permission granted.
https://learn.microsoft.com/en-us/graph/api/user-list?view=graph-rest-1.0&tabs=http 

	private static async Task<string> CallMSGraph(UserSearchParams userSearchParams)
	{
	var tenantId = "<tenant-id>";
	var clientId = "<app-registration-clientid>";
	var clientSecret = "<app-registration-clientsecret>";
	var options = new TokenCredentialOptions { AuthorityHost = AzureAuthorityHosts.AzurePublicCloud };
	var clientSecretCredential = new ClientSecretCredential(tenantId, clientId, clientSecret, options);
	var graphClient = new GraphServiceClient(clientSecretCredential, new[] { "https://graph.microsoft.com/.default" });
	//add each property key and value of userSearchParams into a string
	var properties = userSearchParams.GetType().GetProperties();
	List<string> filterList = new List<string> { };
	foreach (var property in properties)
	{
	string value = property.GetValue(userSearchParams)?.ToString();
	if (!string.IsNullOrWhiteSpace(value))
	{
	filterList.Add($"\"{property.Name}:{value}\"");
	}
	}
	string filterString = string.Join(" AND ", filterList);
	var result = await graphClient.Users.GetAsync((requestConfiguration) =>
	{
	requestConfiguration.QueryParameters.Search = filterString;
	requestConfiguration.QueryParameters.Select = new string[] { "displayName" };
	requestConfiguration.Headers.Add("ConsistencyLevel", "eventual");
	});
	//for each user in result, get the display name and add it to a string
	List<string> userList = new List<string>();
	result?.Value?.ForEach(user => userList.Add(user.DisplayName));
	return string.Join(" , ", userList);
	}

view raw openaifunctiongraph.cs hosted with ❤ by GitHub

This code get the parameters sent from the LLM and uses the Microsoft Graph .NET SDK to call the /users/search endpoint and fetch the users based on the officeLocation, department or jobTitle properties.

Once the users are returned, their displayName value is concatenated into a string and returned to the orchestrator function so that it can be sent again to the LLM.

Finally, lets have a look at our CallChatGPT function which is responsible for talking to the Open AI chat api. 

	private static async Task<Response<ChatCompletions>> CallChatGPT(string userQuestion, string modelDeploymentName, OpenAIClient openAIClient, IList<ChatMessage> functionMessages = null)
	{
	var chatCompletionOptions = new ChatCompletionsOptions();
	chatCompletionOptions.Messages.Add(new ChatMessage(ChatRole.System, "You are a helpful assistant. Only use the functions and parameters you have been provided with."));
	chatCompletionOptions.Messages.Add(new ChatMessage(ChatRole.User, userQuestion));
	if (functionMessages != null)
	{
	foreach (var functionMessage in functionMessages)
	{
	chatCompletionOptions.Messages.Add(functionMessage);
	}
	}
	chatCompletionOptions.Functions.Add(new FunctionDefinition()
	{
	Name = "msgraph_search_users",
	Description = "Call the Microsoft Graph API to search users based on user attributes",
	Parameters = BinaryData.FromString("{\"type\":\"object\",\"required\":[\"officeLocation\",\"department\",\"jobTitle\"],\"properties\":{\"officeLocation\":{\"type\":\"string\",\"enum\":[\"London\",\"Mumbai\",\"New York\"],\"description\":\"The user's location\"},\"department\":{\"type\":\"string\",\"enum\":[\"Engineering\",\"Product Management\",\"Executive\"],\"description\":\"The user's department\"},\"jobTitle\":{\"type\":\"string\",\"description\":\"The user's job title\"}}}")
	});
	chatCompletionOptions.Temperature = 0;
	var result = await openAIClient.GetChatCompletionsAsync(modelDeploymentName, chatCompletionOptions);
	return result;
	}

view raw openaifunctioncallchatgpt.cs hosted with ❤ by GitHub

This function defines the Open AI function which will be included in our Chat API calls. Also, the user's question is sent to the API to determine if the function needs to be called. This function is also called again after the response from the Microsoft Graph is fetched. At that time, this function contains the details fetched from the Graph to generate an output in natural language. 

This way, we can use Open AI function calling together with Microsoft Graph API to chat with your user directory.