Working with OpenAI Assistants: Using code interpreter to generate charts

This is the fourth post in the series where we explore the OpenAI Assistants API. In this post, we will be looking at the code interpreter tool which allows us to generate charts based on some data. This is very powerful for scenarios where you have to do data analysis on JSON, csv or Microsoft Excel files and generate charts and reports based on them.

See the following posts for the entire series:

Working with the OpenAI Assistants API: Create a simple assistant

Working with the OpenAI Assistants API: Using file search 

Working with the OpenAI Assistants API: Chat with Excel files using Code interpreter 

Working with the OpenAI Assistants API: Using code interpreter to generate charts (this post) 

The Code Interpreter tool has access to a sandboxed python code execution environment within the Assistants API. This can provide very useful as the Assistants API can iteratively run code against the files provided to it and generate charts!

So in this post, let's see how we can generate charts based on an excel file with the code interpreter tool. The excel file we will be querying will be the same one we used in the last post. It contains details of customers like their name and the licenses purchased of a fictional product by them:

To generate charts using the Code interpreter, we have to use the following moving pieces: 

• First, we need to upload the excel file using the Open AI File client 
• Then, we need to connect the uploaded file to the Code Interpreter tool in either an assistant or a thread which would enable the assistant to generate a chart on the document.

For the demo code, we will be using the Azure OpenAI service for working with the OpenAI gpt-4o model and since we will be using .NET code, we will need the Azure OpenAI .NET SDK as well as Azure.AI.OpenAI.Assistants nuget packages.

	var azureClient = new AzureOpenAIClient(new Uri(endpoint), new ApiKeyCredential(key));
	OpenAIFileClient fileClient = azureClient.GetOpenAIFileClient();
	AssistantClient assistantClient = azureClient.GetAssistantClient();
	OpenAIFile infoFile = await fileClient.UploadFileAsync("C:\\Users\\vardh\\Documents\\Customers.xlsx", FileUploadPurpose.Assistants);
	AssistantCreationOptions assistantOptions = new()
	{
	Name = "CodeInterpreterProMAX",
	Instructions =
	"You are an assistant that looks up sales data and helps visualize the information based"
	+ " on user queries. When asked to generate a graph, chart, or other visualization, use"
	+ " the code interpreter tool to do so.",
	Tools =
	{
	new CodeInterpreterToolDefinition()
	},
	ToolResources = new()
	{
	CodeInterpreter = new()
	{
	FileIds = { infoFile.Id },
	}
	}
	};
	Assistant assistant = assistantClient.CreateAssistant(deploymentName, assistantOptions);
	ThreadCreationOptions threadOptions = new()
	{
	InitialMessages = { "Can you plot a bar graph for all customers and their purchases?" },
	};
	ThreadRun threadRun = assistantClient.CreateThreadAndRun(assistant.Id, threadOptions);
	do
	{
	Thread.Sleep(TimeSpan.FromSeconds(1));
	Console.WriteLine($"Thread run status: {threadRun.Status}");
	threadRun = assistantClient.GetRun(threadRun.ThreadId, threadRun.Id);
	} while (!threadRun.Status.IsTerminal);
	CollectionResult<ThreadMessage> messages = assistantClient.GetMessages(threadRun.ThreadId, new MessageCollectionOptions() { Order = MessageCollectionOrder.Ascending });
	foreach (ThreadMessage message in messages)
	{
	Console.Write($"[{message.Role.ToString().ToUpper()}]: ");
	foreach (MessageContent contentItem in message.Content)
	{
	if (!string.IsNullOrEmpty(contentItem.Text))
	{
	Console.WriteLine($"{contentItem.Text}");
	}
	if (!string.IsNullOrEmpty(contentItem.ImageFileId))
	{
	OpenAIFile imageInfo = fileClient.GetFile(contentItem.ImageFileId);
	BinaryData imageBytes = fileClient.DownloadFile(contentItem.ImageFileId);
	using FileStream stream = File.OpenWrite($"{imageInfo.Filename}.png");
	imageBytes.ToStream().CopyTo(stream);
	Console.WriteLine($"<image: {imageInfo.Filename}.png>");
	}
	}
	}

view raw openai-assistant-code-interpreter-charts.cs hosted with ❤ by GitHub

[USER]: Can you plot a bar graph for all customers and their purchases?

[ASSISTANT] : Sure, let's first inspect the contents of the uploaded file to understand its structure and extract the necessary data.

[ASSISTANT] : The dataset contains the following columns:

• Customer Name
• Email
• Licenses Purchased
• Total Amount Paid

To create a bar graph of all customers and their purchases, we will use the "Customer Name" and "Licenses Purchased" columns.

Let's proceed with creating the bar graph.

[ASSISTANT] : <image: ea8db0b2-2f7a-420e-9c77-c081b7bd0132.png>

Here is the bar graph showing the number of licenses purchased by each customer. If you need any further analysis or additional visualizations, please let me know!

view raw output.md hosted with ❤ by GitHub

And this is the file generated by the code interpreter tool:

As you can see the code interpreter tool takes a few passes at the data. It tries to understand the document before generating the chart. This is a really powerful feature and the possibilities are endless! 

Hope this helps.

Working with OpenAI Assistants: Chat with Excel files using Code interpreter

This is the third post in the series where we explore the OpenAI Assistants API. In this post, we will be looking at the code interpreter tool which allows us to upload files to the Assistants API and write python code against them. This is very powerful for scenarios where you have to do data analysis on csv or Microsoft Excel files and generate charts and reports on them.

See the following posts for the entire series:

Working with the OpenAI Assistants: Create a simple assistant

Working with the OpenAI Assistants: Using file search 

Working with the OpenAI Assistants: Chat with Excel files using code interpreter (this post) 

Working with OpenAI Assistants: Using code interpreter to generate charts

The Retrieval Augmented Generation (RAG) pattern, which was discussed in previous posts, works great for text based files like Microsoft Word and PDF documents. However, when it comes to structured data files like csv or excel, it comes out short. An this where the Code Interpreter tool can come in very handy. It can repetitively run python code on documents until it is confident that the user's question has been answered.

So in this post, let's see how we can query an excel file with the code interpreter tool. The excel file we will be querying will contain details of customers like their name and the licenses purchased of a fictional product by them:

To upload and analyse documents using the Code interpreter, we have to use the following moving pieces: 

• First, we need to upload files using the Open AI File client 
• Then, we need to connect the uploaded file to the Code Interpreter tool in either an assistant or a thread which would enable the assistant to answer questions based on the document.

For the demo code, we will be using the Azure OpenAI service for working with the OpenAI gpt-4o model and since we will be using .NET code, we will need the Azure OpenAI .NET SDK as well as Azure.AI.OpenAI.Assistants nuget packages.

	var azureClient = new AzureOpenAIClient(new Uri(endpoint), new ApiKeyCredential(key));
	OpenAIFileClient fileClient = azureClient.GetOpenAIFileClient();
	AssistantClient assistantClient = azureClient.GetAssistantClient();
	OpenAIFile infoFile = await fileClient.UploadFileAsync("C:\\Customers.xlsx", FileUploadPurpose.Assistants);
	AssistantCreationOptions assistantOptions = new()
	{
	Name = "CodeInterpreterProMAX",
	Instructions =
	"You are an assistant that looks up sales data and helps visualize the information based"
	+ " on user queries. When asked to generate a graph, chart, or other visualization, use"
	+ " the code interpreter tool to do so.",
	Tools =
	{
	new CodeInterpreterToolDefinition()
	},
	ToolResources = new()
	{
	CodeInterpreter = new()
	{
	FileIds = { infoFile.Id },
	}
	}
	};
	Assistant assistant = assistantClient.CreateAssistant(deploymentName, assistantOptions);
	ThreadCreationOptions threadOptions = new()
	{
	InitialMessages = { "Which customer has purchased the most licenses?" },
	};
	ThreadRun threadRun = assistantClient.CreateThreadAndRun(assistant.Id, threadOptions);
	do
	{
	Thread.Sleep(TimeSpan.FromSeconds(1));
	Console.WriteLine($"Thread run status: {threadRun.Status}");
	threadRun = assistantClient.GetRun(threadRun.ThreadId, threadRun.Id);
	} while (!threadRun.Status.IsTerminal);
	CollectionResult<ThreadMessage> messages = assistantClient.GetMessages(threadRun.ThreadId, new MessageCollectionOptions() { Order = MessageCollectionOrder.Ascending });
	foreach (ThreadMessage message in messages)
	{
	Console.Write($"[{message.Role.ToString().ToUpper()}]: ");
	foreach (MessageContent contentItem in message.Content)
	{
	if (!string.IsNullOrEmpty(contentItem.Text))
	{
	Console.WriteLine($"{contentItem.Text}");
	}
	}
	}

view raw openai-assistant-code-interpreter.cs hosted with ❤ by GitHub

[USER]: Which customer has purchased the most licenses?

[ASSISTANT]: Let's first take a look at the content of the uploaded file to understand its structure. I'll load the file and inspect the first few rows of data.

[ASSISTANT]: The data contains information about customers, including their names, email addresses, the number of licenses they purchased, and the total amount they paid.

To identify the customer who has purchased the most licenses, we will find the customer with the maximum value in the "Licenses Purchased" column.

[ASSISTANT]: The customer who has purchased the most licenses is Alice Johnson, with a total of 25 licenses purchased.

view raw output.md hosted with ❤ by GitHub

As you can see the code interpreter tool takes a few passes at the data. It tries to understand the document before answering the question. This is a really powerful feature and the possibilities are endless! 

Hope this helps.