Building an Agent for Microsoft 365 Copilot: Adding an API plugin

In the previous post, we saw how to get started building agents for Microsoft 365 Copilot. We saw the different agent manifest files and how to configure them. There are some great out of the box capabilities available for agents like using Web search, Code Interpreter and Creating Images.

However, a very common scenario is to bring in data from external systems into Copilot and pass it to the LLM. So in this post, we will have a look at how to connect Copilot with external systems using API Plugins.

In simple terms, API Plugins are AI "wrappers" on existing APIs. We inform the LLM about an API and give it instructions on when to call these APIs and which parameters to pass to them. 

To connect your API to the Copilot agent, we have to create an "Action" and include it in the declarativeAgent.json file we saw in the previous post:

	{
	"$schema": "https://developer.microsoft.com/json-schemas/copilot/declarative-agent/v1.2/schema.json",
	"version": "v1.2",
	"name": "MyFirstAgent${{APP_NAME_SUFFIX}}",
	"description": "This declarative agent helps you with finding car repair records.",
	"instructions": "$[file('instruction.txt')]",
	"conversation_starters": [
	{
	"text": "Show repair records assigned to Karin Blair"
	}
	],
	"actions": [
	{
	"id": "repairPlugin",
	"file": "ai-plugin.json"
	}
	]
	}

view raw declarativeAgent.json hosted with ❤ by GitHub

The ai-plugin.json file contains information on how the LLM will understand our API. We will come to this file later.

Before that, let's understand how our API looks. We have a simple API that can retrieve some sample data about repairs:

	import {
	app,
	HttpRequest,
	HttpResponseInit,
	InvocationContext,
	} from "@azure/functions";
	import repairRecords from "../repairsData.json";
	/**
	* This function handles the HTTP request and returns the repair information.
	*
	* @param {HttpRequest} req - The HTTP request.
	* @param {InvocationContext} context - The Azure Functions context object.
	* @returns {Promise<Response>} - A promise that resolves with the HTTP response containing the repair information.
	*/
	export async function repairs(
	req: HttpRequest,
	context: InvocationContext
	): Promise<HttpResponseInit> {
	context.log("HTTP trigger function processed a request.");
	// Initialize response.
	const res: HttpResponseInit = {
	status: 200,
	jsonBody: {
	results: repairRecords,
	},
	};
	// Get the assignedTo query parameter.
	const assignedTo = req.query.get("assignedTo");
	// If the assignedTo query parameter is not provided, return the response.
	if (!assignedTo) {
	return res;
	}
	// Filter the repair information by the assignedTo query parameter.
	const repairs = repairRecords.filter((item) => {
	const fullName = item.assignedTo.toLowerCase();
	const query = assignedTo.trim().toLowerCase();
	const [firstName, lastName] = fullName.split(" ");
	return fullName === query || firstName === query || lastName === query;
	});
	// Return filtered repair records, or an empty array if no records were found.
	res.jsonBody.results = repairs ?? [];
	return res;
	}
	app.http("repairs", {
	methods: ["GET"],
	authLevel: "anonymous",
	handler: repairs,
	});

view raw api.ts hosted with ❤ by GitHub

Next, we will need the OpenAI specification for this API

	openapi: 3.0.0
	info:
	title: Repair Service
	description: A simple service to manage repairs
	version: 1.0.0
	servers:
	- url: ${{OPENAPI_SERVER_URL}}/api
	description: The repair api server
	paths:
	/repairs:
	get:
	operationId: listRepairs
	summary: List all repairs
	description: Returns a list of repairs with their details and images
	parameters:
	- name: assignedTo
	in: query
	description: Filter repairs by who they're assigned to
	schema:
	type: string
	required: false
	responses:
	'200':
	description: A list of repairs
	content:
	application/json:
	schema:
	type: object
	properties:
	results:
	type: array
	items:
	type: object
	properties:
	id:
	type: string
	description: The unique identifier of the repair
	title:
	type: string
	description: The short summary of the repair
	description:
	type: string
	description: The detailed description of the repair
	assignedTo:
	type: string
	description: The user who is responsible for the repair
	date:
	type: string
	format: date-time
	description: The date and time when the repair is scheduled or completed
	image:
	type: string
	format: uri
	description: The URL of the image of the item to be repaired or the repair process

view raw openapi.yml hosted with ❤ by GitHub

The OpenAPI specification is important as it describes in detail the exact signatures of our APIs to the LLM.

Finally, we will come to the most important file which is the ai-plugin.json file. It does several things. It informs Copilot which API methods are available, which parameters they expect and when to call them. This is done in simple human readable language so that the LLM can best understand it.

Additionally, it also handles formatting of the data before it's shown to the user in Copilot. Whether we want to use Adaptive Cards to show the data or if we want to run any other processing like removing direct links from the responses.

If you are doing plugin development, chances are that you will be spending most of your time on this file.

	{
	"$schema": "https://developer.microsoft.com/json-schemas/copilot/plugin/v2.2/schema.json",
	"schema_version": "v2.2",
	"namespace": "repairs",
	"name_for_human": "MyFirstAgent${{APP_NAME_SUFFIX}}",
	"description_for_human": "Track your repair records",
	"description_for_model": "Plugin for searching a repair list, you can search by who's assigned to the repair.",
	"functions": [
	{
	"name": "listRepairs",
	"description": "Returns a list of repairs with their details and images",
	"capabilities": {
	"response_semantics": {
	"data_path": "$.results",
	"properties": {
	"title": "$.title",
	"subtitle": "$.description"
	},
	"static_template": {
	"type": "AdaptiveCard",
	"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
	"version": "1.5",
	"body": [
	{
	"type": "Container",
	"$data": "${$root}",
	"items": [
	{
	"type": "TextBlock",
	"text": "id: ${if(id, id, 'N/A')}",
	"wrap": true
	},
	{
	"type": "TextBlock",
	"text": "title: ${if(title, title, 'N/A')}",
	"wrap": true
	},
	{
	"type": "TextBlock",
	"text": "description: ${if(description, description, 'N/A')}",
	"wrap": true
	},
	{
	"type": "TextBlock",
	"text": "assignedTo: ${if(assignedTo, assignedTo, 'N/A')}",
	"wrap": true
	},
	{
	"type": "TextBlock",
	"text": "date: ${if(date, date, 'N/A')}",
	"wrap": true
	},
	{
	"type": "Image",
	"url": "${image}",
	"$when": "${image != null}"
	}
	]
	}
	]
	}
	}
	},
	"states": {
	"reasoning": {
	"description": "`listRepairs` returns a list of repairs with their details and images",
	"instructions": [
	"When generating an answer based on the list of repairs, do not show any direct links or `Read More` links in the text."
	]
	},
	"responding": {
	"description": "`listRepairs` will return 200 Success and the list of repair items. It will return an error otherwise.",
	"instructions": [
	"If no error is returned, show a message to the user that the repairs could not be fetched."
	]
	}
	}
	}
	],
	"runtimes": [
	{
	"type": "OpenApi",
	"auth": {
	"type": "None"
	},
	"spec": {
	"url": "apiSpecificationFile/repair.yml",
	"progress_style": "ShowUsageWithInputAndOutput"
	},
	"run_for_functions": ["listRepairs"]
	}
	]
	}

view raw ai-plugin.json hosted with ❤ by GitHub

Once everything is in place, we will run our plugin with simple instructions and this is how it will fetch the data from external database

Hope this helps!