Building an Agent for Microsoft 365 Copilot: Adding an API plugin

In the previous post, we saw how to get started building agents for Microsoft 365 Copilot. We saw the different agent manifest files and how to configure them. There are some great out of the box capabilities available for agents like using Web search, Code Interpreter and Creating Images.

However, a very common scenario is to bring in data from external systems into Copilot and pass it to the LLM. So in this post, we will have a look at how to connect Copilot with external systems using API Plugins.

In simple terms, API Plugins are AI "wrappers" on existing APIs. We inform the LLM about an API and give it instructions on when to call these APIs and which parameters to pass to them. 

To connect your API to the Copilot agent, we have to create an "Action" and include it in the declarativeAgent.json file we saw in the previous post:

	{
	"$schema": "https://developer.microsoft.com/json-schemas/copilot/declarative-agent/v1.2/schema.json",
	"version": "v1.2",
	"name": "MyFirstAgent${{APP_NAME_SUFFIX}}",
	"description": "This declarative agent helps you with finding car repair records.",
	"instructions": "$[file('instruction.txt')]",
	"conversation_starters": [
	{
	"text": "Show repair records assigned to Karin Blair"
	}
	],
	"actions": [
	{
	"id": "repairPlugin",
	"file": "ai-plugin.json"
	}
	]
	}

view raw declarativeAgent.json hosted with ❤ by GitHub

The ai-plugin.json file contains information on how the LLM will understand our API. We will come to this file later.

Before that, let's understand how our API looks. We have a simple API that can retrieve some sample data about repairs:

	import {
	app,
	HttpRequest,
	HttpResponseInit,
	InvocationContext,
	} from "@azure/functions";
	import repairRecords from "../repairsData.json";
	/**
	* This function handles the HTTP request and returns the repair information.
	*
	* @param {HttpRequest} req - The HTTP request.
	* @param {InvocationContext} context - The Azure Functions context object.
	* @returns {Promise<Response>} - A promise that resolves with the HTTP response containing the repair information.
	*/
	export async function repairs(
	req: HttpRequest,
	context: InvocationContext
	): Promise<HttpResponseInit> {
	context.log("HTTP trigger function processed a request.");
	// Initialize response.
	const res: HttpResponseInit = {
	status: 200,
	jsonBody: {
	results: repairRecords,
	},
	};
	// Get the assignedTo query parameter.
	const assignedTo = req.query.get("assignedTo");
	// If the assignedTo query parameter is not provided, return the response.
	if (!assignedTo) {
	return res;
	}
	// Filter the repair information by the assignedTo query parameter.
	const repairs = repairRecords.filter((item) => {
	const fullName = item.assignedTo.toLowerCase();
	const query = assignedTo.trim().toLowerCase();
	const [firstName, lastName] = fullName.split(" ");
	return fullName === query || firstName === query || lastName === query;
	});
	// Return filtered repair records, or an empty array if no records were found.
	res.jsonBody.results = repairs ?? [];
	return res;
	}
	app.http("repairs", {
	methods: ["GET"],
	authLevel: "anonymous",
	handler: repairs,
	});

view raw api.ts hosted with ❤ by GitHub

Next, we will need the OpenAI specification for this API

	openapi: 3.0.0
	info:
	title: Repair Service
	description: A simple service to manage repairs
	version: 1.0.0
	servers:
	- url: ${{OPENAPI_SERVER_URL}}/api
	description: The repair api server
	paths:
	/repairs:
	get:
	operationId: listRepairs
	summary: List all repairs
	description: Returns a list of repairs with their details and images
	parameters:
	- name: assignedTo
	in: query
	description: Filter repairs by who they're assigned to
	schema:
	type: string
	required: false
	responses:
	'200':
	description: A list of repairs
	content:
	application/json:
	schema:
	type: object
	properties:
	results:
	type: array
	items:
	type: object
	properties:
	id:
	type: string
	description: The unique identifier of the repair
	title:
	type: string
	description: The short summary of the repair
	description:
	type: string
	description: The detailed description of the repair
	assignedTo:
	type: string
	description: The user who is responsible for the repair
	date:
	type: string
	format: date-time
	description: The date and time when the repair is scheduled or completed
	image:
	type: string
	format: uri
	description: The URL of the image of the item to be repaired or the repair process

view raw openapi.yml hosted with ❤ by GitHub

The OpenAPI specification is important as it describes in detail the exact signatures of our APIs to the LLM.

Finally, we will come to the most important file which is the ai-plugin.json file. It does several things. It informs Copilot which API methods are available, which parameters they expect and when to call them. This is done in simple human readable language so that the LLM can best understand it.

Additionally, it also handles formatting of the data before it's shown to the user in Copilot. Whether we want to use Adaptive Cards to show the data or if we want to run any other processing like removing direct links from the responses.

If you are doing plugin development, chances are that you will be spending most of your time on this file.

	{
	"$schema": "https://developer.microsoft.com/json-schemas/copilot/plugin/v2.2/schema.json",
	"schema_version": "v2.2",
	"namespace": "repairs",
	"name_for_human": "MyFirstAgent${{APP_NAME_SUFFIX}}",
	"description_for_human": "Track your repair records",
	"description_for_model": "Plugin for searching a repair list, you can search by who's assigned to the repair.",
	"functions": [
	{
	"name": "listRepairs",
	"description": "Returns a list of repairs with their details and images",
	"capabilities": {
	"response_semantics": {
	"data_path": "$.results",
	"properties": {
	"title": "$.title",
	"subtitle": "$.description"
	},
	"static_template": {
	"type": "AdaptiveCard",
	"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
	"version": "1.5",
	"body": [
	{
	"type": "Container",
	"$data": "${$root}",
	"items": [
	{
	"type": "TextBlock",
	"text": "id: ${if(id, id, 'N/A')}",
	"wrap": true
	},
	{
	"type": "TextBlock",
	"text": "title: ${if(title, title, 'N/A')}",
	"wrap": true
	},
	{
	"type": "TextBlock",
	"text": "description: ${if(description, description, 'N/A')}",
	"wrap": true
	},
	{
	"type": "TextBlock",
	"text": "assignedTo: ${if(assignedTo, assignedTo, 'N/A')}",
	"wrap": true
	},
	{
	"type": "TextBlock",
	"text": "date: ${if(date, date, 'N/A')}",
	"wrap": true
	},
	{
	"type": "Image",
	"url": "${image}",
	"$when": "${image != null}"
	}
	]
	}
	]
	}
	}
	},
	"states": {
	"reasoning": {
	"description": "`listRepairs` returns a list of repairs with their details and images",
	"instructions": [
	"When generating an answer based on the list of repairs, do not show any direct links or `Read More` links in the text."
	]
	},
	"responding": {
	"description": "`listRepairs` will return 200 Success and the list of repair items. It will return an error otherwise.",
	"instructions": [
	"If no error is returned, show a message to the user that the repairs could not be fetched."
	]
	}
	}
	}
	],
	"runtimes": [
	{
	"type": "OpenApi",
	"auth": {
	"type": "None"
	},
	"spec": {
	"url": "apiSpecificationFile/repair.yml",
	"progress_style": "ShowUsageWithInputAndOutput"
	},
	"run_for_functions": ["listRepairs"]
	}
	]
	}

view raw ai-plugin.json hosted with ❤ by GitHub

Once everything is in place, we will run our plugin with simple instructions and this is how it will fetch the data from external database

Hope this helps!

Building an Agent for Microsoft 365 Copilot Chat

Microsoft 365 Copilot Chat is a powerful, general-purpose assistant that greatly improves personal productivity, helping users manage tasks like emails, calendars, document searches, and more.

However, the true potential of M365 Copilot lies in its extensibility. By building specialized "vertical" agents on top of M365 Copilot, you can unlock team productivity as well as automate business processes. These custom agents not only help in individual productivity, but also help in building workflows across groups of people.

Agents for Microsoft 365 Copilot leverage the same robust foundation—its orchestrator, foundation models, and trusted AI services—that powers M365 Copilot itself. This ensures consistency, reliability, and security at scale.

So in this post, let's take a look at how to build Agents on top of Microsoft 365 Copilot. 

We will be building a Declarative Agent with the help of the Teams toolkit. Before we start, we need the following prerequisites:

A Microsoft 365 Copilot license

Teams Toolkit Visual Studio Code extension

Enable side loading of Teams apps

Once everything is in place, we will go to Visual Studio Code 

In the Teams Toolkit extension, To create an agent, we will click on "Create new app"

Then, click on "Agent"

When the agent is created, we see a bunch of files getting created as part of the scaffolding.  So let's take a look the difference moving pieces of the agent:

manifest.json

If you have been doing M365 Apps (and Teams apps) for a while, you are familiar with this file. This is the file which represents our Agent in the M365 App catalog. It contains the various details like name, description and capabilities of the app.

	{
	"$schema": "https://developer.microsoft.com/json-schemas/teams/v1.19/MicrosoftTeams.schema.json",
	"manifestVersion": "1.19",
	"version": "1.0.0",
	"id": "${{TEAMS_APP_ID}}",
	"icons": {
	"color": "color.png",
	"outline": "outline.png"
	},
	"name": {
	"short": "MyFirstAgent${{APP_NAME_SUFFIX}}",
	"full": "Full name for MyFirstAgent"
	},
	"description": {
	"short": "Short description for MyFirstAgent",
	"full": "Full description for MyFirstAgent"
	},
	"accentColor": "#FFFFFF",
	"composeExtensions": [],
	"permissions": [
	"identity",
	"messageTeamMembers"
	],
	"copilotAgents": {
	"declarativeAgents": [
	{
	"id": "declarativeAgent",
	"file": "declarativeAgent.json"
	}
	]
	},
	"validDomains": []
	}

view raw m365copilotagentmanifest.json hosted with ❤ by GitHub

However, you will notice the new property in this file which is "copilotAgents" This property will be pointing to a file containing the description of our new declarative agent. So let's look at how that file looks next:

declarativeAgent.json

	{
	"$schema": "https://developer.microsoft.com/json-schemas/copilot/declarative-agent/v1.2/schema.json",
	"version": "v1.2",
	"name": "MyFirstAgent",
	"description": "Declarative agent created with Teams Toolkit",
	"instructions": "$[file('instruction.txt')]",
	"conversation_starters": [
	{
	"title": "Hi",
	"text": "Hello, how are you?"
	},
	{
	"title": "Surprise me",
	"text": "Tell me an interesting fact"
	}
	],
	"actions": [
	{
	"id": "myPlugin",
	"file": "ai-plugin.json"
	}
	],
	"capabilities": [
	{
	"name": "WebSearch"
	},
	{
	"name": "GraphicArt"
	},
	{
	"name": "CodeInterpreter"
	}
	]
	}

view raw m365copilotagentdescription.json hosted with ❤ by GitHub

Lots of interesting things are happening here. 

 

First the "instructions" property is pointing to a file which will contain the "System prompt" of our agent. We will have a look at this file later. 

Next, the "conversation_starters" property contains ready to go prompts which the user can ask the agent. Our agent will be trained to respond to these prompts. This is so that the user is properly onboarded when they land on our agent.

Finally there will be the actions and capabilities properties: 

Actions property contains connections to external APIs which the agent can invoke.

Capabilities property contains the different out of the box "Tools" which we want to allow in our agent. E.g. SharePoint and OneDrive search, image creation, code interpreter to generate charts etc 

We will talk about both these properties in more details in subsequent blog posts.

instruction.txt

And finally we have the instructions file where we can specify the system prompt for the agent. Here, we can guide the agent and assign it personality. We can make it aware of the tools and capabilities it has available and when the use them. We can provide one shot or few shot examples to "train" the agent on responding to users.

You are a declarative agent and were created with Team Toolkit. You should start every response and answer to the user with "Thanks for using Teams Toolkit to create your declarative agent!\n\n" and then answer the questions and help the user.

view raw instruction.txt hosted with ❤ by GitHub

Once all files are in place, you can click "Provision" from the Teams toolkit extension:

Then navigate to https://www.microsoft365.com/chat

And our new agent will be ready!

Here is how the agent will provide the conversation starters when we first lang on it:

Simple conversation using Web search capability:

Conversation using Web search and Code Interpreter capabilities:

Hope this was helpful! We will explore M365 Copilot Agents more in subsequent blog posts.