Troubleshooting a Copilot Studio agent before you reach for Application Insights

Before you start#

• Access to the environment the failure actually occurred in, which is not always the one the reporter was looking at.
• The Bot Transcript Viewer security role if you intend to read production transcripts. Environment Maker does not include it, and only an admin can grant it, usually at the point the agent is shared.
• Power Platform admin center access, or somebody who has it, for capacity and data policy questions.
• A conversation you can point at. One reproducible failure beats a description of five.

1. Capture the conversation ID before you touch anything else#

The conversation ID is the only join key the four layers share. Everything else is timestamps and guesswork, and timestamps stop working the moment two people are testing at once.

In the test pane, type this and read the answer back:

/debug conversationid

In production the same value surfaces in the transcript record’s Name field as {ConversationId}_{BotId}, in environment-level telemetry as gen_ai.conversation.id, in agent-level telemetry inside customDimensions, and in the Copilot Studio Kit Agent Debugger. Record the agent, environment and tenant IDs from Settings > Session details at the same time: you need all three for a support case and none are convenient to find later.

2. Place the symptom in one of nine failure signatures#

Before opening a tool, classify the symptom. The bucket decides the route, and the wrong route is the main cost in an investigation like this.

3. Confirm you are testing what you think you are testing#

This step catches an embarrassing share of reported bugs, and I skip it at my own expense roughly once a quarter.

1. Environment. Confirm the selector top-right matches the environment you mean. A published agent in DEV and a stale one in ACC look identical from a chat window.
2. Draft against published. The test pane runs draft content. Every channel runs the latest published version. If a fix works in test and not in Teams, start here.
3. The publish succeeded. A publish can report success and still leave the runtime on an older version. LatestPublishedVersionNotFound and AIPluginOperationNotFound both clear by republishing, particularly after a solution import.
4. Solution layers. In Power Apps, open Solutions > Solution layers on the bot component and look for an unmanaged layer above the managed one.
5. Connection references. After an import, confirm each is bound to a live connection owned by an identity that still exists and still holds a valid token.
6. Environment variables. Check current values, not default values.
7. Connected agents are published too. An orchestrator can be current while a sub-agent is not, which reports as ConnectedAgentBotNotPublished.

4. Clear the topic checker before you believe anything else#

For topic-authored content this is your compiler. Open Topics, read the Errors column, then open a topic and run Topic checker from the toolbar. Selecting an entry jumps to the offending node.

Errors block publication and warnings do not, which is exactly why warnings are how topics misbehave quietly at runtime. Clear both. For hand-edited YAML, open the code editor on the topic: InvalidContent and ContentError almost always trace back to YAML the visual designer accepted and the runtime rejected.

5. Turn the test pane into a debugger#

The test pane is an instrumented debugger rather than a chat window, but the instrumentation is off by default.

Configure it first#

Open Test, then the three dots. Show activity map when testing serves generative orchestration; Track between topics serves node-by-node execution inside a topic. The two fight each other, because with both on you collapse the map every turn. Use the map to debug planning and tool selection, tracking to debug flow inside a topic, and rarely both.

Read the activity map#

The map renders the plan the orchestrator generated for that turn, one node per step, with per-step duration and invalid parameters flagged inline. Expanding a node is where the answer usually is.

A knowledge node shows the rewritten search query, which frequently differs from what the user typed and on its own explains a good share of “why did it not find that”. It also lists the sources cited and, separately, the sources searched but not used. That second list separates a ranking problem from an access problem, and it is the most useful control on the page.

A tool node shows resolved inputs, raw outputs and duration. Show rationale explains why the orchestrator picked that tool, but Microsoft generates it on demand from agent metadata and says plainly that it might not be accurate. Treat it as a hypothesis generator, never as evidence.

Inspect variables and reasoning#

Test > Variables > Test expands global, environment, system and topic-scoped categories and mutates turn by turn. It catches a variable left blank when a downstream action expects a value, a coercion that produced a string where a number was expected, and a topic variable that did not survive a redirect.

Chain of thought renders before the response on selected reasoning models. Reach for it on behavioural defects, where the map shows a valid plan that should not have been chosen.

Selecting any agent response in the chat takes you to the node that produced it, and fired nodes carry a coloured checkmark. That is the fastest route from “the agent said something odd” to the node that said it.

6. Know what the test pane will not tell you#

The test pane is design-time validation and does not replicate published channel behaviour. Four limits are worth holding in mind.

Timer-based and background-triggered events, inactivity triggers included, might not fire here even when the agent is configured correctly. Event trigger payloads appear as messages only you can see. Reset clears conversation state and saving a topic does not, so stale state survives your edits. Manage connections in the same menu governs which connections the test chat uses, which is where a good share of “works for me” defects live.

7. Read the error code as a signal of who owns the fix#

The code tells you whose problem it is faster than the message does.

Two are misread often enough to name. OpenAIndirectAttack fires on instructions embedded in grounding data rather than in user input, so if you ingest third-party documents it is prompt-injection defence working correctly: read the source document before loosening the filter. ConversationStateTooLarge means conversation variables are accumulating, and the usual culprit is a large JSON payload parked in a variable to pass between topics. Fetch it in a flow at point of use instead.

8. Save a snapshot when the activity map is not enough#

Test > … > Save snapshot produces botContent.zip holding two files. dialog.json carries conversational diagnostics with detailed error descriptions, and it is the richest design-time error artefact available, containing detail the interface summarises away. botContent.yml carries the agent’s topics, entities and variables.

The archive contains all of your agent content and can include sensitive information. Attach it to a Microsoft support case, not to a forum post. It also loads into the Copilot Studio Kit Agent Debugger through Upload Snapshot, which gives you a structured view without Dataverse access.

9. Use the Activity page for sessions that are not yours#

The Activity page records activity in real time, your own test runs included, across test chat, Teams, Microsoft 365 Copilot, SharePoint and autonomous trigger runs. It needs generative orchestration, plus an Exchange licence and mailbox, because activity data lives in Microsoft 365 services under M365 residency terms rather than Azure terms. Raise that in an architecture review if you have residency constraints.

Filter by the status pills, then add Completed steps and Last step through Edit columns, the two that show where activities die. Map view is the only view exposing the Reasoning chevron.

Auth required and Rejected are the two most often reported as “the agent is broken”, and both are usually configuration.

10. Fix routing by rewriting descriptions, not structure#

Generative orchestration selects tools using names, descriptions and instructions. That is the entire routing surface, so when routing is wrong the fix is textual almost every time.

Every tool, topic and connected agent needs a description stating when to use it rather than what it is. “Gets order data” routes badly. “Use when the user asks about the status, contents or delivery date of an existing order” routes well. Overlapping descriptions produce non-deterministic selection, so make the wording mutually exclusive, and state precedence outright in the agent-level instructions: “always prefer X over Y when the user provides an order number”. Connected agents missing a description and instructions throw ConnectedAgentGptComponentNotFound. For classic intent matching, give trigger phrases five to ten lexically varied examples rather than ten rewordings of one sentence.

Multi-level chaining is not supported: an orchestrator can call sub-agents, and those sub-agents cannot call their own. Flatten the hierarchy.

11. Check tool inputs against the declared schema#

Compare the resolved inputs on the activity map node against the tool’s declared schema, then work down this list.

Schema drift is the top cause: if an agent flow’s inputs or outputs changed, the binding breaks and reports BindingKeyNotFoundError. Only Text, Boolean and Number are supported as agent flow parameter types, so a record or a table fails with FlowActionBadRequest. AsyncResponsePayloadTooLarge means the connector returned more than the agent can handle. Power Fx failures surface as ConnectorPowerFxError, so guard with IsBlank(), Coalesce() and IfError() rather than assuming a value arrived.

12. Separate an authentication failure from a permissions failure#

Work this in order, because steps four and five look identical from the chat window and have nothing in common.

1. Settings > Security > Authentication. Establish whether it is No authentication, Authenticate with Microsoft, or Authenticate manually. AuthenticationNotConfigured means a feature needs auth that is not set up.
2. Channel support. Not every channel supports integrated authentication, and the error names the channel when this is the cause.
3. Multi-agent parity. Orchestrator and connected agents must be compatible or you get ConnectedAgentAuthMismatch. If the connected agent has no auth, any orchestrator auth is fine. If it requires auth, the orchestrator must use the same method, and manual OAuth2 on the orchestrator is incompatible with connected agents that require auth.
4. Token lifecycle. HTTP401Unauthorized, InvalidAuthenticationToken and MsalUiException mean delete the connection, recreate it, reauthenticate and retry.
5. Permissions. HTTP403Forbidden means the identity is valid and the grant is not. Check app registration permissions, admin consent, resource-level sharing and environment scoping. The token is fine.
6. ConsentNotProvidedByUser means the user declined the SSO prompt. Not a defect.

13. Read the knowledge node before rewriting content#

When knowledge returns nothing or the wrong sources, resist the urge to rewrite documents. Confirm indexing has finished first, since freshly added sources return nothing for a period. Then open the knowledge node in the activity map and read three things: the rewritten query, the cited sources, and the sources searched but not used. Searched but unused means retrieval reached the source and ranked it low, which is a content and chunking problem. Absent entirely means access, scoping or indexing, which is not.

The error codes in section 24 split the rest: the 429 family is throttling, *SearchFailed is platform-side, and DataverseStructured401 is a permissions grant on the agent’s service principal. If answers are being suppressed rather than absent, adjust the content moderation level before rewriting anything.

14. Republish before blaming the channel#

Channels serve published content only, so republish after every change before investigating anything else. Verify the channel’s own configuration next: Azure Bot Service channel problems surface as 2016 for a missing or misconfigured channel and 2017 for an inaccessible one, which is usually auth. For custom canvas and Direct Line embeds, check the token endpoint, secret rotation and CORS separately from the agent. Rate limiting is channel-specific, and in Teams it appears as 2018 and 2100.

15. Learn when a transcript is written before you query it#

Transcripts are the authoritative record of production behaviour, and the storage model surprises people often enough that it is worth knowing before you go looking.

Access needs the Bot Transcript Viewer security role. Environment Maker does not grant it and only an admin can assign it.

16. Start with the session transcript CSV#

The fastest route to “which sessions went wrong, and roughly why” is the CSV, available for the last 29 days from Analytics, or Monitor in the new agent experience.

Filter to SessionOutcome == Abandoned with a low Turns count, then cluster InitialUserMessage. That set is your routing-gap backlog. High-turn abandons are a different failure, usually a loop or a question the agent cannot parse.

Three blind spots, all of which have cost me time. Options presented to the user are not captured. TopicName is the last topic rather than the path. And each agent response in ChatTranscript is truncated at 512 characters, per response rather than per session, so a long answer is silently cut and the cut is invisible unless you know to look.

17. Go to the Dataverse table for the full record#

In Power Apps, open Tables > All, search conversation, select ConversationTranscript, then Export > Export data.

Name is the correlation anchor: split on _ to recover the conversation ID for Application Insights and the bot ID for filtering. Content holds the transcript as JSON, Metadata holds BotId, BotName and BatchId, and ConversationStartTime is when the conversation began rather than when the record was written.

What is in the Content JSON#

Content is a raw activity log, and five keys per element carry most of the weight. valueType determines what the activity is telling you. replyToId points at the activity this one responds to, which is how you reconstruct causality. from.role is 0 for the agent and 1 for the user. value holds the payload specific to the value type. And channeldata holds DialogErrorDetail, which is the runtime error payload: when a production conversation failed and the user saw a generic message, the real reason is in there.

How I read one#

Sort by timestamp, split by from.role to rebuild the alternation, then find the first IntentRecognition and ask whether the right topic was chosen. If it was not, stop there: that is a routing defect and everything downstream is noise. Otherwise walk the VariableAssignment entries until one holds an unexpected value, which is the divergence point, and follow DialogRedirect to reconstruct the topic path. Read DialogErrorDetail on any failing turn and the terminal SessionInfo for the outcome.

Transcripts reference content by ID rather than display name, so a redirect names its destination topic as a GUID. Export an ID to name map from the botcomponent table once and keep it, or you will reverse-engineer the same GUIDs every time.

18. Turn on enhanced transcripts for node-level tracing#

Default transcripts tell you which topics ran. Enhanced transcripts tell you which nodes ran, under Settings > Advanced > Enhance Transcripts > Include node-level details in transcripts. Each node a topic invokes adds a nodeTraceData activity carrying nodeID, nodeType, startTime, endTime and topicDisplayName.

That answers both “which branch did this user take” and “which node was slow” from production data, since endTime minus startTime is an in-topic latency profile that costs no Application Insights configuration. Leave it on permanently in DEV. In production, weigh it against volume: node-level detail pushes records past the 1 MB threshold and materially increases splitting.

19. Query transcripts programmatically#

Anything beyond ad-hoc export means querying Dataverse directly. FetchXML for recent transcripts belonging to one agent:

<fetch top="50">
  <entity name="conversationtranscript">
    <attribute name="conversationtranscriptid" />
    <attribute name="name" />
    <attribute name="conversationstarttime" />
    <attribute name="content" />
    <attribute name="metadata" />
    <order attribute="conversationstarttime" descending="true" />
    <filter type="and">
      <condition attribute="bot_conversationtranscriptid"
                 operator="eq"
                 value="{YOUR-BOT-GUID}" />
      <condition attribute="conversationstarttime"
                 operator="last-x-days"
                 value="7" />
    </filter>
  </entity>
</fetch>

The Web API, when you have a conversation ID and want that one conversation:

GET https://{org}.crm{n}.dynamics.com/api/data/v9.2/conversationtranscripts
  ?$select=name,conversationstarttime,content,metadata
  &$filter=startswith(name,'{conversationId}')
  &$orderby=createdon asc
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0

Either way, reassemble by BatchId before parsing Content, or a long conversation parses as truncated JSON and you spend twenty minutes blaming the parser:

var query = new QueryExpression("conversationtranscript")
{
    ColumnSet = new ColumnSet("name", "content", "metadata", "conversationstarttime"),
    Criteria = new FilterExpression(LogicalOperator.And)
    {
        Conditions =
        {
            new ConditionExpression("bot_conversationtranscriptid",
                ConditionOperator.Equal, botId),
            new ConditionExpression("conversationstarttime",
                ConditionOperator.LastXDays, 7)
        }
    },
    Orders = { new OrderExpression("conversationstarttime", OrderType.Descending) }
};

var records = service.RetrieveMultiple(query).Entities;

// Records sharing a name and a start time are one conversation, split by size.
var conversations = records
    .GroupBy(e => (
        Name: e.GetAttributeValue<string>("name"),
        Start: e.GetAttributeValue<DateTime>("conversationstarttime")))
    .Select(g => string.Concat(g
        .OrderBy(e => JsonDocument
            .Parse(e.GetAttributeValue<string>("metadata"))
            .RootElement.GetProperty("BatchId").GetInt32())
        .Select(e => e.GetAttributeValue<string>("content"))));

20. Extend retention past thirty days#

A recurring bulk-delete job removes transcripts older than 30 days. To keep them longer, cancel that job and create your own. Power Apps to Settings > Advanced settings, then Settings > System > Data Management > Bulk Record Deletion, view Recurring Bulk Deletion System Jobs. Select Bulk Delete Conversation Transcript Records Older Than 1 Month and cancel it, then create a new job against ConversationTranscripts with ConversationStartTime older than the number of months you need and SchemaType equal to powervirtualagents, running daily.

Changing Dataverse retention affects custom analytics only; the Analytics page is unaffected. For genuinely long retention, export to ADLS Gen2 through Azure Synapse Link rather than hoarding in Dataverse, which is cheaper per GB and gives you a queryable lake. One trap: Synapse Link mirrors deletions by default, so the bulk-delete job strips records from the lake too unless you configure append-only writes.

21. Hand the telemetry layer over to Application Insights#

Layer four answers what the first three cannot: at what rate, with what latency, with what error distribution, correlated to which downstream system. It is the wrong tool for a single reproducible failure and the only tool for “how often has this been happening”. Choose the scope deliberately, and avoid sending both models to the same resource.

Classic topic-authored agents need agent-level, because environment-level does not emit topic events at all. Generative-orchestration agents with tools and sub-agents are far better served by environment-level. Teams running both end up with agent-level in DEV and INT, environment-level in ACC and PRO, pointing at separate resources.

Two operational settings neither companion piece covers. Ingestion sampling silently drops spans and breaks trace completeness, so check it before reading a truncated trace as a runtime failure. And set a daily cap with a cap alert on non-production resources, because node execution events plus conversation-detail logging on a busy agent produces a great deal of data. Alert on rate and ratio rather than raw counts: tool failure rate over fifteen minutes by tool name, P95 turn duration against your SLO, any rate-limit or quota code, and turn volume dropping to zero, which is the only dead-agent detection you get.

Where to click, and the queries themselves, are the subject of the two companion pieces on the Application Insights resource menu and on KQL for Copilot Studio telemetry. A third copy of either would disagree with them within a month, since the schema is still in preview.¹

22. Follow the call into Power Automate and the connectors#

Once telemetry says the failure is outside the agent, follow the call.

For agent flows, open Run history in Power Automate, match by timestamp and inputs, then expand the failing action and read the raw request and response. Three constraints account for most of what you find. The 100-second action limit produces FlowActionTimedOut, and the fix is moving post-response work after the Respond to the agent step, where it runs against the 30-day flow limit instead of the response window. Parameter types are limited to Text, Boolean and Number. FlowMakerConnectionBlocked means admin policy forbids maker credentials, so share the flow with run-only permissions.

For connectors, test the operation in isolation under the same identity before blaming the agent. On custom connectors, confirm the OpenAPI definition still matches the live API, because silent schema drift produces HTTP400BadRequest and HTTP422UnprocessableEntity. Check data policy too: a connector moved to a different data group produces DataLossPreventionViolation at runtime with no design-time warning.

If your tools call Azure Functions, APIM or a custom API, instrument those with the same Application Insights resource. operation_Id then flows through, and the end-to-end transaction view stitches agent turn, tool call, your service and its dependencies into one waterfall. For a solution with custom backends that is the highest-value observability work available, and it costs a connection string.

Capacity is not a defect. EnforcementMessageC2 and the *RateLimitReached family are capacity signals, resolved in PPAC with prepaid capacity or pay-as-you-go, after which chat typically resumes within about five minutes.

23. Look up a classic numeric error code#

24. Look up a named error code by owner#

Your content and configuration:

Authentication and authorisation, all triaged in the order given in section 12:

Multi-agent orchestration:

Tool and HTTP failures, with the owner in the middle column, which is the column that decides what you do next:

Knowledge, Responsible AI and capacity:

25. Run the playbook that matches your signature#

Condensed routes for the signatures from section 2, in order, stopping at the first one that explains the symptom.

Two of these are worth a note. On B, one misroute is an anecdote: count IntentRecognition activities across transcripts before rewriting anything. On G, delegated auth explains most reports in that bucket, so rule it out before the rest.

26. Collect the escalation package before you open a case#

What shortens a support round trip is not the description of the failure. It is the list of things you have already ruled out, stated with what each layer showed.

• Identity. Tenant ID, environment ID and agent ID from Settings > Session details, plus the region of the environment.
• The failing interaction. Conversation ID, the exact utterance and the exact response, the timestamp in UTC with the timezone stated, the channel, the error code and its full text, and whether the user was authenticated and by which method.
• Artefacts. The botContent.zip snapshot marked as containing sensitive content, the relevant transcript records reassembled by BatchId, the telemetry query output with operation_Id, and the flow run URL if a flow is involved.
• Scope. First observed and last known good, whether it reproduces and at what rate, all users or a subset, all channels or one, any deployment or policy change in the window, and whether it reproduces in both the test pane and a published channel.

Test > … > Flag an issue sends the conversation ID to Microsoft, and works alongside a formal case rather than instead of one.

27. Configure observability before you need it#

Retrofitting observability during an incident is how a one-hour investigation becomes a one-day one. This is the list I work through before an agent carries real traffic.

• An Application Insights resource per environment, with separate resources for agent-level and environment-level.
• The telemetry scope chosen and written down, along with why.
• Local authentication enabled on the resource if you are using environment-level export.
• Logging toggles set per environment: conversation details on in DEV and INT, privacy-assessed before production.
• Enhanced transcripts on in DEV, with a recorded decision for production.
• The Bot Transcript Viewer role assigned to whoever will be on call, not only to you.
• Retention extended if 30 days is not enough, and Synapse Link configured with append-only writes for anything longer.
• Alerts on tool failure rate, P95 latency, rate-limit codes and zero traffic, plus a daily ingestion cap on non-production resources.
• Downstream services instrumented into the same workspace so operation_Id correlates.
• Release pipeline validation of solution layers, connection references and environment variable current values.

What else will bite you#

The layer boundaries are less clean than the model suggests, and the seams are where time goes. Activity data lives in Microsoft 365 services under M365 residency terms while transcripts live in Dataverse and telemetry lives in Azure, so a single conversation is subject to three different retention regimes and three different access-control models. Nobody discovers this at design time. They discover it when legal asks where the data is.

The other recurring cost is treating a preview surface as a stable one. Environment-level telemetry is in preview, the new agent experience renames things, and the Activity page and Monitor tab overlap in ways that will resolve eventually. Click paths have held up better than layouts, which is why the paths above are written as paths.

The compromise I have made here is to describe layer four by handing it off. That means this piece has a seam in the middle of it, and somebody reading only this will get to the telemetry section and have to go elsewhere. I would rather have that seam than three documents disagreeing about a schema that is still moving.

What I check first#

Publish state, then the conversation ID, then the activity map. That sequence resolves more tickets than anything further down this page, and it takes about four minutes.

The habit worth building is asking which layer can answer the question before opening a tool, because each layer answers exactly one kind of question well and the others badly. Is the content valid is a topic checker question. What plan did the orchestrator make is an activity map question. What happened to a user who is not me is a transcript question. How often does this happen is a telemetry question, and it is the only one of the four that telemetry answers better than anything else.

The failure mode I still catch myself in is reaching for the layer I am most comfortable in rather than the one that fits. Writing KQL feels like progress in a way that checking a publish timestamp does not, and the publish timestamp is right more often.

Troubleshooting#