feat: AI-powered chat panel for Copilot metrics Q&A (#340)

* feat: add AI-powered chat panel for Copilot metrics Q&A

Add an in-dashboard AI chat panel powered by GitHub Models API with
tool-calling architecture. Instead of stuffing all metrics into the
LLM context, the AI is given tools to query specific data on-demand,
mirroring the dashboard tabs.

New files:
- server/services/ai-tools.ts: OpenAI-compatible tool schemas (9 tools)
- server/services/ai-tool-executor.ts: Tool execution via existing data logic
- server/api/ai/chat.post.ts: Chat endpoint with tool-calling loop
- app/components/AiChatPanel.vue: Vuetify chat panel with FAB trigger
- tests/ai-chat.spec.ts: 30 unit tests for tools, executor, and prompts

Configuration:
- NUXT_PUBLIC_ENABLE_AI_CHAT (default: false) - feature gate
- NUXT_AI_MODEL (default: gpt-4o) - model selection
- NUXT_AI_MAX_TOOL_ROUNDS (default: 5) - max iterations
- Requires models:read PAT scope for GitHub Models API

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add mock mode and Playwright E2E tests for AI chat

Add server-side mock for the AI chat endpoint that generates responses
by executing actual tools against dashboard data when IS_DATA_MOCKED=true.
This enables E2E testing without calling the GitHub Models API.

New files:
- server/services/ai-chat-mock.ts: Mock response generator using real tools
- e2e-tests/ai-chat.spec.ts: 9 Playwright tests (serial, shared page)

Changes:
- server/api/ai/chat.post.ts: Mock mode bypass before token check
- playwright.config.ts: Enable AI chat for E2E tests

Tests cover: FAB visibility, panel open/close, suggested questions,
message send/receive, multi-turn conversation, clear, empty input.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(ai-chat): add NUXT_AI_TOKEN for dedicated GitHub Models API auth

Add separate NUXT_AI_TOKEN env var for GitHub Models API authentication,
falling back to NUXT_GITHUB_TOKEN. This allows using a personal-scoped
fine-grained PAT with models:read for AI chat while keeping the org-scoped
PAT for metrics API access.

Tested with real GitHub Models API (gpt-4o) - tool calling works correctly
with get_metrics_summary, get_language_breakdown, get_seat_analysis, and
get_trend_data tools.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(ai-chat): handle missing token with setup guide and user-provided token

When no AI token is configured on the server, the chat panel now:
- Shows a friendly setup guide explaining both options:
  1. Admin sets NUXT_AI_TOKEN env var on server
  2. User provides their own personal GitHub PAT in the UI
- User-provided tokens are stored in browser sessionStorage and
  sent per-request (never persisted server-side)
- Exempt /api/ai/ routes from GitHub auth middleware since AI
  chat uses its own token (NUXT_AI_TOKEN or user-provided)
- Distinguish missing_token vs invalid_token error codes so
  the frontend can show appropriate guidance

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ai-chat): enable AI chat in preview deployment and fix v-if gate

- Add NUXT_PUBLIC_ENABLE_AI_CHAT=true and NUXT_AI_TOKEN to preview
  workflow (both create and update paths)
- Fix v-if condition to handle string 'true' from env vars
- Add AI chat env vars to .env template

Requires PREVIEW_AI_TOKEN secret to be set in the repository with a
personal fine-grained PAT with Models → Read permission.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ai-chat): user-provided token should take priority over NUXT_GITHUB_TOKEN

The org-scoped NUXT_GITHUB_TOKEN (used for metrics API) was taking
priority over the user-provided personal token, causing Models API
auth failures. Fixed priority order:
  1. NUXT_AI_TOKEN (dedicated, always wins)
  2. User-provided token from chat UI
  3. NUXT_GITHUB_TOKEN (general fallback)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(ai-chat): enable AI chat by default

The chat FAB is now visible by default so users discover it.
If no token is configured, the panel shows a setup guide.
Can still be disabled with NUXT_PUBLIC_ENABLE_AI_CHAT=false.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ai-chat): increase z-index so chat panel stays on top of all Vuetify layers

Vuetify overlays/menus use z-index up to 10000. Bumped the chat panel,
FAB, and card to 10001 so they always render above dashboard content.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add Chat/Agent columns to user table and enhance AI prompt with metrics coverage

- Add Chat and Agent icon columns to UserMetricsViewer table with tooltips
  showing per-feature activity (interactions, code gen, LOC)
- Create shared feature-classification utility (CHAT_FEATURES, AGENT_FEATURES)
  used by UI, report-transformer, and AI tool executor
- Enhance AI system prompt with detailed explanation of what metrics cover:
  acceptance rate (inline completions only), per-user feature breakdown,
  CLI limitations, and agent activity scope
- Update AI tool executor to include feature flags in top-N user summaries
  and full feature breakdown in single-user lookups
- Link to GitHub docs for metrics reference

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: show Chat/Agent as numeric columns and bump chat z-index to 100000

- Replace boolean icons with actual interaction counts in Chat/Agent columns
  (sum of interactions + code gen events per feature group)
- Make columns sortable so users can find top chat/agent users
- Tooltip on hover still shows per-feature breakdown details
- Bump AiChatPanel z-index from 10001 to 100000 to stay above Vuetify's
  dynamic overlay stack (base 2000, increments by 10 per layer)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add 'Understanding your metrics' info panel to user metrics tab

Collapsible panel explaining:
- What acceptance rate actually measures (inline completions only)
- What Chat/Agent columns show and why low acceptance rate is expected
  for agent-heavy users
- What's not captured per user (CLI, github.com, PR stats)
- Adoption tips: look at active days + interactions, not just acceptance rate
- Link to GitHub docs on analyzing Copilot usage

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add GitHub Docs links for Copilot adoption and rollout

Links to official guides: rolling out at scale, driving adoption,
measuring trial success, tracking license usage, and maintaining
codebase standards.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: use max z-index (2147483647) via inline styles for chat panel

Scoped CSS z-index was being overridden by Vuetify's elevation classes
which create stacking contexts via CSS transforms. Using inline styles
with z-index: 2147483647 (max 32-bit int) on the panel, FAB, and card
ensures the chat always renders on top regardless of other stacking
contexts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: set max z-index on all tooltips to prevent rendering behind headers

Vuetify v-tooltip teleports to document body with dynamic z-index that
can render behind sticky table headers and other stacking contexts.
Added :z-index="2147483647" prop to all tooltips in UserMetricsViewer
and AiChatPanel.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: add Playwright z-index regression tests using elementFromPoint

4 tests that catch z-index stacking order issues:
- Chat panel renders above dashboard tiles (elementFromPoint check)
- Chat FAB renders above dashboard content (elementFromPoint check)
- Tooltip z-index value is max int (2147483647)
- Chat panel renders above user metrics table (elementFromPoint check)

Uses document.elementFromPoint() to verify what element is actually
visible at specific coordinates - the most reliable way to detect
z-index regressions where CSS stacking contexts cause elements to
render behind other components.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add Agent LOC column to user metrics table

Shows lines of code added via agent features (chat_panel_agent_mode,
agent_edit) per user. Sortable, styled consistently with Agent column.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: teleport chat panel to body to fix z-index bleed-through

Move AiChatPanel into <Teleport to='body'> so it renders outside
Vuetify's .v-application wrapper, eliminating CSS stacking context
conflicts that caused dashboard tiles to paint over the chat panel.

- Add explicit background: #ffffff since Vuetify theme CSS vars are
  not inherited outside .v-application
- Replace theme variable references in scoped CSS with hardcoded colors
  (indigo #3f51b5 for user bubbles, #f5f5f5 for assistant bubbles)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: restyle tooltips to blue with white text, add missing tooltips

- Global tooltip style: dark navy (#1a237e) background, white text, 13px font
- Added .metric-tooltip and .tooltip-text CSS classes in global.css
- Added tooltips to UserMetricsViewer summary tiles (Total Users, Active Users, Avg Acceptance Rate)
- Added tooltips to PullRequestViewer tiles (PRs Created, PRs Reviewed, PRs Merged)
- Styled Chat/Agent column tooltips with new blue theme
- Changed Agent LOC values from purple to black (no tooltip/link)
- Updated all existing tooltips across 8 components to use new style

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: rename Lines Accepted to Copilot LOC with breakdown tooltip

- Rename column from 'Lines Accepted' to 'Copilot LOC' to clarify it
  includes ALL Copilot features (completions + chat + agent), not just
  inline completions
- Add hover tooltip on Copilot LOC values showing per-feature breakdown
  (inline completions, chat, agent) with agent percentage
- Expand info panel with 'How users are using Copilot' section describing
  inline-first, chat-first, agent-heavy, and CLI user patterns
- Add explanation of Agent LOC, Copilot LOC, and merged PR tracking gaps
- Import COMPLETION_FEATURES for LOC breakdown calculation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: center text vertically in user metrics summary tiles

Remove hidden filler divs and add height: 100% to v-card-item so
flexbox centering works properly within the fixed-height cards.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: replace dead GitHub Docs link with current URL

Old: /copilot/rolling-out-github-copilot-at-scale/analyzing-usage-over-time (404)
New: /copilot/reference/copilot-usage-metrics/interpret-copilot-metrics

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: bump version to 3.2.0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: disable historical mode in CI Playwright tests

The .env file has ENABLE_HISTORICAL_MODE=true which causes the built
server to attempt PostgreSQL connections during the 'test' and
'test-docker-playwright' jobs where no PG is available, resulting in
ECONNREFUSED errors and test timeouts.

Add ENABLE_HISTORICAL_MODE=false to both jobs to override .env.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: karpikpl

.env

@@ -34,4 +34,11 @@ NUXT_PUBLIC_USE_NEW_API=true
 # Comma-separated list of tab names to hide from the dashboard UI.
 # Available tabs: languages, editors, copilot chat, agent activity, pull requests, github.com, seat analysis, user metrics, api response
 # Example: NUXT_PUBLIC_HIDDEN_TABS=api response,agent activity
-NUXT_PUBLIC_HIDDEN_TABS=
+NUXT_PUBLIC_HIDDEN_TABS=
+
+# AI Chat — powered by GitHub Models API
+# Set to true to show the AI chat floating button on the dashboard
+NUXT_PUBLIC_ENABLE_AI_CHAT=true
+# Personal fine-grained PAT with Models → Read permission (scoped to personal account, not org)
+# If not set, users can provide their own token in the chat UI
+# NUXT_AI_TOKEN=

.github/workflows/playwright.yml

@@ -60,6 +60,7 @@ jobs:
           NUXT_SESSION_PASSWORD=foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo \
           NUXT_PUBLIC_GITHUB_ORG=octodemo \
           NUXT_PUBLIC_IS_DATA_MOCKED=true \
+          ENABLE_HISTORICAL_MODE=false \
           RUN_COMMAND='npm run preview' npx playwright test --grep-invert "@seed|@storage"
     - uses: actions/upload-artifact@v4
       if: ${{ !cancelled() }}
@@ -88,6 +89,7 @@ jobs:
           -e NUXT_SESSION_PASSWORD=foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo \
           -e NUXT_PUBLIC_GITHUB_ORG=octodemo \
           -e NUXT_PUBLIC_IS_DATA_MOCKED=true \
+          -e ENABLE_HISTORICAL_MODE=false \
           app:pw npx playwright test --grep-invert "@seed|@storage"
     - uses: actions/upload-artifact@v4
       if: always()

.github/workflows/preview-deploy.yml

@@ -267,6 +267,8 @@ jobs:
                 NUXT_PUBLIC_IS_DATA_MOCKED=false \
                 NUXT_PUBLIC_SCOPE=${{ secrets.PREVIEW_SCOPE || 'organization' }} \
                 NUXT_PUBLIC_GITHUB_ORG=${{ secrets.PREVIEW_GITHUB_ORG }} \
+                NUXT_PUBLIC_ENABLE_AI_CHAT=true \
+                NUXT_AI_TOKEN=${{ secrets.PREVIEW_AI_TOKEN }} \
                 NUXT_PUBLIC_DEPLOY_INFO="${{ steps.deploy-info.outputs.deploy_info }}"
           else
             az containerapp update \
@@ -283,6 +285,8 @@ jobs:
                 NUXT_PUBLIC_IS_DATA_MOCKED=false \
                 NUXT_PUBLIC_SCOPE=${{ secrets.PREVIEW_SCOPE || 'organization' }} \
                 NUXT_PUBLIC_GITHUB_ORG=${{ secrets.PREVIEW_GITHUB_ORG }} \
+                NUXT_PUBLIC_ENABLE_AI_CHAT=true \
+                NUXT_AI_TOKEN=${{ secrets.PREVIEW_AI_TOKEN }} \
                 "NUXT_PUBLIC_DEPLOY_INFO=${{ steps.deploy-info.outputs.deploy_info }}"
           fi
 

app/assets/global.css

@@ -197,4 +197,16 @@
   /* Pagination buttons hover state unify with buttons */
   .v-data-table .v-data-table-footer .v-btn:hover { 
     background-color: var(--app-accent-weak) !important; 
+  }
+
+  /* ── Global tooltip card style ── */
+  .metric-tooltip {
+    background-color: #1a237e !important;
+    max-width: 380px;
+    border-radius: 8px !important;
+  }
+  .metric-tooltip .tooltip-text {
+    color: #ffffff !important;
+    font-size: 13px !important;
+    line-height: 1.45;
   }

app/components/AgentActivityViewer.vue

@@ -10,8 +10,8 @@
               <template #activator="{ props }">
                 <div v-bind="props" class="text-h6 mb-1">Agent Lines Added</div>
               </template>
-              <v-card class="pa-2" style="background-color: #f0f0f0; max-width: 350px;">
-                <span class="text-caption" style="font-size: 10px !important;">Total lines of code added by Copilot agents (agent mode and edit mode) across all IDEs.</span>
+              <v-card class="pa-3 metric-tooltip">
+                <span class="tooltip-text">Total lines of code added by Copilot agents (agent mode and edit mode) across all IDEs.</span>
               </v-card>
             </v-tooltip>
             <div class="text-caption">{{ dateRangeDescription }}</div>
@@ -29,8 +29,8 @@
               <template #activator="{ props }">
                 <div v-bind="props" class="text-h6 mb-1">Agent Lines Deleted</div>
               </template>
-              <v-card class="pa-2" style="background-color: #f0f0f0; max-width: 350px;">
-                <span class="text-caption" style="font-size: 10px !important;">Total lines of code deleted by Copilot agents across all IDEs.</span>
+              <v-card class="pa-3 metric-tooltip">
+                <span class="tooltip-text">Total lines of code deleted by Copilot agents across all IDEs.</span>
               </v-card>
             </v-tooltip>
             <div class="text-caption">{{ dateRangeDescription }}</div>
@@ -48,8 +48,8 @@
               <template #activator="{ props }">
                 <div v-bind="props" class="text-h6 mb-1">Agent Code Generations</div>
               </template>
-              <v-card class="pa-2" style="background-color: #f0f0f0; max-width: 350px;">
-                <span class="text-caption" style="font-size: 10px !important;">Number of distinct code generation events by Copilot agents. Each event represents one code change applied by an agent.</span>
+              <v-card class="pa-3 metric-tooltip">
+                <span class="tooltip-text">Number of distinct code generation events by Copilot agents. Each event represents one code change applied by an agent.</span>
               </v-card>
             </v-tooltip>
             <div class="text-caption">{{ dateRangeDescription }}</div>