Skip to content

Commit c8aa964

Browse files
CopilotkarpikplCopilot
authored
feat: store per-user Copilot metrics in DB for time-series team queries beyond 28 days (#320)
* Initial plan * chore: initial progress plan Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/5335ff71-a8b7-4d3b-94c3-ba4e5298fe24 * feat: derive team-level metrics from per-user Copilot Metrics API - Add UserDayRecord type for the per-user flat-array report format - Add fetchUsersLatestReport() to fetch from users-28-day/latest endpoint - Add mockRequestUsersDownloadLinks() for mock mode support - Create user-metrics-aggregator.ts to filter by team membership and aggregate - Update metrics-util-v2.ts to use per-user aggregation for team scopes - Add mock team membership data to fetchAllTeamMembers for mock mode - Add organization-users-28-day-report.json mock data (4 users across 2 teams) - Add 13 unit tests covering the aggregation logic Fixes #319 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/5335ff71-a8b7-4d3b-94c3-ba4e5298fe24 * feat: add user_metrics DB table and daily sync of per-user records Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/a6664e8e-ae95-4975-b09a-d134cfe6581b * feat: restructure team historical path to always resolve members then aggregate from user_metrics DB Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/a6664e8e-ae95-4975-b09a-d134cfe6581b * refactor: consolidate user_metrics into user_day_metrics — single table, single sync Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/df8cca74-2471-4729-8218-e1226e3a2351 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * refactor: address code review — document active_users threshold change, strengthen window test, extract constant Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/df8cca74-2471-4729-8218-e1226e3a2351 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: scope normalization, team-scope sync gate, and batch chunking for user_day_metrics Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/b6d60a7e-ed39-4ab0-b591-51bb07dcb607 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * feat: fix tests, v3.0.3, remove CHANGELOG, deploy-info in footer, new preview-deploy workflow Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/54dbe4e5-0ee3-4edc-a369-c8785cd01293 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: mock fetchLatestReport uses relative dates so storage pipeline test always passes Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/73635db5-669c-4657-aa2c-07e961d69053 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: historical mode fallback — 503 on DB failure without auth, graceful empty for history, unit tests Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/db83f379-bce8-4f86-989c-73d436e1c4f5 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: chat mock data (fallback), v3.0.4, remove design docs, DB readiness wait in preview deploy Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/96e057e3-efd8-4694-b3c7-a95c8da1eab7 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: same-env guarantee for all containers, fixed Log Analytics workspace name Agent-Logs-Url: https://github.com/github-copilot-resources/copilot-metrics-viewer/sessions/96e057e3-efd8-4694-b3c7-a95c8da1eab7 Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: filter user-metrics by team membership /api/user-metrics was returning all org users regardless of team scope. Added filterByTeamIfNeeded() that resolves team members via GitHub Teams API and filters UserTotals by user_id (primary) and login (fallback). - Historical mode + team scope without auth now returns 503 - Added 5 unit tests for team filtering behavior - Added Playwright real-data test suite (30 tests) for cody-test-org Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip real-data Playwright tests in CI real-data-teams.spec.ts requires a live GitHub API connection and PostgreSQL. Skip via RUN_REAL_DATA_TESTS env var check, plus grepInvert: /@real-data/ in Docker Playwright config for defense in depth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: align metrics/reportData indices in fetchAndStore transformReportToMetrics() sorts day_totals by day, but the store loop iterated the original unsorted report.day_totals array. This caused metrics[i] (sorted) to be saved with dayData.day from the unsorted array, corrupting the metrics_date ↔ data.date mapping in the DB. Fix: use sorted reportData for the store loop so indices stay aligned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: resolve @/model import in sync container The standalone sync container runs via tsx which doesn't have Nuxt's auto-generated tsconfig with @/* path aliases. Added tsconfig.sync.json with explicit @/* -> app/* path mapping and updated Dockerfile.sync to use --tsconfig tsconfig.sync.json flag. This fixes: ERR_MODULE_NOT_FOUND: Cannot find package '@/model' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: don't coerce undefined premium_requests_total to 0 The GitHub API does not include a premium_requests_total field. The DB read path was coercing undefined to 0, which made the UI show the premium column with all zeros. Preserve undefined so the UI's hasPremiumData check correctly hides the column. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: remove fabricated premium_requests_total column The GitHub Copilot Metrics API does not include a premium_requests_total field. This was entirely fabricated in mock data and caused a phantom column to appear in the UI. Removed from: - Vue component (card, filter, table column, chart dataset) - API types and aggregation logic - Storage interfaces and DB read functions - Mock data JSON files - Unit tests (6 tests removed) - E2E tests and page objects (3 tests removed) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: remove premium reference from metrics.spec.ts e2e test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: update README and DEPLOYMENT for v3.0 team metrics and historical mode - Replace outdated legacy API shutdown banner with past-tense note - Add Operating Modes section explaining Direct API vs Historical mode - Document how team metrics are derived from per-user data (no team API) - Add Per-User Metrics tab documentation - Remove broken links to non-existent migration docs - Fix NUXT_PUBLIC_SCOPE values (team-organization/team-enterprise) - Remove old 5+ team member warning (no longer applies) - Update API docs link to new Copilot Usage Metrics API - Make PostgreSQL/sync optional in DEPLOYMENT.md (historical mode only) - Add Direct API mode Docker Compose instructions - Remove USE_LEGACY_API from env vars (legacy API shut down) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: update screenshots and fix team metrics requiring historical mode - Capture fresh screenshots for all tabs using Playwright - Add new user-metrics.png screenshot for User Metrics tab - Fix incorrect claim that team views work in Direct API mode - Team-scoped views require Historical mode (PostgreSQL) because team data is derived from per-user records stored in the database Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * revert: restore original screenshots, keep only new user-metrics.png Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> Co-authored-by: Piotr Karpala <karpik.pl@gmail.com> Co-authored-by: Piotr Karpala <piotrkarpala@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 08881d6 commit c8aa964

44 files changed

Lines changed: 2185 additions & 5077 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/preview-deploy.yml

Lines changed: 246 additions & 119 deletions
Large diffs are not rendered by default.

API_MIGRATION_DESIGN.md

Lines changed: 0 additions & 1054 deletions
This file was deleted.

API_QUICK_REFERENCE.md

Lines changed: 0 additions & 428 deletions
This file was deleted.

CHANGELOG.md

Lines changed: 0 additions & 66 deletions
This file was deleted.

DEPLOYMENT.md

Lines changed: 48 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,36 @@ The app runs in a Docker container, so it can be deployed anywhere containers ar
66

77
## Architecture
88

9-
The application consists of three components:
9+
The application supports two operating modes:
10+
11+
### Direct API Mode (no database)
12+
The simplest setup — the web app fetches metrics directly from GitHub's Copilot Usage Metrics API on each page load, returning the latest 28-day rolling window.
13+
14+
| Component | Description | Required |
15+
|-----------|-------------|----------|
16+
| **Web App** | Nuxt 3 dashboard (port 3000/80) | Yes |
17+
18+
### Historical Mode (with database)
19+
Adds a PostgreSQL database and sync service for persistent storage. This enables metrics beyond the 28-day API window, per-user time-series history, and full historical team views.
1020

1121
| Component | Description | Required |
1222
|-----------|-------------|----------|
1323
| **Web App** | Nuxt 3 dashboard (port 3000/80) | Yes |
1424
| **PostgreSQL** | Database for historical metrics storage | Yes |
1525
| **Sync Service** | Scheduled job that downloads metrics from GitHub API to PostgreSQL | Yes |
1626

17-
The sync service runs daily (2 AM UTC by default) and downloads the latest Copilot usage metrics from GitHub's async download API. The web app reads from the database for fast, reliable dashboard rendering.
27+
The sync service runs daily (2 AM UTC by default) and downloads the latest Copilot usage metrics from GitHub's API. The web app reads from the database for fast, reliable dashboard rendering.
1828

19-
> [!NOTE]
20-
> **Legacy API mode**: Set `USE_LEGACY_API=true` to use the deprecated synchronous API (shutting down April 2, 2026). This mode does not require PostgreSQL but provides limited data.
29+
#### How Team Metrics Work
30+
31+
GitHub's Copilot Usage Metrics API does not provide team-level endpoints. Instead, this application **derives team metrics** by:
32+
1. Downloading per-user daily metrics from the organization/enterprise endpoint
33+
2. Resolving team membership via the GitHub Teams API
34+
3. Filtering and aggregating per-user data in-memory for each team
35+
36+
This approach works in **Historical mode** only:
37+
- **Direct API mode**: Team-scoped views are not available (no per-user data stored to filter)
38+
- **Historical mode**: Team data covers the full stored history, enabling long-term team trend analysis
2139

2240
## Deployment options
2341

@@ -129,7 +147,25 @@ docker compose up web
129147
# Open http://localhost:3000/orgs/your-org?mock=true
130148
```
131149

132-
### Running with Real GitHub Data
150+
### Running with Real GitHub Data (Direct API — no database)
151+
152+
The simplest setup — metrics come directly from the GitHub API (28-day rolling window):
153+
154+
```bash
155+
export NUXT_GITHUB_TOKEN=github_pat_... # Fine-grained PAT with "Copilot metrics" permission
156+
export NUXT_PUBLIC_GITHUB_ORG=your-org
157+
export NUXT_PUBLIC_IS_DATA_MOCKED=false
158+
159+
docker compose up web
160+
# Open http://localhost:3000/orgs/your-org
161+
```
162+
163+
> [!NOTE]
164+
> **Team-scoped views** (e.g., `/orgs/your-org/teams/your-team`) require **Historical mode** with PostgreSQL. Without the database, team metrics cannot be computed because team data is derived by filtering per-user records stored in the database. The Teams Comparison tab is available to browse teams, but individual team drill-down requires Historical mode.
165+
166+
### Running with Historical Mode (database + sync)
167+
168+
Adds PostgreSQL for persistent storage — enables metrics beyond 28 days, per-user time-series history, and full historical team views:
133169

134170
```bash
135171
export NUXT_GITHUB_TOKEN=github_pat_... # Fine-grained PAT with "Copilot metrics" permission
@@ -250,16 +286,15 @@ These endpoints respond in ~200ms without making external API calls and do not r
250286
| Variable | Description | Required |
251287
|----------|-------------|----------|
252288
| `NUXT_GITHUB_TOKEN` | GitHub PAT with Copilot metrics permission | Yes (unless OAuth) |
253-
| `NUXT_PUBLIC_SCOPE` | `organization`, `enterprise`, or `team` | Yes |
254-
| `NUXT_PUBLIC_GITHUB_ORG` | GitHub organization slug | For org scope |
255-
| `NUXT_PUBLIC_GITHUB_ENT` | GitHub enterprise slug | For enterprise scope |
256-
| `NUXT_PUBLIC_GITHUB_TEAM` | GitHub team slug | For team scope |
289+
| `NUXT_PUBLIC_SCOPE` | `organization`, `enterprise`, `team-organization`, or `team-enterprise` | Yes |
290+
| `NUXT_PUBLIC_GITHUB_ORG` | GitHub organization slug | For org/team-org scope |
291+
| `NUXT_PUBLIC_GITHUB_ENT` | GitHub enterprise slug | For enterprise/team-ent scope |
292+
| `NUXT_PUBLIC_GITHUB_TEAM` | GitHub team slug | For team scopes |
257293
| `NUXT_SESSION_PASSWORD` | Session encryption key (min 32 chars) | Yes |
258-
| `DATABASE_URL` | PostgreSQL connection string | Yes |
259-
| `ENABLE_HISTORICAL_MODE` | `true` to read metrics from database | Yes |
260-
| `SYNC_ENABLED` | `true` for sync service, `false` for web app | Per service |
294+
| `DATABASE_URL` | PostgreSQL connection string | Historical mode only |
295+
| `ENABLE_HISTORICAL_MODE` | `true` to read metrics from database | Historical mode only |
296+
| `SYNC_ENABLED` | `true` for sync service, `false` for web app | Historical mode only |
261297
| `SYNC_DAYS_BACK` | Days to sync (default: 1 for daily, 28 for bulk) | Sync only |
262-
| `USE_LEGACY_API` | `true` to use deprecated API (no DB required) | Optional |
263298
| `NUXT_PUBLIC_USING_GITHUB_AUTH` | `true` to enable GitHub OAuth | Optional |
264299
| `NUXT_OAUTH_GITHUB_CLIENT_ID` | GitHub App client ID | For OAuth |
265300
| `NUXT_OAUTH_GITHUB_CLIENT_SECRET` | GitHub App client secret | For OAuth |

0 commit comments

Comments
 (0)