Skip to content

Commit a645278

Browse files
CopilotkarpikplCopilotgithub-advanced-security[bot]
authored
Migration to async Copilot Usage Metrics API - Full Implementation (All Phases Complete) (#299)
* Initial plan * Add comprehensive API migration design documentation Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * Add executive summary for stakeholders Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * Add design review checklist for stakeholders Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * Address feedback: Add Nitro unstorage abstraction and separate sync container support Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * feat: Add Phase 1 - Storage infrastructure with unstorage abstraction - Add unstorage dependency for database-agnostic persistence - Create storage type definitions and interfaces - Implement metrics-storage, seats-storage, and sync-storage modules - Configure filesystem driver in nuxt.config.ts for development - Add initial storage tests (need Nuxt runtime context to run) Storage layer supports PostgreSQL, Redis, MongoDB, filesystem via config Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * feat: Add Phase 2 - New GitHub API integration with mock support - Implement GitHub Copilot Usage Metrics API client for async downloads - Add NDJSON parsing functionality - Create mock API implementation for testing - Support both real and mock modes via runtime config - Add comprehensive tests for API client and NDJSON parsing All tests passing (9/9) Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * feat: Add Phases 3-5 - Sync service, API updates, and scheduler Phase 3: Data Sync Service - Implement core sync service with gap detection - Add admin endpoints for manual sync trigger and status monitoring - Support single-date and date-range synchronization Phase 4: API Layer Updates - Create metrics-util-v2 with storage and new API support - Update metrics endpoint with feature flags (USE_NEW_API, ENABLE_HISTORICAL_MODE) - Maintain backward compatibility with legacy API Phase 5: Background Scheduler - Add daily-sync scheduled task (Nitro) - Create standalone sync-entry.ts for separate container deployment - Add Dockerfile.sync for containerized sync jobs - Create k8s/cronjob.yaml for Kubernetes deployments - Add docker-compose.yml for dev environment All 106 existing tests passing Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * feat: Add Phase 6 - Documentation and configuration - Create comprehensive MIGRATION_GUIDE.md with step-by-step instructions - Create TESTING_GUIDE.md with test scenarios and commands - Create IMPLEMENTATION_SUMMARY.md tracking design vs. implementation - Add .env.example with all new configuration options - Update README with migration status and links to guides - Document all new environment variables and feature flags All 106 tests passing, production build successful Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * test: Add integration tests for migration features - Add api-migration-integration.spec.ts with 10 tests - Test date range calculations, storage keys, URL patterns - Test backward compatibility requirements - Test configuration and feature flags - Verify sync service logic Total: 116 tests passing (106 existing + 10 new) Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * docs: Add security scan results and code review notes - CodeQL security scan passed (0 vulnerabilities) - Code review completed (minor style notes only) - Logger pattern consistent with existing codebase - All quality checks passed Ready for production deployment Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * Merge main branch updates (dependency updates) - Merge changes from main branch - Update package-lock.json with latest dependencies - Includes minimatch, rollup, ajv updates from main - Regenerated package-lock.json to resolve conflicts - All 116 tests still passing Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * test: Replace placeholder tests with meaningful integration tests - Remove placeholder tests that don't test actual functionality - Add real tests for storage key generation using actual functions - Add tests for NDJSON parsing with mock data generation - Add tests for date range calculations - Add tests for API URL construction patterns - All tests now verify actual implementation code Total: 123 tests passing (106 existing + 17 new meaningful tests) Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * docs: Update implementation summary with improved test count - Update test count to 123 (from 115) - Mark integration tests as complete (not placeholder) - Update status to reflect all 6 phases complete All tests passing, implementation complete Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: Address code review feedback - remove unused imports, fix auth error, update docs - Fix Dockerfile.sync to compile TypeScript instead of using tsx runtime - Make SYNC_SCHEDULE configurable via environment variable - Remove unused imports (getMetrics, detectGaps, SyncResult) - Fix auth error to return 401 instead of 500 (use createError) - Sort pending/failed syncs by creation date (most recent first) - Update .env.example to clarify ADMIN_API_SECRET is not yet implemented - Update API_MIGRATION_DESIGN.md status to "Implementation Complete" - Update MIGRATION_SUMMARY.md status to reflect completed implementation - Update README to show all 6 phases complete - Remove tautological tests (URL construction with constants) All 118 tests passing Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: Simplify Dockerfile.sync to use tsx runtime with all dependencies - Changed from compile-time TS compilation to runtime tsx execution - Install all dependencies (not just production) to include tsx - Simpler and more reliable for sync container - Add test-docker-sync.sh script for Docker build verification All 118 tests passing Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * docs: Add security warnings for admin endpoints and fix test count - Add prominent security warning in MIGRATION_GUIDE.md for admin endpoints - Update .env.example with ⚠️ SECURITY WARNING prefix - Fix test count in IMPLEMENTATION_SUMMARY.md (118 not 123) - Improve test-docker-sync.sh with pipefail and better error handling - Add nginx example for securing admin endpoints All 118 tests passing, build successful Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: Resolve package-lock.json conflicts with main - Take package-lock.json from main branch (latest dependency updates) - Reinstall to properly add unstorage dependency - Regenerate package-lock.json with both main updates and unstorage - All 118 tests passing, build successful Resolves merge conflicts with main branch Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> * fix: Resolve high severity vulnerabilities (minimatch ReDoS, serialize-javascript RCE) - npm audit fix resolves minimatch ReDoS (GHSA-3ppc-4f35-3m26, GHSA-7r86-cg39-jmmj, GHSA-23c5-xmqv-rm74) - Add serialize-javascript ^7.0.4 override to fix RCE vulnerability (GHSA-5c6j-r48x-rmvq) - npm audit now reports 0 vulnerabilities Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Implement new Copilot Usage Metrics download API as default Replace the broken USE_NEW_API implementation with a working one that matches the actual GitHub Copilot Usage Metrics API spec: - Fix API client: response is { download_links: [...], report_day } not { download_url, expires_at }. Files are single JSON, not NDJSON. - Use efficient 28-day/latest endpoint (1 API call) instead of day-by-day - Add report-transformer.ts to convert new report format (day_totals with totals_by_ide/feature/language_feature/model_feature) into CopilotMetrics format (copilot_ide_code_completions/copilot_ide_chat) the UI expects - Make new API the default path with automatic fallback to legacy API - Replace USE_NEW_API flag with FORCE_LEGACY_API escape hatch - Update mock service to generate realistic report data - Add 126 passing unit tests (was 118), including transformer tests New API endpoints used: - GET /orgs/{org}/copilot/metrics/reports/organization-28-day/latest - GET /enterprises/{ent}/copilot/metrics/reports/enterprise-28-day/latest Tested with real GitHub API (cody-test-org): 81 days of data, language/editor/model breakdowns, all dashboard tabs working. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: Replace fallback with COPILOT_METRICS_API env switch Replace silent fallback behavior with an explicit env var switch: COPILOT_METRICS_API=new (default) - download-based reports API COPILOT_METRICS_API=legacy - deprecated REST API (dies April 2, 2026) This makes failures explicit instead of silently degrading to a soon-to-be-dead API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Rewrite sync/storage layer for new download-based API - Sync service uses 28-day bulk download (1 API call for 28 days) - Stores both CopilotMetrics (UI) and ReportDayTotals (aggregation) - Daily cron task downloads latest report, saves new days only - Standalone container uses bulk download instead of day-by-day - Admin endpoint supports sync-bulk action - Storage types extended with optional reportData field - 21 new tests covering sync, storage round-trip, and transformer Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Update docker-compose for new API and add playwright test service - Replace USE_NEW_API with COPILOT_METRICS_API env var - Pass env vars from host via ${VAR:-default} syntax - Add playwright service (profile: test) for E2E testing - Fix playwright docker config to set NITRO_PORT=3000 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Historical mode with DB storage and sync-on-miss - Middleware: Skip auth for metrics/seats when ENABLE_HISTORICAL_MODE=true (set empty headers, let handler decide based on storage availability) - Middleware: Skip auth for admin routes (admin endpoints handle auth internally) - metrics-util-v2: Rewrite decision tree: 1. Mock mode (env var IS_DATA_MOCKED) → mock data, no DB, no API 2. Historical mode (ENABLE_HISTORICAL_MODE) → DB with sync-on-miss 3. Direct API mode → fetch from GitHub API - metrics-util-v2: Default to last 28 days when no date range specified - Admin sync endpoint: Use isMockMode() env check instead of per-request param - Options.ts: Fix mock detection to check === true or === 'true' (not truthy) - Docker compose: Add sync-seed and playwright-storage services for full pipeline E2E testing with shared volume - E2E test: Two-phase storage pipeline test: Phase 1 (@seed): Sync mock data to shared volume Phase 2 (@storage): Dashboard reads from DB without GitHub token Tested: 147 unit tests pass, 123 mock E2E tests pass, 12 storage pipeline E2E tests pass (6 seed + 6 storage, 3 browsers each) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: Add docker-compose documentation and usage guide - Add comprehensive header comments to docker-compose.yml with quick start, testing commands, services overview, and environment variables reference - Add inline comments for each service explaining purpose and modes - Add 'Scenario 4: Docker Compose' section to DEPLOYMENT.md covering: local development (mock, real API, historical mode), sync service, E2E tests (mock and storage pipeline), services table, cleanup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: Fix docker-compose — DB only needed for historical mode - Remove hard depends_on db from web service (DB not needed for mock/direct API) - Update Quick Start: 'docker compose up web' for mock and direct API modes, 'docker compose up web db' only for historical mode - Add note in DEPLOYMENT.md clarifying when db service is needed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Update Dockerfile.sync to node:24-alpine3.23 Matches the main Dockerfile base image. Fixes npm ci failure caused by peer dependency version mismatch between npm 10 (node 20) and the lockfile generated by npm 11 (node 24). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Standalone storage for sync container - Replace useStorage() with getStorage() helper that falls back to unstorage+fsDriver when running outside Nitro context (sync-entry.ts) - Fix isMockMode() to fall back to process.env when useRuntimeConfig is unavailable in standalone mode - Remove USER node from Dockerfile.sync for volume write access - Add sync_data volume mount to web service for historical mode - Tested: sync stores 28 days, web reads from DB successfully Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Add database name to PostgreSQL healthcheck pg_isready defaults to checking a database matching the username, causing FATAL errors. Specify -d copilot_metrics explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Replace unstorage with PostgreSQL for metrics persistence Replace the filesystem-based unstorage layer with PostgreSQL: - Add server/storage/db.ts: connection pool, schema init (metrics, sync_status, seats tables with JSONB columns and indexes) - Rewrite metrics-storage.ts: single SQL query for date ranges instead of N file reads, proper upsert with ON CONFLICT - Rewrite sync-storage.ts and seats-storage.ts with SQL - Add server/plugins/db-init.ts: auto-init schema on startup - Add tests/pg-storage.spec.ts: 19 tests using pg-mem (in-memory PG) - Fix import for standalone sync container (ofetch fallback) - Fix ofetch responseType for report downloads - Wire DATABASE_URL through docker-compose for all services Tested: sync saves 26 days to PG, web reads from PG, all 161 unit tests + 123 E2E tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Fix acceptance rate and add agent/PR dashboards - Fix report-transformer to filter code_completion feature only for suggestion/acceptance metrics (was including agent_edit, chat) - Fix model mapping to scope by feature type (code_completion vs chat) - Add PR types (ReportPullRequests, ReportCliTotals) to API types - Update UI labels: 'Suggestions' → 'Code Completions' across all views - Update chat labels: 'Turns' → 'Chat Interactions', 'Acceptances' → 'Code Actions' - Add AgentActivityViewer component: agent LOC charts, feature breakdown, model-feature table (showing claude-opus, gpt-5.3-codex, etc.) - Add PullRequestViewer component: PRs created/reviewed/merged, Copilot contributions, review suggestions - Pass raw ReportDayTotals through API response for new components - Add getReportDataByDateRange() for DB queries - Add comprehensive acceptance rate test with real-world data structure Acceptance rate was 0.4% because transformer mapped ALL features' activity counts as 'suggestions'. Now correctly filters to code_completion only, yielding ~15-40% rates matching real data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Route mock mode through new API transformer and add E2E value assertions - Mock mode now uses generateMockReport() → transformReportToMetrics() when COPILOT_METRICS_API=new, exercising the same code path as production - Legacy mock mode still reads old JSON files when COPILOT_METRICS_API=legacy - Added acceptance rate assertion (5-80% range) to catch the 0.4% regression - Added suggestion count assertion to verify non-zero completions - Updated E2E locators to match new UI labels: - 'Total Code Completions' (was 'Total count of Suggestions (Prompts)') - 'Cumulative Chat Interactions' (was 'Cumulative Number of Turns') - 'Top 5 Languages by accepted code completions' (was 'by accepted suggestions') - All 27 E2E tests pass, all 162 unit tests pass Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Add retry with backoff for PostgreSQL schema init on startup The web container can start before PostgreSQL is fully ready to accept connections, even with depends_on/service_healthy. Added retry logic (up to 10 attempts with linear backoff) for PG error code 57P03 ('database system is starting up') and ECONNREFUSED. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: Add test-results-old to .gitignore Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: Add real API mock data files and full HTTP download flow Mock mode now simulates the complete two-step GitHub API flow: 1. requestDownloadLinks() → returns URLs to localhost static files 2. downloadReport() → fetches the report via HTTP (no special mock bypass) New mock data files in public/mock-data/new-api/: - organization-28-day-report.json — anonymized real API data from cody-test-org (26 days, 12 models, 6 features, pull request data, 30.4% acceptance rate) - enterprise-28-day-report.json — enterprise scope variant - organization-1-day-report.json — single day report - download-links-*.json — mock API responses pointing to local files Key changes: - mockRequestDownloadLinks() returns localhost URLs (port-aware via env) - downloadReport() no longer has mock bypass — always fetches via HTTP - generateMockReport() reads from static file, falls back to generator - metrics-util-v2 mock path calls fetchLatestReport() for full pipeline - Unit tests updated for new mock API Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Lines acceptance rate inflated by agent_edit LOC (548% → ~36%) totals_by_language_model has no 'feature' field, so it included LOC from all features (agent_edit adds thousands of lines). Switched to using totals_by_language_feature filtered to code_completion, which correctly scopes lines suggested/accepted. Real data: 2,235 lines suggested → 811 accepted (36.3%) Bug showed: 12,749 suggested → 71,188 accepted (558%) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: Chat active users always 0, chat acceptances missing code actions - total_engaged_users on copilot_ide_chat was hardcoded to 0; now uses monthly_active_chat_users from the report day data - total_chat_insertion_events was hardcoded to 0; now maps code_generation_activity_count from chat features (code generated in chat responses that users can insert/apply) - total_chat_copy_events already used code_acceptance_activity_count Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: github.com tab uses legacy API path, shows no model data /api/github-stats was calling getMetricsData() (legacy) instead of getMetricsDataV2(). Switched to v2 so it gets the same CopilotMetrics data from the new download-based API, including IDE chat models and code completion models. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: redesign github.com tab for new API, fix sync re-download - /api/github-stats now uses getMetricsDataV2 and returns reportData stats (model-feature table, feature/model summaries, active users) - AgentModeViewer shows rich model usage tables when reportData available, falls back to legacy expansion panels for old API - hasMetrics() now requires report_data IS NOT NULL so sync will re-download days missing raw report data - Updated pg-storage tests to include reportData fixtures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: replace COPILOT_METRICS_API/USE_NEW_API with USE_LEGACY_API Flip the flag semantics: new API is now the default with zero config. Legacy API requires explicit USE_LEGACY_API=true opt-in. This ensures: - No legacy API calls are made unless explicitly opted in - New implementation is fully exercised by default - Mock mode, historical mode, and direct mode all use new API - Legacy path only reachable via USE_LEGACY_API=true Updated: metrics-util-v2.ts, nuxt.config.ts, docker-compose.yml, .env.example, tests, and all documentation references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: stabilize E2E tests for redesigned github.com tab - Rewrite GitHubTab page object for new AgentModeViewer component - Scope card locators to .github-com-container to avoid false matches - Fix Chat card locator: use v-card-title child selector instead of regex ^Chat$ which failed against multi-line card content - Add explicit waits for github-stats API data in beforeAll - Configure github-tab tests as serial with shared page - Update DashboardPage.gotoGitHubTab() to wait for content load - All 47 E2E tests pass consistently (5/5 runs) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: patch CVE-2026-29063 prototype pollution in immutable Upgrade immutable from 5.0.3 to 5.1.5 (via sass-embedded dependency) to fix high-severity prototype pollution vulnerability in mergeDeep(), mergeDeepWith(), merge(), Map.toJS(), and Map.toObject() APIs. Also patches GHSA-xpqw-6gx7-v673 (svgo DoS via entity expansion). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add sync Docker image to CI/CD workflows - playwright.yml: Add test-docker-sync job to verify Dockerfile.sync builds on PRs and pushes to main - deploy_to_ghcr.yml: Add push_sync_to_ghcr job to build and push sync image as ghcr.io/<repo>-sync on push to main - deploy_to_ghcr_tag_release.yaml: Add push_sync_to_ghcr job to build and push sync image on release with version tags Sync image is published as a separate package (-sync suffix) with the same tag strategy as the main image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: CI test failures - add PostgreSQL, fix test ordering Playwright CI fixes: - Add test-storage-pipeline job with PostgreSQL 16 service container Phase 1 (@seed): seeds DB with mock data via admin/sync API Phase 2 (@storage): verifies dashboard reads from DB - Exclude @seed/@storage from test and test-docker-playwright jobs (these need PostgreSQL, not available in those jobs) - Fix teams-comparison.spec.ts: add serial mode and reorder tests so 'empty state' runs before 'team selection' (shared page state) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: storage pipeline CI - init DB schema when DATABASE_URL is set Root cause: Phase 1 started the server with DATABASE_URL but without ENABLE_HISTORICAL_MODE=true, so the db-init plugin skipped schema creation. The sync API then failed with 'relation metrics does not exist'. Fix: - db-init plugin now also runs when DATABASE_URL is set (not just ENABLE_HISTORICAL_MODE) — prevents this class of bug - Also added ENABLE_HISTORICAL_MODE=true to Phase 1 workflow step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for code scanning alert no. 17: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * fix: exclude /api/_auth/ from GitHub auth middleware The auth middleware was applying GitHub token authentication to /api/_auth/session, causing 500 errors in storage-only mode where no GitHub token is configured. The _auth/session endpoint is handled by nuxt-auth-utils and doesn't need GitHub API headers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: return empty seats in historical mode without auth In historical mode (storage pipeline), the seats endpoint returned 401 when no GitHub token was available, causing the dashboard to fail to render. Now returns empty array instead, allowing the dashboard to display metrics from the database without seats data. Verified: @storage E2E tests pass with this fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add PostgreSQL and sync service to all deployment options All deployment options now provision PostgreSQL and a scheduled sync service for the new Copilot download API: Azure (ARM one-click): - PostgreSQL Flexible Server (Burstable B1ms, 32GB) - Container App Job for daily sync (2 AM UTC) - DATABASE_URL + ENABLE_HISTORICAL_MODE env vars - Portal UI updated with database password section - Updated cost estimate (~$15/month) Azure (AZD Bicep): - New infra/shared/postgresql.bicep module - New infra/app/copilot-metrics-sync.bicep (Container App Job) - Updated main.bicep with PostgreSQL + sync modules - azure.yaml includes sync service definition Kubernetes: - New k8s/deployment.yaml with health probes and DB config - Updated k8s/cronjob.yaml with GHCR image and DATABASE_URL Docker Compose: Already had PostgreSQL (no changes needed) DEPLOYMENT.md: - Replaced docker run examples with docker compose - Added architecture overview (web + db + sync) - Added environment variables reference table - Updated Kubernetes section with full setup instructions - Added legacy API mode note Also fixes seats endpoint returning 401 in historical mode (now returns empty array when no auth available). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump version to 3.0.0 and add CHANGELOG.md Breaking change: new Copilot Usage Metrics API is now the default. Legacy API available via USE_LEGACY_API=true (sunset April 2, 2026). PostgreSQL storage, sync service, and historical mode added. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: require authentication for admin API endpoints Admin endpoints (/api/admin/*) now go through the same auth middleware as other API routes. This ensures OAuth-authenticated deployments properly protect sync trigger and status endpoints. Background sync (scheduled tasks, standalone container) is unaffected as it calls sync functions directly, not via HTTP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: update API version header and add migration banner - Fix X-GitHub-Api-Version from 2022-11-28 to 2026-03-10 for new Copilot Usage Metrics API endpoints (was causing 400 Bad Request) - Fix duplicate API version headers from case-sensitive object keys - Add error body logging for new API debugging - Add v3.0 migration banner to dashboard explaining new permission requirement (Organization Copilot metrics: Read) - Update DEPLOYMENT.md with new Copilot Metrics permission - Update README.md with v3.0 migration note about permissions - Update CHANGELOG.md with permission requirement Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add v3.0 upgrade notice with legacy API shutdown warning Prominent banner at top of README explaining: - Legacy API shutdown date (April 2, 2026) - Last legacy Docker image tag (v2.1.4) - Required GitHub App permission update - Link to PostgreSQL/historical mode setup - Quick upgrade instructions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: karpikpl <3539908+karpikpl@users.noreply.github.com> Co-authored-by: Piotr Karpala <piotrkarpala@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Piotr Karpala <karpik.pl@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
1 parent 72cb47b commit a645278

83 files changed

Lines changed: 32245 additions & 664 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.env

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ NUXT_PUBLIC_IS_DATA_MOCKED=true
66
NUXT_PUBLIC_SCOPE=organization
77

88
# Determines the enterprise or organization name to target API calls.
9-
NUXT_PUBLIC_GITHUB_ORG=octodemo
9+
NUXT_PUBLIC_GITHUB_ORG=cody-test-org
1010

1111
NUXT_PUBLIC_GITHUB_ENT=
1212

@@ -17,7 +17,7 @@ NUXT_PUBLIC_USING_GITHUB_AUTH=false
1717

1818
# Determines the GitHub Personal Access Token to use for API calls.
1919
# Create with scopes copilot, manage_billing:copilot or manage_billing:enterprise, read:enterprise AND read:org
20-
# NUXT_GITHUB_TOKEN=<TOKEN>
20+
NUXT_GITHUB_TOKEN=<TOKEN>
2121

2222
NUXT_SESSION_PASSWORD=something_long_and_random_thats_at_least_32_characters
2323

@@ -27,3 +27,6 @@ NUXT_OAUTH_GITHUB_CLIENT_SECRET=
2727

2828
# to use a corporate proxy
2929
# HTTP_PROXY=http://proxy.company.com:8080
30+
31+
ENABLE_HISTORICAL_MODE=true
32+
NUXT_PUBLIC_USE_NEW_API=true

.env.example

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# ==============================================================================
2+
# GitHub Copilot Metrics Viewer Configuration
3+
# ==============================================================================
4+
5+
# ------------------------------------------------------------------------------
6+
# Core Configuration
7+
# ------------------------------------------------------------------------------
8+
9+
# Determines if mocked data should be used instead of making API calls.
10+
NUXT_PUBLIC_IS_DATA_MOCKED=true
11+
12+
# Determines the scope of the API calls.
13+
# Can be 'enterprise', 'organization', 'team-organization', or 'team-enterprise'
14+
NUXT_PUBLIC_SCOPE=organization
15+
16+
# Determines the enterprise or organization name to target API calls.
17+
NUXT_PUBLIC_GITHUB_ORG=octodemo
18+
NUXT_PUBLIC_GITHUB_ENT=
19+
20+
# Determines the team name if exists to target API calls.
21+
NUXT_PUBLIC_GITHUB_TEAM=
22+
23+
# Enable GitHub OAuth authentication
24+
NUXT_PUBLIC_USING_GITHUB_AUTH=false
25+
26+
# ------------------------------------------------------------------------------
27+
# Authentication
28+
# ------------------------------------------------------------------------------
29+
30+
# GitHub Personal Access Token for API calls
31+
# Required scopes: copilot, manage_billing:copilot, manage_billing:enterprise, read:enterprise, read:org
32+
# NUXT_GITHUB_TOKEN=ghp_your_token_here
33+
34+
# Session password for encrypting user sessions (REQUIRED!)
35+
# Must be at least 32 characters long
36+
NUXT_SESSION_PASSWORD=something_long_and_random_thats_at_least_32_characters
37+
38+
# GitHub OAuth App credentials (if using GitHub auth)
39+
NUXT_OAUTH_GITHUB_CLIENT_ID=
40+
NUXT_OAUTH_GITHUB_CLIENT_SECRET=
41+
42+
# ------------------------------------------------------------------------------
43+
# API Migration Configuration (New Features)
44+
# ------------------------------------------------------------------------------
45+
46+
# Use the LEGACY Copilot Metrics API (deprecated, shuts down April 2, 2026)
47+
# Default is false — the new download-based API is used unless this is explicitly set to true.
48+
# Only set USE_LEGACY_API=true if you have a specific reason to use the old /copilot/metrics endpoint.
49+
USE_LEGACY_API=false
50+
51+
# Enable storage-backed historical mode
52+
# Set to 'true' to query storage for date ranges instead of live API calls
53+
# Requires storage backend configuration (see below)
54+
NUXT_PUBLIC_ENABLE_HISTORICAL_MODE=false
55+
56+
# ------------------------------------------------------------------------------
57+
# Storage Configuration
58+
# ------------------------------------------------------------------------------
59+
60+
# Storage backend for metrics data
61+
# Options: filesystem (dev), postgresql (prod), redis, mongodb
62+
# Configured in nuxt.config.ts nitro.storage section
63+
64+
# For PostgreSQL backend (recommended for production):
65+
# DATABASE_URL=postgresql://user:password@localhost:5432/copilot_metrics
66+
67+
# For Redis backend:
68+
# REDIS_URL=redis://localhost:6379
69+
70+
# For filesystem backend (default, development only):
71+
# Data stored in ./.data/metrics directory (already configured)
72+
73+
# ------------------------------------------------------------------------------
74+
# Data Sync Configuration
75+
# ------------------------------------------------------------------------------
76+
77+
# Enable automatic daily sync (for background scheduler)
78+
# Set to 'true' to enable the scheduled task
79+
SYNC_ENABLED=false
80+
81+
# Cron schedule for sync job (default: 2 AM daily)
82+
# Format: minute hour day month weekday
83+
SYNC_SCHEDULE=0 2 * * *
84+
85+
# Number of days to backfill on first sync
86+
SYNC_BACKFILL_DAYS=28
87+
88+
# Data retention period (days)
89+
SYNC_RETENTION_DAYS=365
90+
91+
# ------------------------------------------------------------------------------
92+
# Network Configuration
93+
# ------------------------------------------------------------------------------
94+
95+
# HTTP Proxy (for corporate environments)
96+
# HTTP_PROXY=http://proxy.company.com:8080
97+
98+
# Custom CA certificate path (for corporate proxies with custom CA)
99+
# CUSTOM_CA_PATH=/path/to/ca-cert.pem
100+
101+
# Server port (default: 80 in Docker, 3000 in dev)
102+
# NITRO_PORT=3000
103+
104+
# ------------------------------------------------------------------------------
105+
# Admin API Configuration
106+
# ------------------------------------------------------------------------------
107+
108+
# ⚠️ SECURITY WARNING: Admin endpoints are NOT authenticated by default!
109+
#
110+
# The /api/admin/* endpoints (sync trigger, status) are currently UNPROTECTED.
111+
# If you expose these endpoints publicly, you MUST secure them via:
112+
# - Reverse proxy authentication (nginx, Apache, etc.)
113+
# - API gateway with auth (AWS API Gateway, Azure APIM, etc.)
114+
# - Firewall rules restricting to internal networks only
115+
# - VPN or private network access only
116+
#
117+
# ADMIN_API_SECRET is reserved for future native authentication but has NO EFFECT
118+
# in the current version. Do NOT rely on this variable for security.
119+
# ADMIN_API_SECRET=your_secure_random_secret_here
120+
121+
# ==============================================================================
122+
# Migration Notes
123+
# ==============================================================================
124+
125+
# The legacy GitHub Copilot Metrics API will shut down on April 2, 2026.
126+
# This application uses the new download-based API by default. No configuration needed.
127+
#
128+
# API modes:
129+
# Default (new API) — download-based reports, richer data, model/feature breakdowns
130+
# Legacy (opt-in) — set USE_LEGACY_API=true to use deprecated /copilot/metrics
131+
#
132+
# Recommended setup:
133+
# 1. Test with mock data (IS_DATA_MOCKED=true)
134+
# 2. Set real GitHub token (NUXT_GITHUB_TOKEN)
135+
# 3. Configure PostgreSQL + enable historical mode (ENABLE_HISTORICAL_MODE=true)
136+
# 4. Enable sync (SYNC_ENABLED=true)
137+
#
138+
# See MIGRATION_GUIDE.md for full details
139+
# ==============================================================================

.github/workflows/deploy_to_ghcr.yml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,45 @@ jobs:
5555
labels: ${{ steps.meta.outputs.labels }}
5656
push: true
5757

58+
push_sync_to_ghcr:
59+
runs-on: ubuntu-latest
60+
steps:
61+
- name: Checkout GitHub Action
62+
uses: actions/checkout@v4
63+
64+
- name: Login to GitHub Container Registry
65+
uses: docker/login-action@v3
66+
with:
67+
registry: ghcr.io
68+
username: ${{ github.actor }}
69+
password: ${{ secrets.GITHUB_TOKEN }}
70+
71+
- name: Extract metadata
72+
id: meta
73+
uses: docker/metadata-action@v5
74+
with:
75+
images: |
76+
ghcr.io/${{ github.repository }}-sync
77+
labels: |
78+
org.opencontainers.image.title=copilot-metrics-viewer-sync
79+
org.opencontainers.image.description=Sync service for GitHub Copilot metrics data
80+
org.opencontainers.image.source=${{ github.repository }}
81+
tags: |
82+
type=sha
83+
type=ref,event=branch
84+
type=ref,event=pr
85+
type=semver,pattern={{version}}
86+
type=semver,pattern={{major}}.{{minor}}
87+
88+
- name: Build and push sync image
89+
uses: docker/build-push-action@v5
90+
with:
91+
context: .
92+
file: Dockerfile.sync
93+
tags: ${{ steps.meta.outputs.tags }}
94+
labels: ${{ steps.meta.outputs.labels }}
95+
push: true
96+
5897
deploy:
5998
runs-on: ubuntu-latest
6099
needs: push_to_ghcr

.github/workflows/deploy_to_ghcr_tag_release.yaml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,3 +102,45 @@ jobs:
102102
tags: ${{ steps.meta.outputs.tags }}
103103
labels: ${{ steps.meta.outputs.labels }}
104104
push: true
105+
106+
push_sync_to_ghcr:
107+
runs-on: ubuntu-latest
108+
steps:
109+
- name: Checkout GitHub Action
110+
uses: actions/checkout@v4
111+
with:
112+
ref: ${{ github.event.release.tag_name }}
113+
114+
- name: Login to GitHub Container Registry
115+
uses: docker/login-action@v3
116+
with:
117+
registry: ghcr.io
118+
username: ${{ github.actor }}
119+
password: ${{ secrets.GITHUB_TOKEN }}
120+
121+
- name: Extract metadata
122+
id: meta
123+
uses: docker/metadata-action@v5
124+
with:
125+
images: |
126+
ghcr.io/${{ github.repository }}-sync
127+
labels: |
128+
org.opencontainers.image.title=copilot-metrics-viewer-sync
129+
org.opencontainers.image.description=Sync service for GitHub Copilot metrics data
130+
org.opencontainers.image.source=${{ github.repository }}
131+
tags: |
132+
type=raw,value=latest,enable=true
133+
type=sha
134+
type=ref,event=branch
135+
type=ref,event=pr
136+
type=semver,pattern={{version}}
137+
type=semver,pattern={{major}}.{{minor}}
138+
139+
- name: Build and push sync image
140+
uses: docker/build-push-action@v6
141+
with:
142+
context: .
143+
file: Dockerfile.sync
144+
tags: ${{ steps.meta.outputs.tags }}
145+
labels: ${{ steps.meta.outputs.labels }}
146+
push: true

.github/workflows/playwright.yml

Lines changed: 92 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
name: Playwright Tests
2+
permissions:
3+
contents: read
24
on:
35
push:
46
branches: [ main, master ]
@@ -19,6 +21,20 @@ jobs:
1921
load: true
2022
tags: ui:test
2123

24+
test-docker-sync:
25+
timeout-minutes: 15
26+
runs-on: ubuntu-latest
27+
steps:
28+
- uses: actions/checkout@v4
29+
- name: Set up Docker Buildx
30+
uses: docker/setup-buildx-action@v3
31+
- name: Build sync image
32+
uses: docker/build-push-action@v6
33+
with:
34+
file: Dockerfile.sync
35+
load: true
36+
tags: sync:test
37+
2238
test:
2339
timeout-minutes: 10
2440
permissions:
@@ -44,7 +60,7 @@ jobs:
4460
NUXT_SESSION_PASSWORD=foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo \
4561
NUXT_PUBLIC_GITHUB_ORG=octodemo \
4662
NUXT_PUBLIC_IS_DATA_MOCKED=true \
47-
RUN_COMMAND='npm run preview' npm run test:e2e
63+
RUN_COMMAND='npm run preview' npx playwright test --grep-invert "@seed|@storage"
4864
- uses: actions/upload-artifact@v4
4965
if: ${{ !cancelled() }}
5066
with:
@@ -72,10 +88,84 @@ jobs:
7288
-e NUXT_SESSION_PASSWORD=foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo \
7389
-e NUXT_PUBLIC_GITHUB_ORG=octodemo \
7490
-e NUXT_PUBLIC_IS_DATA_MOCKED=true \
75-
app:pw
91+
app:pw npx playwright test --grep-invert "@seed|@storage"
7692
- uses: actions/upload-artifact@v4
7793
if: always()
7894
with:
7995
name: playwright-report-docker
8096
path: test-results-docker/
8197
retention-days: 30
98+
99+
test-storage-pipeline:
100+
timeout-minutes: 15
101+
runs-on: ubuntu-latest
102+
services:
103+
postgres:
104+
image: postgres:16-alpine
105+
env:
106+
POSTGRES_USER: testuser
107+
POSTGRES_PASSWORD: testpass
108+
POSTGRES_DB: copilot_metrics
109+
ports:
110+
- 5432:5432
111+
options: >-
112+
--health-cmd pg_isready
113+
--health-interval 10s
114+
--health-timeout 5s
115+
--health-retries 5
116+
steps:
117+
- uses: actions/checkout@v4
118+
- uses: actions/setup-node@v4
119+
with:
120+
node-version: lts/*
121+
- name: Install dependencies
122+
run: npm ci
123+
- name: Build the app
124+
run: npm run build
125+
- name: Install Playwright Browsers
126+
run: npx playwright install --with-deps chromium
127+
- name: "Phase 1: Seed DB with mock data"
128+
run: |
129+
NUXT_SESSION_PASSWORD=foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo \
130+
NUXT_PUBLIC_GITHUB_ORG=storage-test-org \
131+
NUXT_PUBLIC_IS_DATA_MOCKED=true \
132+
ENABLE_HISTORICAL_MODE=true \
133+
DATABASE_URL=postgresql://testuser:testpass@localhost:5432/copilot_metrics \
134+
node .output/server/index.mjs &
135+
SERVER_PID=$!
136+
137+
# Wait for server to be ready
138+
for i in $(seq 1 30); do
139+
curl -sf http://localhost:3000/api/health > /dev/null 2>&1 && break
140+
sleep 1
141+
done
142+
143+
CI=true npx playwright test --grep "@seed" --project=chromium
144+
145+
kill $SERVER_PID
146+
wait $SERVER_PID 2>/dev/null || true
147+
- name: "Phase 2: Verify dashboard reads from DB"
148+
run: |
149+
NUXT_SESSION_PASSWORD=foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo-foo \
150+
NUXT_PUBLIC_GITHUB_ORG=storage-test-org \
151+
NUXT_PUBLIC_IS_DATA_MOCKED=false \
152+
ENABLE_HISTORICAL_MODE=true \
153+
DATABASE_URL=postgresql://testuser:testpass@localhost:5432/copilot_metrics \
154+
node .output/server/index.mjs &
155+
SERVER_PID=$!
156+
157+
for i in $(seq 1 30); do
158+
curl -sf http://localhost:3000/api/health > /dev/null 2>&1 && break
159+
sleep 1
160+
done
161+
162+
CI=true npx playwright test --grep "@storage" --project=chromium
163+
164+
kill $SERVER_PID
165+
wait $SERVER_PID 2>/dev/null || true
166+
- uses: actions/upload-artifact@v4
167+
if: ${{ !cancelled() }}
168+
with:
169+
name: playwright-report-storage
170+
path: playwright-report/
171+
retention-days: 30

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,4 +29,5 @@ playwright-logs
2929

3030
# Test results
3131
test-results
32+
test-results-old
3233
results.xml

.nuxtrc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
setups.@nuxt/test-utils="3.23.0"

0 commit comments

Comments
 (0)