Commit 864be8f
authored
Foundation: Migrate from NX/Express/MongoDB/Next.js to Turborepo/Hono/PostgreSQL/Vite (#83)
* feat: replace NX/Express/MongoDB/Next.js with Turborepo/Hono/Drizzle/Vite foundation
Clean-slate rewrite of the monorepo foundation to align with the
authoritative .kiro/ steering docs:
- Replace NX with Turborepo for monorepo orchestration
- Move packages under packages/ directory (backend, frontend, shared, openapi)
- Replace Express with Hono on Bun.serve() (minimal entry point)
- Replace MongoDB/Mongoose deps with PostgreSQL/Drizzle deps
- Replace Next.js with Vite + React 19
- Replace Jest with bun:test
- Update tsconfig.base.json for ESNext modules + bundler resolution
- Update biome.json (remove Next.js overrides)
- Create .env.example with PostgreSQL, Redis, MinIO, JWT config
- All pipelines pass: turbo build, lint, type-check, test
* feat: add Drizzle schema, drizzle-zod schemas, Docker Compose infrastructure
Task 2 — Infrastructure and development environment:
- docker-compose.yml with PostgreSQL 16, Redis 7, MinIO (S3-compatible)
- Drizzle table definitions in shared/src/db/schema.ts for all 15 tables:
Identity (users, projects, project_users), Agents (agents, agent_errors,
operating_systems), Resources (hash_lists, hash_items, hash_types,
word_lists, rule_lists, mask_lists), Campaigns (campaigns, attacks, tasks)
- Zod schemas generated from Drizzle via drizzle-zod (insert + select)
- TypeScript types inferred via z.infer (no manual duplication)
- Custom API schemas: loginRequest, createCampaignRequest, agentHeartbeat
- All pipelines pass: build, lint, type-check, test
* feat: add backend foundation — Hono app, Drizzle connection, Redis/BullMQ, MinIO S3
Set up the backend package with Hono on Bun.serve(), Pino structured logging,
CORS/security middleware, health check endpoint, Drizzle ORM connection,
ioredis + BullMQ queues, and S3/MinIO storage client. Includes Zod-validated
env config, request-id middleware, and unit tests for health/404 endpoints.
* feat: implement authentication and authorization system
Add AuthService with JWT tokens (jose), Bun.password bcrypt hashing,
and session management via HttpOnly cookies. Create requireSession and
requireAgentToken middleware using Hono HTTPException pattern, plus
RBAC middleware (requireRole, requireProjectAccess) for project-scoped
permission checks. Wire auth routes at /api/v1/dashboard/auth with
login, logout, and /me endpoints using @hono/zod-validator.
Includes 15 passing tests covering password hashing, JWT lifecycle,
session/agent middleware, and health checks.
* feat: add project management service and dashboard API routes
Create ProjectService with CRUD operations, membership management, and
role updates using Drizzle queries. Wire project routes at
/api/v1/dashboard/projects with session auth, Zod validation, and
admin-only role checks for mutations. Supports listing user projects,
creating projects (auto-adds creator as admin), updating project info,
and managing project members with role assignment.
* feat: add agent management system with Agent API and OpenAPI spec
Create AgentService with token authentication, heartbeat processing,
capability/hardware profile tracking, and error logging. Implement
Agent API endpoints at /api/v1/agent/* (sessions, heartbeat, tasks/next,
tasks/:id/report, errors) with JWT-authenticated routes. Add Dashboard
agent routes at /api/v1/dashboard/agents for listing, detail, update,
and error retrieval. Include OpenAPI 3.1 specification for the Agent API
contract at packages/openapi/agent-api.yaml.
* feat: add resource management service and dashboard API routes
Implements hash list, wordlist, rulelist, and masklist CRUD with file
upload to MinIO. Uses a generic resource factory pattern for the three
list types to avoid duplication.
* feat: add hash analysis service with type detection and API endpoint
Implements pattern-based hash type identification covering 25+ formats
(bcrypt, MD5, SHA family, NTLM, Kerberos, etc.) with confidence scoring.
Structured formats rank higher than ambiguous raw hex. Includes 12 unit
tests for detection accuracy.
* feat: add campaign orchestration with DAG validation and lifecycle management
Implements campaign CRUD, attack management, lifecycle state machine
(draft → running → paused/completed/cancelled), and DAG cycle detection
using Kahn's algorithm. Includes 9 unit tests for DAG validation.
* feat: add task distribution system with keyspace partitioning and retry logic
Implements task generation from attack keyspace, agent-scoped assignment
via PostgreSQL transactions, progress tracking with cracked hash ingestion,
retry logic with max 3 attempts, and stale task reassignment. Wires up
Agent API /tasks/next and /tasks/:id/report endpoints.
* feat: add real-time event system with WebSocket streaming
Implements EventService with project-scoped broadcasting, per-type
throttling (250ms), and convenience emitters for agent status, campaign
lifecycle, task updates, and crack results. WebSocket endpoint at
/events/stream with token authentication and subscription filtering.
* feat: add frontend foundation with React 19, routing, auth, and layout
Set up the frontend SPA with React Router v7, TanStack Query v5, and
Zustand. Includes login page with React Hook Form + Zod validation,
protected route guard, sidebar with project selector, dashboard with
placeholder cards, and API client. Uses Tailwind v4 via @tailwindcss/vite
plugin.
* feat: add dashboard monitoring UI with real-time WebSocket events
Implement WebSocket client hook with exponential backoff reconnection,
TanStack Query hooks for dashboard stats/agents/campaigns, and live
connection indicator. Add agent list with status filtering, agent detail
page with hardware info and error log, and stat cards wired to real-time
query invalidation.
* feat: add campaign wizard, campaign/attack management, and resource UI
Campaign creation wizard with 3-step flow (basic info, attacks, review)
backed by Zustand state. Campaign list with status filtering, detail view
with lifecycle controls (start/pause/stop/cancel), and attack table.
Resource browser with tabbed interface for hash lists, wordlists,
rulelists, masklists, plus hash type detection with confidence bars.
* Add data migration tooling for CipherSwarm → HashHive import
NDJSON-based migration script that reads Rails/MongoDB exports and imports
into PostgreSQL via Drizzle with ID mapping, batch inserts, and validation.
* Add testing infrastructure, contract tests, and frontend component tests
- Backend: test fixtures/factories, Agent API + Dashboard API contract tests
- Frontend: happy-dom setup, Testing Library utils, StatusBadge + StatCard tests
- Playwright config for E2E tests
- Fix HTTPException handling in onError (was converting 401→500)
71 tests passing (58 backend + 13 frontend).
* Add deployment infrastructure with Docker, compose, and operational scripts
- Multi-stage Dockerfiles for backend (Bun) and frontend (nginx)
- Production docker-compose with health checks and env-based secrets
- Database backup script with 30-day pruning
- Deployment script (build/up/down/migrate/status/logs)
- Graceful shutdown handler (SIGTERM/SIGINT) in backend
- nginx config with SPA routing, API proxy, and WebSocket support
* Add integration smoke tests and complete final validation (Task 23)
Integration tests validate the full request/response cycle:
- Health check with field validation
- Agent API auth enforcement and schema validation
- Dashboard API auth guards and hash type detection
- Cross-cutting: request IDs, 404 handling, security headers
81 tests passing across backend (68) and frontend (13).
* docs: add TypeScript strictness, Hono error handling, and testing gotchas to AGENTS.md
* chore: remove MERN_GUIDANCE.md as it is no longer relevant
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* docs: make .kiro/ authoritative in AGENTS.md, remove stale MERN_GUIDANCE.md reference
* feat: Implement Dashboard & Real-Time Monitoring UI with stat cards and WebSocket integration
feat: Add Login & Project Selection UI with authentication and project selector logic
feat: Configure MinIO Storage & File Management for binary artifacts with environment-driven setup
feat: Implement Project Selection & User Authentication API with JWT and RBAC enforcement
feat: Establish Real-Time Events & WebSocket Infrastructure for dashboard updates
feat: Create Resource Management API with async hash list parsing and resource CRUD operations
feat: Develop Resource Management UI with file upload and hash type detection features
feat: Implement Results Analysis & Export UI with filters, search, and CSV export functionality
feat: Add Task Distribution & Assignment with strict task assignment and hybrid task generation
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* feat: Add hookify configurations to block banned packages, npm/yarn/pnpm usage, and wrong test runners
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* feat: Update user roles and authentication methods in documentation and architecture specifications
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* feat: Implement BullMQ queue architecture with Redis integration
Add async job processing layer using BullMQ with three queues:
- task-distribution: generates tasks for campaign attacks with priority support
- hash-list-parsing: downloads S3 files, batch-inserts hash items (1000/batch)
- heartbeat-monitor: repeatable 60s job reassigning stale agent tasks
QueueManager runs workers in-process alongside Hono server with non-blocking
init (server works if Redis is down). Health endpoint now reports Redis status
and per-queue depths. Services enqueue jobs via module-level context accessor
with graceful degradation when Redis is unavailable.
* fix: Route task-generation jobs through dedicated job queue
Campaign start now enqueues to `jobs:task-generation` instead of the
per-priority task queues (`tasks:high/normal/low`). The worker consumes
from the single dedicated queue with BullMQ priority ordering. Priority
task queues remain available for downstream task dispatch.
* fix: Route task-generation to priority queues and add Redis reconnect recovery
Replace the single TASK_GENERATION queue with priority-based routing
(TASKS_HIGH/NORMAL/LOW) so campaign priority actually affects processing
order. Add getTaskQueueForPriority() helper and a worker entrypoint that
spawns one task-generator per priority queue. Extract QueueManager queue
creation into createQueues() and listen for the Redis ready event so
queues are initialized after a failed initial connect without requiring
a process restart.
* feat: Implement Login & Project Selection UI with authentication and auto-select logic
- Added login page with email/password form and validation using React Hook Form and Zod.
- Created project selector page with project cards and auto-select logic for single projects.
- Implemented "remember last project" preference and updated protected route wrapper.
- Added logout functionality in user menu.
feat: Configure MinIO Storage & File Management for binary artifacts
- Configured MinIO client with environment-driven bucket names.
- Implemented file upload to MinIO and generated presigned URLs for downloads.
- Added health checks for MinIO to the /health endpoint.
feat: Develop Project Selection & User Authentication API with JWT sessions
- Implemented user login/logout with JWT and HttpOnly sessions.
- Added project selector endpoint with auto-select logic and RBAC enforcement.
- Ensured server-side scoping for dashboard API.
feat: Establish Real-Time Events & WebSocket Infrastructure for dashboard updates
- Implemented in-memory WebSocket broadcasting and automatic polling fallback.
- Added WebSocket authentication and project-scoped event filtering.
feat: Create Resource Management API with async hash list parsing
- Implemented resource upload endpoints and async hash list parsing with idempotency.
- Added resource CRUD endpoints and hash type detection.
feat: Develop Resource Management UI with file upload and hash type detection
- Implemented tabbed resource management page with drag-and-drop file upload.
- Added resource listing and delete confirmation functionality.
feat: Implement Results API & CSV Export for cracked hashes
- Added project-scoped results API and CSV export functionality.
- Implemented result attribution for cracked hashes.
feat: Create Results Analysis & Export UI for comprehensive result analysis
- Developed global results page with filters, search, and CSV export.
- Implemented campaign-specific and hash list results views.
feat: Establish Task Distribution & Assignment with strict assignment logic
- Implemented strict task assignment with project scoping and capability matching.
- Added hybrid task generation and task reassignment logic.
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* feat: Integrate WebSocket support with authentication and project authorization in event routes
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* feat: Refactor agent authentication to use pre-shared tokens and remove session endpoint
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
* chore: Remove unused frontend dependencies
Remove @radix-ui/react-slot, class-variance-authority, lucide-react,
autoprefixer, and postcss — none are imported in the frontend source.
Also remove phantom tailwindcss-animate import from tailwind.config.ts
(was never installed and no animation classes are in use).
* feat: Fix architecture review issues across backend, frontend, and shared packages
Add missing worker-jobs entrypoint, PostgreSQL health check, database indexes
on hot query paths, RBAC middleware on all dashboard routes with unified
projectId query parameter, shared request/response schemas, error boundary
with 404 handler, 401 session expiry interceptor, scoped WebSocket query
invalidation, campaign wizard cleanup, and env.example modernization.
* perf: Add code splitting, connection pooling, missing indexes, and deduplicate event handlers
- Route-level code splitting with React.lazy + Suspense for all page components
- Configure postgres connection pool (max: 20, idle_timeout: 30s, connect_timeout: 10s)
- Add missing DB indexes: hash_items(cracked_at), hash_lists(status), tasks(status, campaign_id)
- Disable TanStack Query refetchOnWindowFocus and refetchOnReconnect defaults
- Skip request logger on /health endpoint to reduce log noise
- Deduplicate WebSocket event handlers: consolidate all query invalidation in useEvents hook
* refactor: Simplify RBAC, resource routes, and frontend hooks (-101 lines)
- Extract shared checkMembership() in RBAC middleware to remove duplication
- Remove 12 passthrough resource service wrappers, export generic functions
- Simplify route factory from 5 params to (prefix, table)
- Consolidate 3 identical resource hooks into parameterized useResourceList()
- Replace WebSocket event if/else chain with lookup map
* chore: Remove stale NX, MongoDB, and Next.js references from config and docs
Delete nx.json, rewrite justfile for Turborepo, update CI workflows
to remove Testcontainers/mongo pre-pulls, rewrite CONTRIBUTING.md and
copilot-instructions.md for current stack, remove .next/ from
pre-commit excludes.
* fix: Address CodeRabbit review findings across security, bugs, and config
Security: timing-safe login (dummy bcrypt on miss), Content-Disposition
header injection sanitization, RBAC on project GET routes, atomic
createProject transaction, cross-campaign attack verification.
Backend bugs: await Redis disconnect in workers, null-safe job.data
access, position penalty filter logic, onConflictDoNothing for hash
upserts, deferred event emission for running campaigns, lastEmitTimes
Map cleanup.
Frontend bugs: stale campaignId in campaign wizard, login loading state,
mutation cache invalidation using variables.projectId, malformed query
string in dashboard stats, 204 No Content handling, stabilized WS types
array.
Config: removed duplicate pre-commit hooks, corrected OpenAPI
bearerFormat from JWT to token.
* docs: Add TanStack Query and Drizzle mock patterns to AGENTS.md
* fix: Address PR bot feedback across security, correctness, and config
- Fix MinIO healthcheck to use curl instead of mc (not in image)
- Validate JWT claim types before trusting payload
- Fix agent auth to reject error state, not non-active status
- Add campaign existence check on GET /:id/attacks
- Fix Oracle hash regex to be case-insensitive
- Fix CSS border-color missing hsl() wrapper
- Fix dashboard stats using server totals instead of client-side limit=0
- Fix truthy check on limit/offset dropping 0 values
- Add resource_update event type to WS invalidation map
- Throw after 401 redirect to prevent control flow continuation
- Add Suspense fallback loading indicator
- Remove token from login response body (HttpOnly cookie only)
- Update AGENTS.md: remove stale sessions endpoint, fix doc path
- Update auth docs with correct RBAC role names
---------
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>1 parent 81b72e4 commit 864be8f
241 files changed
Lines changed: 14294 additions & 12438 deletions
File tree
- .claude
- instincts
- skills
- .github
- workflows
- .kiro
- specs/mern-migration
- steering
- backend
- src
- config
- db
- examples
- middleware
- models
- routes
- services
- types
- utils
- tests
- helpers
- integration
- unit
- docs
- frontend
- app
- e2e
- openapi
- packages
- backend
- src
- config
- db
- middleware
- queue
- workers
- routes
- agent
- dashboard
- scripts
- services
- tests
- integration
- unit
- workers
- frontend
- src
- components/features
- hooks
- lib
- pages
- stores
- tests
- components
- openapi
- scripts
- spec
- specs
- tickets
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
0 commit comments