Skip to content

Commit 864be8f

Browse files
authored
Foundation: Migrate from NX/Express/MongoDB/Next.js to Turborepo/Hono/PostgreSQL/Vite (#83)
* feat: replace NX/Express/MongoDB/Next.js with Turborepo/Hono/Drizzle/Vite foundation Clean-slate rewrite of the monorepo foundation to align with the authoritative .kiro/ steering docs: - Replace NX with Turborepo for monorepo orchestration - Move packages under packages/ directory (backend, frontend, shared, openapi) - Replace Express with Hono on Bun.serve() (minimal entry point) - Replace MongoDB/Mongoose deps with PostgreSQL/Drizzle deps - Replace Next.js with Vite + React 19 - Replace Jest with bun:test - Update tsconfig.base.json for ESNext modules + bundler resolution - Update biome.json (remove Next.js overrides) - Create .env.example with PostgreSQL, Redis, MinIO, JWT config - All pipelines pass: turbo build, lint, type-check, test * feat: add Drizzle schema, drizzle-zod schemas, Docker Compose infrastructure Task 2 — Infrastructure and development environment: - docker-compose.yml with PostgreSQL 16, Redis 7, MinIO (S3-compatible) - Drizzle table definitions in shared/src/db/schema.ts for all 15 tables: Identity (users, projects, project_users), Agents (agents, agent_errors, operating_systems), Resources (hash_lists, hash_items, hash_types, word_lists, rule_lists, mask_lists), Campaigns (campaigns, attacks, tasks) - Zod schemas generated from Drizzle via drizzle-zod (insert + select) - TypeScript types inferred via z.infer (no manual duplication) - Custom API schemas: loginRequest, createCampaignRequest, agentHeartbeat - All pipelines pass: build, lint, type-check, test * feat: add backend foundation — Hono app, Drizzle connection, Redis/BullMQ, MinIO S3 Set up the backend package with Hono on Bun.serve(), Pino structured logging, CORS/security middleware, health check endpoint, Drizzle ORM connection, ioredis + BullMQ queues, and S3/MinIO storage client. Includes Zod-validated env config, request-id middleware, and unit tests for health/404 endpoints. * feat: implement authentication and authorization system Add AuthService with JWT tokens (jose), Bun.password bcrypt hashing, and session management via HttpOnly cookies. Create requireSession and requireAgentToken middleware using Hono HTTPException pattern, plus RBAC middleware (requireRole, requireProjectAccess) for project-scoped permission checks. Wire auth routes at /api/v1/dashboard/auth with login, logout, and /me endpoints using @hono/zod-validator. Includes 15 passing tests covering password hashing, JWT lifecycle, session/agent middleware, and health checks. * feat: add project management service and dashboard API routes Create ProjectService with CRUD operations, membership management, and role updates using Drizzle queries. Wire project routes at /api/v1/dashboard/projects with session auth, Zod validation, and admin-only role checks for mutations. Supports listing user projects, creating projects (auto-adds creator as admin), updating project info, and managing project members with role assignment. * feat: add agent management system with Agent API and OpenAPI spec Create AgentService with token authentication, heartbeat processing, capability/hardware profile tracking, and error logging. Implement Agent API endpoints at /api/v1/agent/* (sessions, heartbeat, tasks/next, tasks/:id/report, errors) with JWT-authenticated routes. Add Dashboard agent routes at /api/v1/dashboard/agents for listing, detail, update, and error retrieval. Include OpenAPI 3.1 specification for the Agent API contract at packages/openapi/agent-api.yaml. * feat: add resource management service and dashboard API routes Implements hash list, wordlist, rulelist, and masklist CRUD with file upload to MinIO. Uses a generic resource factory pattern for the three list types to avoid duplication. * feat: add hash analysis service with type detection and API endpoint Implements pattern-based hash type identification covering 25+ formats (bcrypt, MD5, SHA family, NTLM, Kerberos, etc.) with confidence scoring. Structured formats rank higher than ambiguous raw hex. Includes 12 unit tests for detection accuracy. * feat: add campaign orchestration with DAG validation and lifecycle management Implements campaign CRUD, attack management, lifecycle state machine (draft → running → paused/completed/cancelled), and DAG cycle detection using Kahn's algorithm. Includes 9 unit tests for DAG validation. * feat: add task distribution system with keyspace partitioning and retry logic Implements task generation from attack keyspace, agent-scoped assignment via PostgreSQL transactions, progress tracking with cracked hash ingestion, retry logic with max 3 attempts, and stale task reassignment. Wires up Agent API /tasks/next and /tasks/:id/report endpoints. * feat: add real-time event system with WebSocket streaming Implements EventService with project-scoped broadcasting, per-type throttling (250ms), and convenience emitters for agent status, campaign lifecycle, task updates, and crack results. WebSocket endpoint at /events/stream with token authentication and subscription filtering. * feat: add frontend foundation with React 19, routing, auth, and layout Set up the frontend SPA with React Router v7, TanStack Query v5, and Zustand. Includes login page with React Hook Form + Zod validation, protected route guard, sidebar with project selector, dashboard with placeholder cards, and API client. Uses Tailwind v4 via @tailwindcss/vite plugin. * feat: add dashboard monitoring UI with real-time WebSocket events Implement WebSocket client hook with exponential backoff reconnection, TanStack Query hooks for dashboard stats/agents/campaigns, and live connection indicator. Add agent list with status filtering, agent detail page with hardware info and error log, and stat cards wired to real-time query invalidation. * feat: add campaign wizard, campaign/attack management, and resource UI Campaign creation wizard with 3-step flow (basic info, attacks, review) backed by Zustand state. Campaign list with status filtering, detail view with lifecycle controls (start/pause/stop/cancel), and attack table. Resource browser with tabbed interface for hash lists, wordlists, rulelists, masklists, plus hash type detection with confidence bars. * Add data migration tooling for CipherSwarm → HashHive import NDJSON-based migration script that reads Rails/MongoDB exports and imports into PostgreSQL via Drizzle with ID mapping, batch inserts, and validation. * Add testing infrastructure, contract tests, and frontend component tests - Backend: test fixtures/factories, Agent API + Dashboard API contract tests - Frontend: happy-dom setup, Testing Library utils, StatusBadge + StatCard tests - Playwright config for E2E tests - Fix HTTPException handling in onError (was converting 401→500) 71 tests passing (58 backend + 13 frontend). * Add deployment infrastructure with Docker, compose, and operational scripts - Multi-stage Dockerfiles for backend (Bun) and frontend (nginx) - Production docker-compose with health checks and env-based secrets - Database backup script with 30-day pruning - Deployment script (build/up/down/migrate/status/logs) - Graceful shutdown handler (SIGTERM/SIGINT) in backend - nginx config with SPA routing, API proxy, and WebSocket support * Add integration smoke tests and complete final validation (Task 23) Integration tests validate the full request/response cycle: - Health check with field validation - Agent API auth enforcement and schema validation - Dashboard API auth guards and hash type detection - Cross-cutting: request IDs, 404 handling, security headers 81 tests passing across backend (68) and frontend (13). * docs: add TypeScript strictness, Hono error handling, and testing gotchas to AGENTS.md * chore: remove MERN_GUIDANCE.md as it is no longer relevant Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * docs: make .kiro/ authoritative in AGENTS.md, remove stale MERN_GUIDANCE.md reference * feat: Implement Dashboard & Real-Time Monitoring UI with stat cards and WebSocket integration feat: Add Login & Project Selection UI with authentication and project selector logic feat: Configure MinIO Storage & File Management for binary artifacts with environment-driven setup feat: Implement Project Selection & User Authentication API with JWT and RBAC enforcement feat: Establish Real-Time Events & WebSocket Infrastructure for dashboard updates feat: Create Resource Management API with async hash list parsing and resource CRUD operations feat: Develop Resource Management UI with file upload and hash type detection features feat: Implement Results Analysis & Export UI with filters, search, and CSV export functionality feat: Add Task Distribution & Assignment with strict task assignment and hybrid task generation Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat: Add hookify configurations to block banned packages, npm/yarn/pnpm usage, and wrong test runners Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat: Update user roles and authentication methods in documentation and architecture specifications Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat: Implement BullMQ queue architecture with Redis integration Add async job processing layer using BullMQ with three queues: - task-distribution: generates tasks for campaign attacks with priority support - hash-list-parsing: downloads S3 files, batch-inserts hash items (1000/batch) - heartbeat-monitor: repeatable 60s job reassigning stale agent tasks QueueManager runs workers in-process alongside Hono server with non-blocking init (server works if Redis is down). Health endpoint now reports Redis status and per-queue depths. Services enqueue jobs via module-level context accessor with graceful degradation when Redis is unavailable. * fix: Route task-generation jobs through dedicated job queue Campaign start now enqueues to `jobs:task-generation` instead of the per-priority task queues (`tasks:high/normal/low`). The worker consumes from the single dedicated queue with BullMQ priority ordering. Priority task queues remain available for downstream task dispatch. * fix: Route task-generation to priority queues and add Redis reconnect recovery Replace the single TASK_GENERATION queue with priority-based routing (TASKS_HIGH/NORMAL/LOW) so campaign priority actually affects processing order. Add getTaskQueueForPriority() helper and a worker entrypoint that spawns one task-generator per priority queue. Extract QueueManager queue creation into createQueues() and listen for the Redis ready event so queues are initialized after a failed initial connect without requiring a process restart. * feat: Implement Login & Project Selection UI with authentication and auto-select logic - Added login page with email/password form and validation using React Hook Form and Zod. - Created project selector page with project cards and auto-select logic for single projects. - Implemented "remember last project" preference and updated protected route wrapper. - Added logout functionality in user menu. feat: Configure MinIO Storage & File Management for binary artifacts - Configured MinIO client with environment-driven bucket names. - Implemented file upload to MinIO and generated presigned URLs for downloads. - Added health checks for MinIO to the /health endpoint. feat: Develop Project Selection & User Authentication API with JWT sessions - Implemented user login/logout with JWT and HttpOnly sessions. - Added project selector endpoint with auto-select logic and RBAC enforcement. - Ensured server-side scoping for dashboard API. feat: Establish Real-Time Events & WebSocket Infrastructure for dashboard updates - Implemented in-memory WebSocket broadcasting and automatic polling fallback. - Added WebSocket authentication and project-scoped event filtering. feat: Create Resource Management API with async hash list parsing - Implemented resource upload endpoints and async hash list parsing with idempotency. - Added resource CRUD endpoints and hash type detection. feat: Develop Resource Management UI with file upload and hash type detection - Implemented tabbed resource management page with drag-and-drop file upload. - Added resource listing and delete confirmation functionality. feat: Implement Results API & CSV Export for cracked hashes - Added project-scoped results API and CSV export functionality. - Implemented result attribution for cracked hashes. feat: Create Results Analysis & Export UI for comprehensive result analysis - Developed global results page with filters, search, and CSV export. - Implemented campaign-specific and hash list results views. feat: Establish Task Distribution & Assignment with strict assignment logic - Implemented strict task assignment with project scoping and capability matching. - Added hybrid task generation and task reassignment logic. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat: Integrate WebSocket support with authentication and project authorization in event routes Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * feat: Refactor agent authentication to use pre-shared tokens and remove session endpoint Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io> * chore: Remove unused frontend dependencies Remove @radix-ui/react-slot, class-variance-authority, lucide-react, autoprefixer, and postcss — none are imported in the frontend source. Also remove phantom tailwindcss-animate import from tailwind.config.ts (was never installed and no animation classes are in use). * feat: Fix architecture review issues across backend, frontend, and shared packages Add missing worker-jobs entrypoint, PostgreSQL health check, database indexes on hot query paths, RBAC middleware on all dashboard routes with unified projectId query parameter, shared request/response schemas, error boundary with 404 handler, 401 session expiry interceptor, scoped WebSocket query invalidation, campaign wizard cleanup, and env.example modernization. * perf: Add code splitting, connection pooling, missing indexes, and deduplicate event handlers - Route-level code splitting with React.lazy + Suspense for all page components - Configure postgres connection pool (max: 20, idle_timeout: 30s, connect_timeout: 10s) - Add missing DB indexes: hash_items(cracked_at), hash_lists(status), tasks(status, campaign_id) - Disable TanStack Query refetchOnWindowFocus and refetchOnReconnect defaults - Skip request logger on /health endpoint to reduce log noise - Deduplicate WebSocket event handlers: consolidate all query invalidation in useEvents hook * refactor: Simplify RBAC, resource routes, and frontend hooks (-101 lines) - Extract shared checkMembership() in RBAC middleware to remove duplication - Remove 12 passthrough resource service wrappers, export generic functions - Simplify route factory from 5 params to (prefix, table) - Consolidate 3 identical resource hooks into parameterized useResourceList() - Replace WebSocket event if/else chain with lookup map * chore: Remove stale NX, MongoDB, and Next.js references from config and docs Delete nx.json, rewrite justfile for Turborepo, update CI workflows to remove Testcontainers/mongo pre-pulls, rewrite CONTRIBUTING.md and copilot-instructions.md for current stack, remove .next/ from pre-commit excludes. * fix: Address CodeRabbit review findings across security, bugs, and config Security: timing-safe login (dummy bcrypt on miss), Content-Disposition header injection sanitization, RBAC on project GET routes, atomic createProject transaction, cross-campaign attack verification. Backend bugs: await Redis disconnect in workers, null-safe job.data access, position penalty filter logic, onConflictDoNothing for hash upserts, deferred event emission for running campaigns, lastEmitTimes Map cleanup. Frontend bugs: stale campaignId in campaign wizard, login loading state, mutation cache invalidation using variables.projectId, malformed query string in dashboard stats, 204 No Content handling, stabilized WS types array. Config: removed duplicate pre-commit hooks, corrected OpenAPI bearerFormat from JWT to token. * docs: Add TanStack Query and Drizzle mock patterns to AGENTS.md * fix: Address PR bot feedback across security, correctness, and config - Fix MinIO healthcheck to use curl instead of mc (not in image) - Validate JWT claim types before trusting payload - Fix agent auth to reject error state, not non-active status - Add campaign existence check on GET /:id/attacks - Fix Oracle hash regex to be case-insensitive - Fix CSS border-color missing hsl() wrapper - Fix dashboard stats using server totals instead of client-side limit=0 - Fix truthy check on limit/offset dropping 0 values - Add resource_update event type to WS invalidation map - Throw after 401 redirect to prevent control flow continuation - Add Suspense fallback loading indicator - Remove token from login response body (HttpOnly cookie only) - Update AGENTS.md: remove stale sessions endpoint, fix doc path - Update auth docs with correct RBAC role names --------- Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
1 parent 81b72e4 commit 864be8f

241 files changed

Lines changed: 14294 additions & 12438 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
name: block-banned-packages
3+
enabled: true
4+
event: bash
5+
pattern: \bbun\s+add\s+.*(express|fastify|next|prisma|mongodb|mongoose|eslint|prettier|redux|mobx|recoil|jotai|nx|lerna|create-react-app)\b
6+
action: block
7+
---
8+
9+
**Banned package detected.** This project has explicit technology constraints:
10+
11+
| Banned | Use Instead |
12+
|--------|-------------|
13+
| express, fastify | **Hono** (on Bun.serve()) |
14+
| next, create-react-app | **Vite + React 19** |
15+
| prisma | **Drizzle ORM** |
16+
| redis *(direct)*, bull *(v3)* | **ioredis + BullMQ** (approved for async jobs) |
17+
| mongodb, mongoose | **PostgreSQL + Drizzle** |
18+
| eslint, prettier | **Biome** |
19+
| redux, mobx, recoil, jotai | **Zustand** (UI state) + **TanStack Query** (server state) |
20+
| nx, lerna | **Turborepo + Bun workspaces** |
21+
22+
See `.kiro/steering/tech.md` for the full constraints.
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
name: block-npm-yarn-pnpm
3+
enabled: true
4+
event: bash
5+
pattern: \b(npm|yarn|pnpm)\s+(install|i|add|remove|run|ci|test|start|exec|init|create|publish|pack)\b
6+
action: block
7+
---
8+
9+
**This project uses Bun exclusively as its runtime and package manager.**
10+
11+
Do not use npm, yarn, or pnpm. Replace with the Bun equivalent:
12+
13+
- `npm install` -> `bun install`
14+
- `npm add <pkg>` -> `bun add <pkg>`
15+
- `npm run <script>` -> `bun run <script>` or `bun <script>`
16+
- `npm test` -> `bun test`
17+
- `npx <cmd>` -> `bunx <cmd>`
18+
19+
See `.kiro/steering/tech.md` and `AGENTS.md`.
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
name: block-wrong-test-runners
3+
enabled: true
4+
event: bash
5+
pattern: \b(jest|vitest)\b
6+
action: block
7+
---
8+
9+
**This project uses `bun:test` for all tests.**
10+
11+
Do not use Jest or Vitest. Replace with:
12+
13+
- `jest` -> `bun test`
14+
- `vitest` -> `bun test`
15+
- `npx jest` -> `bun test`
16+
17+
Test files use `import { describe, expect, it } from 'bun:test'`.
18+
19+
See `AGENTS.md` and `.kiro/steering/tech.md`.
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
name: warn-bun-add
3+
enabled: true
4+
event: bash
5+
pattern: \bbun\s+(add|remove|install)\b
6+
action: warn
7+
---
8+
9+
**Check for running tests before modifying packages.**
10+
11+
Running `bun add` or `bun remove` while `bun test` is active corrupts `node_modules` mid-flight, causing ENOENT errors and broken module resolution.
12+
13+
Before proceeding, verify no test processes are running.
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
id: hashhive-biome-literal-keys
3+
trigger: "when configuring Biome or accessing properties on index-signature types"
4+
confidence: 0.95
5+
domain: biome-config
6+
source: local-repo-analysis
7+
---
8+
9+
# Keep useLiteralKeys Off
10+
11+
## Action
12+
13+
Never enable `useLiteralKeys` in biome.json. It conflicts with TypeScript's `noPropertyAccessFromIndexSignature`.
14+
15+
Always use bracket notation for index-signature access:
16+
```typescript
17+
// With noPropertyAccessFromIndexSignature: true
18+
const val = body['key']; // GOOD
19+
const val = body.key; // BAD — TS error on index signatures
20+
```
21+
22+
## Evidence
23+
24+
- `noPropertyAccessFromIndexSignature: true` in tsconfig.base.json
25+
- `useLiteralKeys: "off"` in biome.json
26+
- Enabling it creates an irreconcilable conflict between Biome wanting `obj.key` and TS requiring `obj['key']`
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
id: hashhive-httpexception-handling
3+
trigger: "when adding or modifying Hono onError handler"
4+
confidence: 0.95
5+
domain: hono
6+
source: local-repo-analysis
7+
---
8+
9+
# Check HTTPException Before Generic Error Handling
10+
11+
## Action
12+
13+
Hono's `app.onError()` catches ALL errors including `HTTPException`. Always check for `HTTPException` first and return its response, otherwise auth 401s become 500s.
14+
15+
```typescript
16+
import { HTTPException } from 'hono/http-exception';
17+
18+
app.onError((err, c) => {
19+
if (err instanceof HTTPException) {
20+
return err.getResponse();
21+
}
22+
// ... generic 500 handling
23+
});
24+
```
25+
26+
## Evidence
27+
28+
- Discovered when all 401 auth responses became 500s
29+
- Auth middleware uses `throw new HTTPException(401, { res: ... })`
30+
- The onError handler was catching it and returning a generic 500
31+
- Fixed in packages/backend/src/index.ts

.claude/instincts/schema-flow.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
id: hashhive-schema-flow
3+
trigger: "when adding or modifying database tables, Zod schemas, or TypeScript types"
4+
confidence: 0.95
5+
domain: architecture
6+
source: local-repo-analysis
7+
---
8+
9+
# Follow the One-Way Schema Flow
10+
11+
## Action
12+
13+
Schema changes flow in ONE direction. Never create types or schemas manually — always derive them.
14+
15+
```
16+
shared/src/db/schema.ts ← EDIT HERE (Drizzle tables)
17+
18+
shared/src/schemas/index.ts ← Add createInsertSchema/createSelectSchema
19+
20+
shared/src/types/index.ts ← Add z.infer<typeof schema> exports
21+
```
22+
23+
When adding a new table:
24+
1. Add Drizzle table definition in `schema.ts`
25+
2. Add `createInsertSchema(table)` + `createSelectSchema(table)` in `schemas/index.ts`
26+
3. Add `type Insert* = z.infer<...>` + `type Select* = z.infer<...>` in `types/index.ts`
27+
4. Run `bun --filter shared build` to regenerate
28+
29+
These three files ALWAYS change together.
30+
31+
## Evidence
32+
33+
- All 15 database tables follow this exact pattern
34+
- Co-change analysis shows schema.ts, schemas/index.ts, and types/index.ts always modified together
35+
- drizzle-zod generates runtime-safe Zod validators from Drizzle definitions

0 commit comments

Comments
 (0)