Skip to content

Latest commit

 

History

History
865 lines (689 loc) · 24.6 KB

File metadata and controls

865 lines (689 loc) · 24.6 KB

Code Kit Ultra — Automation Implementation Guides

This document provides step-by-step implementation guides for each automation feature. Each feature maintains security, governance, and audit trail requirements.


Overview: Automation Features

Feature Complexity Security Risk Benefits Timeline
Auto-Approval Chains Medium Low 20-30% fewer manual approvals 2-3 days
Alert Auto-Acknowledge Low Low 40-50% less alert noise 1-2 days
Test Verification Medium Low Catches gate issues early 2-3 days
Automatic Healing Medium Medium Reduces manual retry work 1-2 days
Automatic Rollback High Medium 5-10x faster recovery 2-3 days

Key Principle: All automation must be auditable, reversible, and have opt-out mechanisms.


Feature 1: Auto-Approval Chains

Goal: Automatically approve dependent gates when prerequisites are met.

Use Case:

- Security gate passes (0.95+ score) → Architecture gate can auto-approve
- QA gate passes (85%+ coverage) → Build gate can auto-approve
- Cost gate passes → Deployment gate can auto-approve

Implementation Steps

Step 1: Define Gate Dependencies

Create file: packages/governance/src/gate-dependencies.ts

/**
 * Define which gates can auto-approve based on other gate conditions.
 * Only gates listed here can be auto-approved by the system.
 */
export const GateDependencies = {
  // If security_gate passes with high score, architecture_gate can auto-approve
  security_gate: {
    dependents: ['architecture_gate'],
    condition: (bundle) => bundle.security.score > 0.9,
    reason: 'Security gate passed with high confidence'
  },
  
  // If qa_gate passes with good coverage, build_gate can auto-approve
  qa_gate: {
    dependents: ['build_gate'],
    condition: (bundle) => bundle.testCoverage > 0.85,
    reason: 'QA gate passed with adequate coverage'
  },
  
  // If cost_gate passes, deployment_gate can auto-approve
  cost_gate: {
    dependents: ['deployment_gate'],
    condition: (bundle) => bundle.costScore > 0.8,
    reason: 'Cost gate passed'
  },
};

export interface AutoApprovalConfig {
  enabled: boolean;  // Set via CKU_AUTO_APPROVAL_ENABLED
  maxChainDepth: number;  // Prevent infinite loops
  allowedGates: Set<string>;  // Only these gates can auto-approve
}

export function getAutoApprovalConfig(): AutoApprovalConfig {
  return {
    enabled: process.env.CKU_AUTO_APPROVAL_ENABLED !== 'false',
    maxChainDepth: parseInt(process.env.CKU_AUTO_APPROVAL_CHAIN_DEPTH || '3'),
    allowedGates: new Set(
      (process.env.CKU_AUTO_APPROVABLE_GATES || 
        'architecture_gate,build_gate,deployment_gate').split(',')
    ),
  };
}

Step 2: Implement Auto-Approval Logic

Create file: packages/governance/src/auto-approval.ts

import { loadRunBundle } from "../../memory/src/run-store";
import { getApprovedGates } from "./gate-store";
import { ApprovalService } from "../../control-service/src/services/approval-service";
import { AuditEventBuilder, AuditActions } from "../../control-service/src/lib/audit-builder";
import type { AuthContext } from "../../control-service/src/lib/handler-utils";
import { GateDependencies, getAutoApprovalConfig } from "./gate-dependencies";

const systemContext: AuthContext = {
  actor: {
    id: "system",
    name: "system:auto-approval",
    type: "system",
    authMode: "internal",
  },
  tenant: {
    orgId: "system",
    workspaceId: "system",
    projectId: "system",
  },
};

/**
 * Evaluate and apply auto-approvals based on gate dependencies.
 * Called after any gate is manually approved.
 */
export async function evaluateAutoApprovals(runId: string): Promise<string[]> {
  const config = getAutoApprovalConfig();
  if (!config.enabled) return [];

  const bundle = loadRunBundle(runId);
  if (!bundle) throw new Error(`Run not found: ${runId}`);

  const approvedGates = await getApprovedGates(runId);
  const autoApprovedGates: string[] = [];
  let chainDepth = 0;
  let changed = true;

  // Keep evaluating until no new gates can be auto-approved
  // (prevents infinite loops with maxChainDepth limit)
  while (changed && chainDepth < config.maxChainDepth) {
    changed = false;
    chainDepth++;

    // Check each approved gate to see if it enables dependent gates
    for (const approvedGate of approvedGates) {
      const dependency = GateDependencies[approvedGate as keyof typeof GateDependencies];
      if (!dependency) continue; // This gate has no dependents

      // Check each dependent gate
      for (const dependentGate of dependency.dependents) {
        // Skip if not an auto-approvable gate
        if (!config.allowedGates.has(dependentGate)) {
          continue;
        }

        // Skip if already approved
        if (approvedGates.includes(dependentGate)) {
          continue;
        }

        // Check if condition is met
        try {
          if (dependency.condition(bundle)) {
            // Auto-approve the dependent gate
            await ApprovalService.approve(runId, systemContext.actor.name);

            // Log in audit trail
            await new AuditEventBuilder(AuditActions.GATE_AUTO_APPROVED, systemContext)
              .withRunId(runId)
              .withGateId(dependentGate)
              .withResult("success")
              .withDetails({
                reason: dependency.reason,
                triggeredBy: approvedGate,
                chainDepth,
              })
              .emit();

            approvedGates.push(dependentGate);
            autoApprovedGates.push(dependentGate);
            changed = true;
          }
        } catch (err) {
          // Log but don't fail the overall operation
          console.error(`Auto-approval evaluation failed for ${dependentGate}:`, err);
        }
      }
    }
  }

  return autoApprovedGates;
}

/**
 * Check if a gate CAN be auto-approved (regardless of whether it should be right now).
 * Useful for UI to show which gates might auto-approve.
 */
export function canBeAutoApproved(gateId: string): boolean {
  const config = getAutoApprovalConfig();
  return config.enabled && config.allowedGates.has(gateId);
}

Step 3: Integrate with Approval Flow

Update file: apps/control-service/src/handlers/approve-gate.ts

// After approving the gate, evaluate auto-approvals
await ApprovalService.approve(runId, context.actor.name);

// Check if this approval triggers any auto-approvals
const autoApprovedGates = await evaluateAutoApprovals(runId);

// Include in response
res.json({
  status: "approved",
  approvedBy: context.actor.name,
  autoApprovalsTriggered: autoApprovedGates,
});

Step 4: Environment Variables

Add to .env or deployment config:

# Enable/disable auto-approval system
CKU_AUTO_APPROVAL_ENABLED=true

# Maximum chain depth (prevent infinite loops)
CKU_AUTO_APPROVAL_CHAIN_DEPTH=3

# Which gates can be auto-approved
CKU_AUTO_APPROVABLE_GATES=architecture_gate,build_gate,deployment_gate

Step 5: Testing

Create file: packages/governance/src/auto-approval.test.ts

import { describe, it, expect, beforeEach, vi } from "vitest";
import { evaluateAutoApprovals, canBeAutoApproved } from "./auto-approval";

describe("Auto-Approval System", () => {
  it("should auto-approve dependent gate when condition is met", async () => {
    // Mock run with security gate passed and high score
    const mockRun = {
      state: { runId: "run-001" },
      security: { score: 0.95 },
    };
    
    vi.mock("../../memory/src/run-store", () => ({
      loadRunBundle: () => mockRun,
    }));

    const autoApproved = await evaluateAutoApprovals("run-001");
    expect(autoApproved).toContain("architecture_gate");
  });

  it("should not auto-approve if condition not met", async () => {
    // Mock run with security gate passed but low score
    const mockRun = {
      state: { runId: "run-001" },
      security: { score: 0.80 },
    };
    
    vi.mock("../../memory/src/run-store", () => ({
      loadRunBundle: () => mockRun,
    }));

    const autoApproved = await evaluateAutoApprovals("run-001");
    expect(autoApproved).not.toContain("architecture_gate");
  });

  it("should respect max chain depth to prevent infinite loops", async () => {
    // Create circular dependency (shouldn't happen, but test anyway)
    // With maxChainDepth=3, should stop after 3 levels
    
    const autoApproved = await evaluateAutoApprovals("run-001");
    expect(autoApproved.length).toBeLessThanOrEqual(3);
  });

  it("should create audit events for all auto-approvals", async () => {
    const auditSpy = vi.spyOn("audit", "writeAuditEvent");
    
    await evaluateAutoApprovals("run-001");
    
    // Verify each auto-approval was audited
    expect(auditSpy).toHaveBeenCalledWith(
      expect.objectContaining({
        action: "GATE_AUTO_APPROVED",
        actorName: "system:auto-approval",
      })
    );
  });
});

Security Considerations

  • Audit trail — Each auto-approval logged with reason
  • Actor tracking — Shows as system:auto-approval
  • Opt-out — Can disable with CKU_AUTO_APPROVAL_ENABLED=false
  • Allowlist — Only specific gates can auto-approve
  • Condition-based — Requires specific conditions to be met
  • Depth limit — Prevents infinite loop attacks
  • Manual override — User can still manually approve

Expected Impact

  • Approval time reduced by 20-30% for low-risk runs
  • Gate approval chains work seamlessly
  • Audit trail is complete and auditable

Feature 2: Alert Auto-Acknowledge

Goal: Automatically acknowledge alerts when their condition resolves.

Use Case:

- Alert: "Test coverage dropped to 80%"
- Developer commits fix (coverage now 92%)
- Alert automatically acknowledged + closure logged

Implementation Steps

Step 1: Alert Condition Monitor

Create file: apps/control-service/src/services/alert-condition-monitor.ts

import { getOpenAlerts, acknowledgeAlert } from "./alert-service";
import { AuditEventBuilder, AuditActions } from "../lib/audit-builder";
import type { AuthContext } from "../lib/handler-utils";

const systemContext: AuthContext = {
  actor: {
    id: "system",
    name: "system:auto-acknowledge",
    type: "system",
    authMode: "internal",
  },
  tenant: {
    orgId: "system",
    workspaceId: "system",
    projectId: "system",
  },
};

/**
 * Check if an alert's condition has been resolved.
 * Returns true if the underlying issue no longer exists.
 */
async function isAlertResolved(alertId: string): Promise<boolean> {
  const alert = await getAlertDetails(alertId);
  
  switch (alert.type) {
    case "test_coverage_low":
      // Check current test coverage
      const coverage = await getCurrentTestCoverage();
      return coverage >= 85; // Condition resolved when >= 85%
      
    case "auth_failures_high":
      // Check recent auth failures
      const failures = await getRecentAuthFailures();
      return failures < 5; // Condition resolved when < 5 failures/min
      
    case "db_pool_exhausted":
      // Check if pool has capacity
      const poolCapacity = await getDbPoolCapacity();
      return poolCapacity > 0.3; // Resolved when >30% available
      
    case "redis_unavailable":
      // Check if Redis is available
      const redisHealth = await checkRedisHealth();
      return redisHealth === "healthy";
      
    default:
      return false; // Unknown alert type, don't auto-acknowledge
  }
}

/**
 * Automatically acknowledge resolved alerts.
 * Called periodically (every 5 minutes).
 */
export async function autoAcknowledgeResolvedAlerts(): Promise<number> {
  if (process.env.CKU_AUTO_ACKNOWLEDGE_ALERTS !== 'true') {
    return 0; // Feature disabled
  }

  const openAlerts = await getOpenAlerts();
  let acknowledged = 0;

  for (const alert of openAlerts) {
    try {
      const resolved = await isAlertResolved(alert.id);
      
      if (resolved) {
        // Acknowledge the alert
        await acknowledgeAlert(alert.id, {
          acknowledgedBy: systemContext.actor.name,
          reason: `Condition resolved automatically: ${alert.condition}`,
          timestamp: new Date(),
        });

        // Log in audit trail
        await new AuditEventBuilder(AuditActions.ALERT_ACKNOWLEDGED, systemContext)
          .withDetails({
            alertId: alert.id,
            alertType: alert.type,
            resolvedCondition: alert.condition,
          })
          .emit();

        acknowledged++;
      }
    } catch (err) {
      console.error(`Failed to check alert ${alert.id}:`, err);
      // Continue checking other alerts
    }
  }

  return acknowledged;
}

/**
 * Schedule automatic alert acknowledgment.
 */
export function scheduleAlertMonitoring(): void {
  if (process.env.CKU_AUTO_ACKNOWLEDGE_ALERTS !== 'true') {
    return; // Feature disabled
  }

  // Check every 5 minutes
  const intervalId = setInterval(async () => {
    try {
      const count = await autoAcknowledgeResolvedAlerts();
      if (count > 0) {
        console.log(`Auto-acknowledged ${count} resolved alerts`);
      }
    } catch (err) {
      console.error("Error in alert auto-acknowledge:", err);
    }
  }, 5 * 60 * 1000);

  // Allow graceful shutdown
  process.on("SIGTERM", () => {
    clearInterval(intervalId);
  });
}

Step 2: Integration in Server Startup

Update file: apps/control-service/src/index.ts

import { scheduleAlertMonitoring } from "./services/alert-condition-monitor.js";

async function startServer() {
  try {
    logger.info('Starting Code Kit Ultra Control Service');

    // ... existing startup code ...

    // Schedule alert monitoring if enabled
    if (process.env.CKU_AUTO_ACKNOWLEDGE_ALERTS === 'true') {
      logger.info('Alert auto-acknowledge monitoring enabled');
      scheduleAlertMonitoring();
    }

    const server = app.listen(PORT, () => {
      logger.info({ port: PORT }, 'Control Service started');
    });

    // ... rest of startup ...
  }
}

Step 3: Environment Variables

# Enable/disable auto-acknowledge
CKU_AUTO_ACKNOWLEDGE_ALERTS=true

Security Considerations

  • Only acknowledges resolved conditions — Not based on manual dismissal
  • Audit trail — Each acknowledgment logged
  • Non-destructive — Original alert immutable
  • Opt-out — Can disable feature
  • Cannot suppress critical alerts — P0 alerts still visible

Expected Impact

  • Alert noise reduced by 40-50%
  • Signal-to-noise ratio improved
  • Cleaner dashboard

Feature 3: Automatic Test Verification

Goal: Run tests on gate approval to verify the decision was correct.

Use Case:

- User approves QA gate
- System auto-runs: npx vitest run qa/
- If tests fail, gate remains approved but run flagged for review

Implementation Steps

Step 1: Test Command Configuration

Create file: packages/governance/src/gate-test-commands.ts

/**
 * Define which tests to run for each gate approval.
 */
export const GateTestCommands = {
  security_gate: {
    command: 'npx vitest run packages/auth/src --run',
    timeout: 30000,
    minPass: 40, // At least 40 tests must pass
  },
  qa_gate: {
    command: 'npx vitest run apps/control-service/test/ --run',
    timeout: 60000,
    minPass: 44, // Smoke + regression tests
  },
  build_gate: {
    command: 'npm run build --workspaces 2>&1 | grep -i error | wc -l',
    timeout: 120000,
    maxErrors: 0, // Zero build errors allowed
  },
  architecture_gate: {
    command: 'npx vitest run packages/governance/src --run',
    timeout: 30000,
    minPass: 30,
  },
};

export function canVerifyGate(gateId: string): boolean {
  return gateId in GateTestCommands;
}

export function getTestCommand(gateId: string): typeof GateTestCommands[keyof typeof GateTestCommands] | null {
  return GateTestCommands[gateId as keyof typeof GateTestCommands] || null;
}

Step 2: Test Verification Service

Create file: apps/control-service/src/services/gate-verification.ts

import { exec } from 'node:child_process';
import { promisify } from 'node:util';
import { GateTestCommands, canVerifyGate } from '../../../packages/governance/src/gate-test-commands';
import { AuditEventBuilder, AuditActions } from '../lib/audit-builder';
import type { AuthContext } from '../lib/handler-utils';
import { loadRunBundle, updateRunState } from '../../../packages/memory/src/run-store';
import { logger } from '../lib/logger';

const execAsync = promisify(exec);

const systemContext: AuthContext = {
  actor: {
    id: "system",
    name: "system:gate-verification",
    type: "system",
    authMode: "internal",
  },
  tenant: {
    orgId: "system",
    workspaceId: "system",
    projectId: "system",
  },
};

interface VerificationResult {
  passed: boolean;
  testCount?: number;
  duration: number;
  output: string;
}

/**
 * Run verification tests for a gate.
 * Returns result but doesn't block approval (non-blocking verification).
 */
export async function verifyGateApproval(
  gateId: string,
  runId: string
): Promise<VerificationResult> {
  if (!canVerifyGate(gateId)) {
    return { passed: true, duration: 0, output: 'No verification for this gate' };
  }

  const testConfig = GateTestCommands[gateId as keyof typeof GateTestCommands];
  const startTime = Date.now();

  try {
    logger.info(`Starting gate verification for ${gateId}`);

    // Execute test command with timeout
    const { stdout, stderr } = await execAsync(testConfig.command, {
      timeout: testConfig.timeout,
      maxBuffer: 10 * 1024 * 1024, // 10MB buffer
    });

    const output = stdout + stderr;
    const duration = Date.now() - startTime;
    const passed = determineSuccess(gateId, output);

    // Log verification result
    await new AuditEventBuilder(AuditActions.GATE_VERIFICATION_COMPLETED, systemContext)
      .withRunId(runId)
      .withGateId(gateId)
      .withResult(passed ? 'success' : 'failure')
      .withDetails({
        testsPassed: passed,
        duration,
        outputLength: output.length,
      })
      .emit();

    // If verification failed, flag run for manual review
    if (!passed) {
      const bundle = loadRunBundle(runId);
      if (bundle) {
        bundle.state.requiresManualReview = true;
        bundle.state.reviewReason = `Gate verification failed: ${gateId}`;
        updateRunState(runId, bundle.state);
      }
    }

    return { passed, duration, output };
  } catch (err: any) {
    const duration = Date.now() - startTime;
    
    // Timeout or execution error
    logger.error(`Gate verification failed for ${gateId}:`, err.message);

    await new AuditEventBuilder(AuditActions.GATE_VERIFICATION_FAILED, systemContext)
      .withRunId(runId)
      .withGateId(gateId)
      .withResult('failure')
      .withDetails({
        error: err.message,
        duration,
      })
      .emit();

    return { passed: false, duration, output: err.message };
  }
}

/**
 * Determine if test output indicates success based on gate config.
 */
function determineSuccess(gateId: string, output: string): boolean {
  const config = GateTestCommands[gateId as keyof typeof GateTestCommands];
  
  // Check for passed test count
  if ('minPass' in config) {
    const passMatch = output.match(/(\d+) passed/);
    if (!passMatch) return false;
    return parseInt(passMatch[1]) >= config.minPass;
  }

  // Check for max errors
  if ('maxErrors' in config) {
    const errorCount = parseInt(output) || 0;
    return errorCount <= config.maxErrors;
  }

  // Default: success if no errors/failures in output
  return !output.includes('FAIL') && !output.includes('error');
}

/**
 * Non-blocking verification trigger.
 * Called after gate approval but doesn't wait for completion.
 */
export function triggerGateVerificationAsync(
  gateId: string,
  runId: string
): void {
  // Fire and forget
  verifyGateApproval(gateId, runId)
    .catch(err => logger.error('Async verification error:', err));
}

Step 3: Integration with Approval Handler

Update file: apps/control-service/src/handlers/approve-gate.ts

import { triggerGateVerificationAsync } from '../services/gate-verification.js';

export async function approveGateHandler(req: Request, res: Response) {
  try {
    const context = extractAuthContext(req);
    const runId = extractRunId(req);
    const gateId = extractGateId(req);

    // ... approval logic ...

    // Trigger verification (non-blocking)
    triggerGateVerificationAsync(gateId, runId);

    res.json({
      status: 'approved',
      approvedBy: context.actor.name,
      verificationInProgress: true,
    });
  } catch (err: any) {
    return sendInternalError(res, err, 'approve_gate');
  }
}

Security Considerations

  • Non-blocking — Doesn't revoke approval if tests fail
  • Flagged for review — Failed verification visible to user
  • Audit trail — All verification results logged
  • Sandboxed tests — Run in isolated environment
  • Timeout protection — Tests can't run forever

Expected Impact

  • Catches gate issues early before execution
  • Prevents common approval mistakes
  • Builds confidence in gate decisions

Feature 4: Automatic Healing (Enhancement)

See existing implementation in:

  • packages/orchestrator/src/healing-strategies.ts
  • packages/orchestrator/src/retry-handler.ts

Enhancement: Trigger healing automatically on specific failure patterns instead of waiting for manual retry.

Implementation: Auto-Trigger Healing

// In step failure handler
export async function handleStepFailure(runId: string, stepId: string, error: any) {
  const bundle = loadRunBundle(runId);
  const step = bundle.plan.tasks.find(t => t.id === stepId);
  
  if (!step) return;

  // Check if auto-healing is enabled for this failure type
  const shouldAutoHeal = getHealingStrategy(step.type, error.code);
  
  if (shouldAutoHeal && bundle.state.healingAttemptsRemaining > 0) {
    try {
      await ApprovalService.retry(runId, stepId, 'system:auto-heal');
      
      await new AuditEventBuilder('STEP_AUTO_HEALED', systemContext)
        .withRunId(runId)
        .withStepId(stepId)
        .withDetails({ strategy: shouldAutoHeal.name })
        .emit();
    } catch (healErr) {
      bundle.state.requiresManualReview = true;
      updateRunState(runId, bundle.state);
    }
  }
}

Feature 5: Automatic Rollback (Production Only)

Goal: Automatically rollback deployments that cause P0 alerts.

Risk Level: HIGH — Only for production environments

Implementation: Deployment Health Monitor

export async function monitorDeploymentHealth() {
  const recentDeployment = getLatestDeployment();
  if (!recentDeployment) return;

  const timeSinceDeploy = Date.now() - recentDeployment.timestamp;
  
  // Only monitor first 5 minutes after deploy
  if (timeSinceDeploy > 5 * 60 * 1000) return;

  const recentAlerts = getAlertsInWindow(
    recentDeployment.timestamp,
    5 * 60 * 1000
  );
  
  // Only trigger on P0 alerts
  const p0Alerts = recentAlerts.filter(a => a.severity === 'P0');
  const alertRate = p0Alerts.length / (5 / 60);
  
  if (alertRate > 20) {
    // Trigger auto-rollback
    await initiateAutoRollback(recentDeployment.previousVersion, {
      reason: `Auto-rollback: ${alertRate.toFixed(1)} P0 alerts/min`,
      alerts: p0Alerts.map(a => a.id),
    });
    
    // Notify on-call
    await notifyOncall({
      severity: 'P0',
      action: 'AUTO_ROLLBACK_INITIATED',
    });
  }
}

Environment Variable:

CKU_AUTO_ROLLBACK_ENABLED=true  # Only in production
CKU_AUTO_ROLLBACK_THRESHOLD_MINUTES=5  # Monitor window
CKU_AUTO_ROLLBACK_ALERT_THRESHOLD=20   # Alerts per minute

Summary: Implementation Roadmap

Phase Feature Effort Start Status
1 Refactoring (utilities) 6h Done
2 Auto-Approval Chains 6h Week 2 📋
2 Alert Auto-Acknowledge 4h Week 2 📋
3 Test Verification 4h Week 3 📋
3 Healing Enhancement 3h Week 3 📋
3 Rollback Monitor 6h Week 4 📋

Testing Automation Features

For each feature, include tests:

describe("Auto-Approval System", () => {
  it("should auto-approve dependent gate", async () => { ... });
  it("should NOT auto-approve if condition not met", async () => { ... });
  it("should create audit event for auto-approval", async () => { ... });
  it("should respect max chain depth", async () => { ... });
});

Questions?

Refer to:

  • docs/REFACTORING_AND_AUTOMATION_PLAN.md — Overall strategy
  • docs/REFACTORING_COMPLETION_REPORT.md — Phase 1 results
  • This file — Implementation details