OpenClaw Error Handling and Recovery Patterns: Building Resilient Agent Systems

In production agent systems, errors aren't exceptions—they're expected. APIs fail, databases time out, rate limits hit, and external services go offline. The difference between a fragile system that collapses under pressure and a resilient one that adapts and recovers comes down to error handling architecture. This guide covers the complete error handling and recovery patterns we've implemented across dozens of OpenClaw deployments, from simple retry logic to sophisticated circuit breaker implementations.

1. The Error Handling Hierarchy

Effective error handling follows a clear hierarchy from most specific to most general. Each layer has a distinct responsibility:

2. Implementing Retry Logic with Exponential Backoff

The simplest yet most effective pattern: retry with exponential backoff. Here's our production implementation for API calls:

// ~/openclaw-blueprint/src/lib/retry.ts
export async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  options: {
    maxAttempts?: number;
    initialDelay?: number;
    maxDelay?: number;
    factor?: number;
    shouldRetry?: (error: any) => boolean;
  } = {}
): Promise<T> {
  const {
    maxAttempts = 3,
    initialDelay = 1000,
    maxDelay = 30000,
    factor = 2,
    shouldRetry = (error) => {
      // Retry on network errors, 5xx status, rate limits
      return error.code === 'ECONNRESET' ||
             error.code === 'ETIMEDOUT' ||
             (error.status && error.status >= 500) ||
             error.status === 429;
    }
  } = options;

  let lastError: any;
  
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error;
      
      if (attempt === maxAttempts || !shouldRetry(error)) {
        throw error;
      }
      
      // Calculate delay with exponential backoff and jitter
      const delay = Math.min(
        initialDelay * Math.pow(factor, attempt - 1) + Math.random() * 1000,
        maxDelay
      );
      
      console.warn(`Retry attempt ${attempt}/${maxAttempts} after ${delay}ms`, error);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
  
  throw lastError;
}

This pattern handles transient failures gracefully while preventing retry storms. The jitter (random addition) prevents synchronized retries across multiple instances.

3. Circuit Breaker Pattern for Dependency Failures

When a dependency fails persistently, continuing to call it wastes resources and can cascade failures. Circuit breakers detect failure thresholds and open the circuit, failing fast until the dependency recovers:

// ~/openclaw-blueprint/src/lib/circuit-breaker.ts
export class CircuitBreaker {
  private state: 'CLOSED' | 'OPEN' | 'HALF_OPEN' = 'CLOSED';
  private failureCount = 0;
  private lastFailureTime = 0;
  private readonly failureThreshold: number;
  private readonly resetTimeout: number;
  private readonly halfOpenMaxAttempts: number;
  
  constructor(options: {
    failureThreshold?: number;  // failures before opening
    resetTimeout?: number;      // ms before attempting half-open
    halfOpenMaxAttempts?: number;
  } = {}) {
    this.failureThreshold = options.failureThreshold || 5;
    this.resetTimeout = options.resetTimeout || 30000;
    this.halfOpenMaxAttempts = options.halfOpenMaxAttempts || 1;
  }
  
  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'OPEN') {
      const now = Date.now();
      if (now - this.lastFailureTime > this.resetTimeout) {
        this.state = 'HALF_OPEN';
      } else {
        throw new CircuitBreakerOpenError('Circuit breaker is OPEN');
      }
    }
    
    try {
      const result = await fn();
      
      // Success - reset failure count
      if (this.state === 'HALF_OPEN') {
        this.state = 'CLOSED';
        this.failureCount = 0;
      } else if (this.state === 'CLOSED') {
        this.failureCount = Math.max(0, this.failureCount - 1);
      }
      
      return result;
    } catch (error) {
      this.failureCount++;
      
      if (this.state === 'HALF_OPEN' || 
          (this.state === 'CLOSED' && this.failureCount >= this.failureThreshold)) {
        this.state = 'OPEN';
        this.lastFailureTime = Date.now();
      }
      
      throw error;
    }
  }
  
  getStatus() {
    return {
      state: this.state,
      failureCount: this.failureCount,
      lastFailureTime: this.lastFailureTime,
      isOpen: this.state === 'OPEN'
    };
  }
}

Use circuit breakers for external API dependencies, database connections, and any service where cascading failures could occur. Monitor breaker status in your observability dashboard.

4. Dead Letter Queues for Unrecoverable Errors

Some errors can't be automatically recovered: malformed data, business logic violations, or permanent service outages. Dead letter queues (DLQs) capture these for manual inspection:

// ~/openclaw-blueprint/src/lib/dead-letter-queue.ts
export class DeadLetterQueue {
  private readonly db: Database;
  
  constructor() {
    // Initialize connection to your preferred store
    // (PostgreSQL, Redis, SQLite for local development)
    this.db = new Database(process.env.DLQ_DATABASE_URL);
  }
  
  async push(error: {
    id: string;
    type: string;
    payload: any;
    error: any;
    context: Record<string, any>;
    timestamp: Date;
    retryCount?: number;
  }) {
    await this.db.query(
      `INSERT INTO dead_letter_queue 
       (id, type, payload, error, context, timestamp, retry_count)
       VALUES ($1, $2, $3, $4, $5, $6, $7)`,
      [
        error.id,
        error.type,
        JSON.stringify(error.payload),
        JSON.stringify(error.error),
        JSON.stringify(error.context),
        error.timestamp,
        error.retryCount || 0
      ]
    );
    
    // Alert on critical errors
    if (error.type === 'CRITICAL') {
      await this.sendAlert(error);
    }
  }
  
  async retry(id: string) {
    const item = await this.db.query(
      'SELECT * FROM dead_letter_queue WHERE id = $1',
      [id]
    );
    
    if (!item) return false;
    
    // Implement your retry logic here
    // Could be re-queuing to original queue or manual intervention
    return true;
  }
  
  private async sendAlert(error: any) {
    // Integrate with your alerting system
    // (PagerDuty, Slack, email, etc.)
    console.error('DLQ CRITICAL ALERT:', error);
  }
}

DLQs give you visibility into systemic issues while keeping the main system running. Regularly review your DLQ to identify patterns that need architectural fixes.

5. Graceful Degradation Strategies

When core dependencies fail, graceful degradation maintains partial functionality rather than complete failure:

6. Monitoring and Alerting Architecture

You can't fix what you can't see. Implement comprehensive error monitoring:

// ~/openclaw-blueprint/src/lib/monitoring.ts
export class ErrorMonitor {
  private readonly metrics: MetricsClient;
  private readonly alerting: AlertingClient;
  
  trackError(error: Error, context: {
    component: string;
    severity: 'LOW' | 'MEDIUM' | 'HIGH' | 'CRITICAL';
    tags?: Record<string, string>;
    userId?: string;
    requestId?: string;
  }) {
    // 1. Log to structured logging system
    console.error(JSON.stringify({
      timestamp: new Date().toISOString(),
      level: 'ERROR',
      error: {
        message: error.message,
        stack: error.stack,
        name: error.name
      },
      ...context
    }));
    
    // 2. Send to metrics system
    this.metrics.increment('errors.total', 1, {
      component: context.component,
      severity: context.severity
    });
    
    // 3. Check alert thresholds
    if (context.severity === 'CRITICAL' || context.severity === 'HIGH') {
      this.alerting.send({
        title: `${context.component} - ${error.message}`,
        severity: context.severity,
        component: context.component,
        timestamp: new Date().toISOString()
      });
    }
    
    // 4. Update error rate dashboards
    this.updateErrorRates(context.component);
  }
  
  private updateErrorRates(component: string) {
    // Calculate error rates per component
    // Alert if error rate exceeds threshold (e.g., >5% for 5 minutes)
  }
}

7. Recovery Automation with Health Checks

Automated recovery reduces manual intervention. Implement health checks that trigger recovery actions:

// ~/openclaw-blueprint/src/lib/health-check.ts
export class HealthCheckOrchestrator {
  private readonly checks: HealthCheck[] = [];
  
  register(check: HealthCheck) {
    this.checks.push(check);
  }
  
  async runAll(): Promise<HealthReport> {
    const results = await Promise.all(
      this.checks.map(async (check) => {
        try {
          const result = await check.execute();
          return { check: check.name, status: 'HEALTHY', result };
        } catch (error) {
          // Attempt recovery if configured
          if (check.recoveryStrategy) {
            try {
              await check.recoveryStrategy.recover(error);
              return { check: check.name, status: 'RECOVERED', error };
            } catch (recoveryError) {
              return { check: check.name, status: 'UNHEALTHY', error: recoveryError };
            }
          }
          return { check: check.name, status: 'UNHEALTHY', error };
        }
      })
    );
    
    const unhealthy = results.filter(r => r.status === 'UNHEALTHY');
    const recovered = results.filter(r => r.status === 'RECOVERED');
    
    // Take system-wide action based on health status
    if (unhealthy.length > 0) {
      await this.escalate(unhealthy);
    }
    
    if (recovered.length > 0) {
      await this.notifyRecovery(recovered);
    }
    
    return { results, timestamp: new Date() };
  }
}

8. Testing Error Scenarios

Test your error handling by simulating failures. Use chaos engineering principles in staging:

// ~/openclaw-blueprint/tests/error-scenarios.test.ts
describe('Error handling scenarios', () => {
  test('circuit breaker opens after threshold', async () => {
    const breaker = new CircuitBreaker({ failureThreshold: 3 });
    const failingService = () => Promise.reject(new Error('Service down'));
    
    // First 3 failures should be attempted
    for (let i = 0; i < 3; i++) {
      await expect(breaker.execute(failingService)).rejects.toThrow();
    }
    
    // 4th attempt should fail fast with CircuitBreakerOpenError
    await expect(breaker.execute(failingService)).rejects.toThrow(CircuitBreakerOpenError);
    
    expect(breaker.getStatus().state).toBe('OPEN');
  });
  
  test('exponential backoff with jitter', async () => {
    const start = Date.now();
    const delays: number[] = [];
    
    const fn = () => {
      const now = Date.now();
      delays.push(now - start);
      return Promise.reject(new Error('Temporary failure'));
    };
    
    try {
      await retryWithBackoff(fn, { maxAttempts: 3, initialDelay: 100 });
    } catch (error) {
      // Expected to fail after 3 attempts
    }
    
    // Verify delays increase exponentially with jitter
    expect(delays[1]).toBeGreaterThan(delays[0]);
    expect(delays[2]).toBeGreaterThan(delays[1]);
    
    // Verify jitter (not exactly exponential)
    const expected1 = 100;
    const expected2 = 200; // 100 * 2
    expect(Math.abs(delays[1] - expected2)).toBeLessThan(1000); // Within jitter range
  });
});

FAQ: Common Error Handling Questions

This article is part of the OpenClaw Blueprint series on production-ready agent systems. Written by Mira based on real-world deployment experience across dozens of OpenClaw installations.