Worker Service

The Worker Service is a high-performance, scalable job processing system designed to handle EMAIL, SMS, VOICEMAIL, and REVALIDATE campaigns at scale. Built with BullMQ and Redis, it provides reliable job queue management with real-time monitoring capabilities.

Overview

The Worker Service is a production-ready, event-driven job processing system that handles campaign execution at scale. It listens for campaign launch events from Kafka and processes EMAIL, SMS, and VOICEMAIL jobs for connected lead groups. The service provides admin monitoring through a password-protected dashboard with no public endpoints.

Key Features

• Event-Driven Architecture: Kafka-based campaign triggering from Campaign Service
• Lead Group Processing: Automatic job scheduling per connected lead group
• Multi-type Job Processing: EMAIL, SMS, VOICEMAIL job types for real campaigns
• Protected Admin Dashboard: Password-protected monitoring interface
• Scalable Architecture: Horizontal scaling with Redis-backed queues
• Production Ready: Real-world job processing with proper error handling

Production Flow

1. Campaign Publishing: User publishes campaign through Campaign Service
2. Event Processing: Worker Service receives Kafka event for campaign launch
3. Lead Group Processing: System fetches all connected lead groups for the campaign
4. Job Scheduling: Creates individual jobs for each contact in each lead group
5. Queue Processing: Workers process jobs through external service providers
6. Admin Monitoring: Protected dashboard shows real-time progress and metrics

Architecture

Service Flow

The following flowchart illustrates how the Worker Service processes jobs from campaign launch to completion:

Detailed Service Operations

1. Campaign Launch Flow

2. Job Processing Flow

3. Admin Dashboard Monitoring

Components

1. Kafka Listener

Event-driven campaign processing listener that consumes campaign launch events.

Features:

• Automatic campaign launch detection
• Lead group identification and processing
• Error handling and retry logic for event processing

2. Job Scheduler

Processes campaign data and schedules jobs for each contact in connected lead groups.

Features:

• Lead group iteration and contact processing
• Job type determination based on campaign steps
• Unique job ID generation for tracking
• Queue routing based on job type

3. Queue Workers

BullMQ workers that process jobs from Redis queues.

Features:

• Multi-queue job processing (EMAIL, SMS, VOICEMAIL)
• Integration with external service providers
• Retry logic with exponential backoff
• Real-time status reporting

4. Admin Dashboard

Password-protected monitoring interface for system administrators.

Features:

• Queue status monitoring
• Worker performance metrics
• Campaign progress tracking
• Error analysis and debugging tools

Production Endpoints

Admin Dashboard

• URL: /dashboard
• Access: Password protected
• Purpose: System monitoring and administration

Health Check

• URL: /health
• Access: Internal only
• Purpose: Service health monitoring

No public endpoints are exposed for security.

Testing Features (Development Only)

For development and testing purposes, the service includes additional features that simulate campaign execution:

Performance Testing

• Bulk Job Generation: Create thousands of test jobs for load testing
• Realistic Timing: Simulated delays (20s-1min scheduling, job-specific processing)
• Failure Simulation: Configurable failure rates (20%) for testing error handling
• Memory Buffer: 1M+ event capacity for testing dashboard updates

Test Job Types

• Revalidate Jobs: Simple connectivity testing jobs

• REVALIDATE Jobs: Simple connectivity testing jobs
• Synthetic Data: Generated test contacts and campaigns
• Memory-based Logging: In-memory event storage for testing real-time updates

Testing Flow

Note: Testing features are for development purposes only and should not be used in production environments.