Worker Service

The Worker Service (apps/worker, port 3007) is a dedicated Express + BullMQ process that handles all asynchronous, time-sensitive campaign execution. It shares @ce-sdr/lib and @ce-sdr/db with the main app but runs as a separate process to isolate job workloads.

Responsibilities

• Process email, SMS, and voicemail jobs enqueued by the campaign service
• Run cron jobs (email warmup resets, follow-up scheduling)
• Expose a health check endpoint at /health
• Provide Bull Board queue monitoring at /bull (password protected)

Architecture

Queue Processing Pipeline

Campaign execution flows through three BullMQ queue stages:

Stage 1 — send_campaign

Trigger: Enqueued by app when a campaign is launched (5-minute delay in prod).

Processing:

1. Fetch campaign with all lead groups and contacts from DB
2. Verify campaign is still ACTIVE
3. Deduplicate contacts across lead groups
4. Filter banned leads and contacts who have already replied
5. For each step group: calculate scheduledAt = now + group_day + cumulative_wait_days
6. Enqueue one send_campaign_step job per step, delayed to its scheduledAt

Stage 2 — send_campaign_step

Trigger: Delayed job fired at the scheduled time for each campaign step.

Processing:

1. Fetch step details and verify campaign is still ACTIVE
2. Get all enrolled contacts not yet replied
3. Distribute contacts across business-hour time slots (avoids burst sends)
4. For each contact: create/update StepExecution record
5. Enqueue individual send jobs (email-queue, sms-queue, or voicemail-queue) with calculated delay

Stage 3 — email-queue / sms-queue / voicemail-queue

Trigger: Individual per-contact send jobs.

Email processing:

1. Verify campaign is still ACTIVE; skip if paused/cancelled
2. Verify contact hasn't replied (auto-stop on response)
3. Check sender account warmup daily limit; reschedule to next day if exhausted
4. Select Gmail sender account (rotation + quota-aware — see Campaign Service)
5. Render email template with contact variable substitution
6. Send via Gmail OAuth API
7. Update StepExecution → SENT, increment sendsToday
8. On failure: retry with exponential backoff → Dead Letter Queue after max retries

Cron Jobs

Name	Schedule	Action
Email Warmup Reset	Daily midnight UTC	Resets sendsToday = 0 on all sender accounts; increments warmup progression limits
Follow-up Processor	Periodic	Finds contacts past their follow-up window and re-enqueues

Error Handling

Scenario	Behavior
Campaign paused mid-flight	Step worker enqueues cancel_active_step job; pending executions cancelled
Contact replied	Per-contact jobs check reply status and skip
Sender quota exceeded	Job is rescheduled to next day rather than failing
Provider error (4xx/5xx)	BullMQ retry with exponential backoff
Max retries exhausted	Job moved to Dead Letter Queue; metrics logged

Monitoring

• Bull Board UI — accessible at /bull on app:3000. Shows queue depths, active/waiting/failed jobs, and retry controls. Password protected.
• Health Check — GET /health on worker:3007. Returns service status; used by ECS health checks.
• CloudWatch Logs — All worker activity is forwarded to cloudwatch:3003 for structured log storage.

Production Notes

• Worker runs as a separate ECS task from app, allowing independent scaling.
• Redis is the only dependency between app and worker — no direct HTTP calls between them.
• Multiple worker instances can run concurrently; BullMQ handles job locking to prevent duplicate processing.