Scrap Jobs

I split the system into five independently deployed services — an API, a worker, a scheduler, a Python agent for WhatsApp, and a React frontend — mainly to isolate failure domains, not to scale each one independently. The clearest case is headless scraping: it drives a real Chromium instance inside the worker, so it runs on its own queue at a concurrency of exactly one — a stuck tab can slow down headless scraping, but it can never starve matching, email or WhatsApp, because those run through separate queues and separate concurrency pools.

Every unit of work — scraping a site, matching a user against new jobs, sending a digest, answering a WhatsApp message — goes through Redis-backed queues (asynq), not a direct function call. That buys three things I actually use: the scheduler enqueues work and returns immediately instead of blocking on scraping or email; failed tasks retry automatically with backoff instead of silently disappearing; and uniqueness locks stop the same match or campaign task from being enqueued twice while one is already running.

A job posting is uniquely identified by the site it came from plus the source site’s own posting ID, enforced with a partial unique index on (site_id, requisition_id). Re-scraping a site that still lists the same posting doesn’t insert a new row — it hits ON CONFLICT and just bumps last_seen_at. A nightly cleanup job then deletes anything not seen in the last scrape window, so the table only ever holds postings that are still actually live.

The expensive step — an LLM call — never runs on the bulk path. Matching every active user against every new job, which happens twice a day, is pure keyword and location matching with no model involved. The LLM only runs when a user explicitly asks to analyze a job against their resume, and that result is cached per user and job so asking twice doesn’t cost twice. Every request is also structured as JSON output with a detailed system prompt instead of relying on temperature for consistency, and an idempotency ledger means a retried request returns the already-paid-for result instead of billing the same analysis again.

The WhatsApp agent itself holds no state — it’s a stateless Python service that receives the conversation history with every request and returns text. The actual state lives in Go: the last 20 turns and onboarding progress sit in Redis with a 7-day TTL for the hot path, while every message, every agent run and every tool call it made are written to Postgres for a durable, auditable history — including token counts and cost per run, so a runaway conversation shows up in the data, not just in a bill at the end of the month.