In a previous role, I worked on a platform that processed claims using our own AI models. Every claim arrived as a bundle of documents, and speed was the whole point. This article covers the problems that come with a system like that, from scaling and event-driven design to running several models in production, and how I would handle them today with event-driven scaling on Kubernetes and KEDA.
The system I worked on
The goal was fast claim processing. A single claim could carry several kinds of documents: a PAN card for identity, invoices, and supporting reports.
We ran it on a Lambda-based architecture. Each document travelled to the model built for it. PAN cards went to the PAN model, invoices to the invoice model, reports to the report analysis model, and a separate fraud detection model looked for signs of fraud.
Splitting the work this way kept each model focused, and we could update one without touching the others. It also meant every model had its own traffic pattern, its own limits and its own ways to fail. Most of what follows comes from that.
1. Scaling on the right signal
With Lambda, scaling feels automatic at first. It scales with incoming events, but it comes with its own limits: account-level concurrency caps, cold starts that get worse as model packages grow, a 15-minute maximum run time, and no GPUs. Those limits hit hardest when every document type has its own model.
On Kubernetes, KEDA gives you more control. A ScaledObject scales a worker Deployment on queue length instead of CPU. For example, when an SQS queue grows, it scales the workers from 2 replicas to 20, then back down when the queue drains. Either way, the same traps apply:
Queue depth alone is misleading. A PAN card and a long, scanned report are not the same unit of work. Better signals are the age of the oldest message plus depth, or a separate queue per document type so each model scales on its own load.
In-flight messages are invisible. SQS
ApproximateNumberOfMessagesleaves out messages being processed. Scale on that alone and KEDA can remove workers while they are mid-job. AddApproximateNumberOfMessagesNotVisibleto the calculation.Scale-down kills work in progress. A worker stopped halfway through inference wastes compute and causes duplicate processing. Set
terminationGracePeriodSecondsproperly, use a preStop hook to stop polling, and let the current message finish.ScaledObject vs ScaledJob. For long-running jobs, ScaledJob is often the better fit. Each message gets its own Job, so scaling down never interrupts anything.
Visibility timeout mismatch. If processing takes longer than the visibility timeout, the message reappears and another worker picks it up. That is how one claim gets processed twice.
Flapping. Claim traffic comes in bursts, which causes scale up and down churn. Tune
cooldownPeriod,pollingIntervaland the HPA stabilization window.Cold starts. Scaling to zero saves money, but the first claim after a quiet period waits for a container, an image pull and a model load. The bigger the model, the longer the wait.
2. The bottleneck just moves downstream
Adding workers rarely fixes the system on its own. In a pipeline with several models, a claim is only as fast as its slowest document. Scale the PAN model all you like; if the report model is backed up, the claim still waits.
Other places the pressure lands:
Database connection pools (20 workers x 10 connections each adds up fast)
Rate limits on any external OCR or AI service, where more workers just means more 429 errors
The core claims or hospital system that has to accept the results
Node provisioning lag on Kubernetes: pods are ready to schedule, but Karpenter or Cluster Autoscaler takes minutes to add nodes, especially GPU nodes
The fix is back pressure. Cap maxReplicaCount (or Lambda reserved concurrency) at what the downstream systems can handle, not at what the queue wants.
3. Event-driven design problems
Routing documents to separate models is clean on a whiteboard. In production, the gaps show up between the boxes.
Wrong routing. If a document is classified wrongly, it goes to the wrong model and comes back with a confident but useless answer. You need a classification step you can trust, and a fallback path for documents nobody recognises.
Putting the claim back together. Documents are processed in parallel, but the claim is only done when all of them are. Something has to track claim state across separate functions, such as Step Functions or a state table, and decide what happens when one document fails.
At-least-once delivery means duplicates. Workers must be idempotent, for example by deduplicating on document ID or content hash. A claim paid twice is a real cost, not just a data issue.
Poison messages. A corrupted PDF fails forever and keeps eating capacity. Use a dead-letter queue with retry limits, and have someone review what lands there.
Large payloads. Don't push documents through the queue. Store them in S3 and pass a pointer (the claim-check pattern).
No priority. An urgent claim should not wait behind a backlog of routine ones. Use separate queues per priority, each with its own scaling rule.
4. MLOps: running four models, not one
One model in production is a project. Four models that depend on each other is an operation, and each one needs its own care.
Model size vs the runtime. Large models make every new instance slow to become useful, and Lambda caps package size and memory. Options are baked container images, a shared model cache, or a separate model server (Triton, KServe) that workers call.
GPU scheduling. GPUs are expensive, scarce and slow to provision. Scaling to 20 workers means nothing if only 4 GPUs exist.
Version tracking per model. Each model ships on its own schedule. Halfway through a rollout, some documents are handled by v1 and some by v2. Record which version of which model touched every document, or you can't explain a decision later.
Drift. A new PAN card print format, a new invoice layout from a partner, or poor-quality phone scans can quietly lower accuracy. Track confidence scores and extraction quality per model over time.
Fraud labels arrive late. You often learn a claim was fraudulent weeks after it was processed. The fraud model needs a feedback loop that brings those outcomes back into training.
Human review. Low-confidence results should go to a review queue, which is itself a queue that needs scaling and an SLA.
Training on sensitive data. Consent, masking and a clear record of what data went into each training set.
5. AIOps and observability
When a claim is slow, the first question is always "where is it stuck?" With documents spread across several models, that question is harder than it sounds.
One ID through the whole journey. A claim passes through upload, routing, several models and a final decision. Without a correlation ID carried through every step, debugging is guesswork.
Per-model and per-claim views. You need dashboards for each model (queue age, errors, latency) and a way to see a single claim end to end.
Alert fatigue. Queue alerts that fire every time a normal daily peak arrives get ignored. Anomaly detection should learn daily and weekly patterns, such as month-end claim surges.
The right SLOs. "Functions healthy" is not the metric. "95% of claims processed within 5 minutes" is.
Cost per claim. Autoscaling without a cost view can quietly double the bill. Track compute and model cost per document type and per claim.
Silent failures. The function succeeds but extracts nothing useful. Health checks pass while output quality drops.
6. Healthcare and compliance
Claim documents are some of the most sensitive data a company holds. Identity documents like PAN cards sit right next to medical and financial records, and every autoscaled worker that touches them widens the risk.
PHI in logs and messages. Debug logs that dump document contents are a common leak. Encrypt queues with KMS and scrub logs.
Audit trails. Who or what accessed which record, and when, across every autoscaled pod.
Compliance. HIPAA if US-facing; for India, the DPDP Act and ABDM guidelines. Data residency affects which regions and AI APIs you can use.
Availability expectations. Clinical workflows can't tolerate "the queue was backed up for 3 hours."
What I would do differently today
Scale on queue age and in-flight work, not just queue depth
Give each document type, and each priority, its own queue and scaling rule
Cap workers at what downstream systems can take
Make every worker idempotent and add a dead-letter queue
Track claim state so parallel documents come back together cleanly
Record which version of which model handled every document
Trace every claim end to end with one ID
Keep identity and health data out of logs
None of this is exotic. Most of it is the boring work that decides whether an AI pipeline holds up on its busiest day.
