Scaling Kafka consumers on KEDA with IAM authentication

CPU is a bad autoscaling signal for a Kafka consumer. A consumer that is falling badly behind can sit at low CPU while it waits on a downstream call, and a consumer that is perfectly healthy can spike during a rebalance. The number that describes whether the system is keeping up is consumer lag, and scaling on anything else is scaling on a proxy for the thing you care about.

KEDA’s kafka scaler reads lag directly. The complication is authentication: with MSK using IAM, the scaler needs to authenticate as an AWS principal rather than with SASL/SCRAM credentials, which changes how the trigger authentication is wired.

The shape of the setup

Three pieces have to line up:

1. A TriggerAuthentication that points KEDA at an AWS identity — typically via a service account annotation and IRSA, not a static access key.
2. A service account with an IAM role attached, and that role granted the cluster and topic-level permissions the scaler needs.
3. A ScaledObject whose trigger names the consumer group and the lag threshold at which to add a replica.

The permission set is smaller than people expect: the scaler is only describing the group and reading offsets, not consuming. Granting it the same policy as the workload is the common shortcut and it’s worth not taking.

A trimmed trigger looks roughly like this:

triggers:
  - type: kafka
    metadata:
      bootstrapServers: boot-....amazonaws.com:9098
      consumerGroup: payments-worker
      lagThreshold: "100"
      offsetResetPolicy: latest
      sasl: aws_msk_iam
      tls: enable

The exact auth wiring lives in the TriggerAuthentication and the IRSA role, not in the trigger itself — keep credentials out of the ScaledObject.

The parts that were harder than the docs implied

The scaler authenticates separately from the workload. It’s easy to assume that because the consumer pods can reach the cluster, KEDA can too. KEDA runs in its own namespace under its own identity. Whichever identity the TriggerAuthentication resolves to needs its own grant.

Lag threshold is per-replica, not total. The threshold is compared against lag divided by replica count. Setting it as if it were a total lag figure gives you a fleet that scales far more aggressively than intended.

Partition count is the ceiling. KEDA won’t usefully scale a consumer group beyond the partition count of the topic, because the extra pods have nothing to consume. If the scaling maximum is higher than the partition count, the autoscaler looks broken when it’s actually correct.

Scaling to zero interacts with rebalancing. Every scale event triggers a consumer group rebalance, and a rebalance pauses consumption. An aggressively tuned scaler can spend a meaningful fraction of its time rebalancing. Cooldown period and stabilisation window matter more here than in a typical HTTP autoscaling setup.

What I’d check first if it isn’t working

1. KEDA operator logs — auth failures surface there, not on the ScaledObject.
2. Whether the consumer group has ever committed an offset. A brand-new group reports no lag, so it looks identical to a fully caught-up one.
3. That the IRSA role is attached to the identity the TriggerAuthentication actually resolves to, not the workload’s.
4. Partition count vs maxReplicaCount — if max is higher, you will “fail” to scale past a point that is correct behaviour.