Interview Experience | AI Infra | π₯π₯π₯ | Amazon
2026 (AugβSep) Software Development Engineer - AI Infrastructure @ AWS β Virtual Interview | π Neutral | Medium-Hard | Pending
The team sits under AWS and works on inference. No LeetCode in this round. The interviewer saw vLLM on my resume, so the first few minutes were project questions, and then almost everything after that was one system design.
He never stopped at my first version. After each design he asked the next thing: what if there were more machines, what if the version was wrong, what if the rollout was halfway done, what if you needed to scale right now.
Virtual interview β PagedAttention, weights, and rollout
PagedAttention
The first question was direct. Explain how PagedAttention works in vLLM.
I had prepared this one. The KV cache no longer needs one whole contiguous block of GPU memory per request. It is split into blocks, and blocks get allocated when they are needed instead of being reserved up front.
He asked why that matters for serving.
I said memory waste, and sequences having different lengths. If you reserve the maximum length for every request, most of that reservation is never used, and a short request ends up costing the same as a long one. With blocks you pay for what the sequence actually uses, so more sequences fit on the same GPU, which is what the throughput comes down to.
Then block size. How do you choose it.
This part was not as smooth. I said too large a block wastes memory at the tail of a sequence, and too small a block means more blocks to manage and more overhead per lookup. I did not remember the actual default or the range people tune, so I stopped there instead of guessing.
Model weights distribution
The design prompt was a few words: model weights distribution.
I confirmed the scope first. Roughly: one very large model, deployed to many inference machines, how do the weights get to all of them.
My first version was simple. Put the weights in object storage, and when a new machine comes up it downloads the artifact for its model version.
He asked: not ten machines, but a few thousand starting at the same time.
That is when I started thinking about cache layers. If every machine pulls a few hundred GB from the same place at the same moment, both startup time and the network fall over. So I changed the design: the origin keeps the full artifact, each region or cluster has a cache layer in front of it, and a new machine pulls from whichever copy is closest to it.
Then he asked how I guarantee that every machine gets the same version.
I said the file name cannot be the thing you trust. There should be an immutable version, plus a manifest and checksums. A machine verifies the download and only then loads it.
We stayed on this part for a while.
The question I did not see coming: the new version has already gone out to some of the machines, and then someone finds a problem with the model. What do you do.
My first reaction was rollback.
He asked how.
That is where I had skipped something. You cannot overwrite the old weights. I only got there after he pushed on it: keep both versions, and record separately which version is currently serving. Then a rollback is switching back to the previous stable version, not re-downloading anything or rebuilding it.
EKS deployment and autoscaling
He asked how I would set up the deployment strategy on EKS.
Not a full Kubernetes architecture. The question was how to avoid replacing all the model servers at once. I said rolling with a canary in front: bring up a small subset first, look at health and metrics, and continue only if that subset is fine.
What metrics count as fine.
I said at least error rate, latency, startup failures, and some model-specific validation. I was scattered here. I mixed infrastructure metrics and model quality metrics into the same answer.
The last question was how to set up autoscaling.
I said GPU utilization first.
He asked whether scaling on utilization is already too late. That is a fair point. A new inference node cannot take traffic the moment it exists. It still has to pull the weights and load the model.
So I added queue depth and request latency as earlier signals. I also said that for models with steady traffic you probably want to keep some warm capacity instead of waiting for a metric to move. He then went back to the beginning of the design: with more nodes, does weights distribution become a bottleneck again.
Behavioral
The last few minutes were not a problem. He asked about the hardest technical decision I have made.
I talked about a time I picked the simpler option so I could hit the deadline, and had to redo the work once traffic grew.
He did not make me walk through the whole STAR structure. He kept asking instead: why did you decide that at the time, did you not know the risks, and if you did it again, which assumption would you validate first.
Those follow-ups were harder than the question itself.
PagedAttention felt fine. On weights distribution there were a few points I only reached after he asked. Rollback especially. My first answer, "switch back to the old version", was too shallow.