Anthropic | General SWE | Master's | Full-time | 2026 Q1 | headhunter | technical phone | Fail | job switch
The question was a web crawler. Start from one URL and return all reachable URLs under the same host.
Coding
I used BFS. The queue held URLs that still needed to be processed, and a set handled deduplication. For each URL, I called the parser, went through the links on that page, and added links that had the same host and had not been seen before. DFS would work too. BFS was just easier to write here.
One small detail was URL handling. Relative links need to be resolved against the current page. Fragments should not create a new crawl task, and anything outside the starting host should be ignored. I normalized the URL before putting it in the set so the same page would not be crawled under slightly different strings.
The follow-up was how to make it multithreaded. I said I would use a fixed-size thread pool, a blocking queue, and a concurrent set. The check and add on the visited set have to happen together. Otherwise two workers can discover the same link at the same time and both put it into the queue.
The next question was when the crawl is actually finished. An empty queue is not enough because another worker may still be fetching a page and may add more URLs later. My first answer only checked whether the queue was empty. After the follow-up, I changed it to track pending work. The counter goes up when a new URL is accepted and goes down after that URL has been processed. The crawl can stop when the count reaches zero. The counter update and worker wake-up also have to be synchronized so a worker does not exit early.
There was also a short follow-up about request failures. I said each fetch should have a timeout and a small retry limit. After the limit, record the failure and move on. I would keep retry state separate from the visited set; removing the URL from visited could let several workers schedule it again.
I did not finish all of the concurrent code. Most of that part stayed as discussion and partial implementation. That was the whole screen, and I got rejected afterward.