Databricks Interview Question

How would you implement a web-crawler