How to write a webcrawler for Large Language Models at scale (Part – 3)
Introduction Lets discuss the architecture design of how to run multiple webcrawler for different websites to collect desired data for large language Models. How one should go about designing such systems? In this discussion I shall pan out the details about the design in the following order. Define the scope of the problem Let me…