A Simple Web Crawler 🕷

A Simple Crawler with an option to crawl a particular number of links. Only crawl linkes of root url.Store all links to a CSV file.

Features:

• Crawl all the links and store in a CSV file
• Can crawl a certain number of links .
• Avoid crawl all other non root urls

Dependency:

• BeautifulSoup
• pip

How to Crawl :

• Install pip in your enviroment(As the code will auto install BeautifulSoup using pip)
• Then run the crawler.py in our terminal and give the inputs i.e root url and number of url to crawl
• If you want the crawler to crawl all links of a website then simply press enter when ask for the number of links.