Skip to content

Net-Spider is a web scraping tool designed to retrieve the source code for a web page, including front-end elements such as JavaScript, CSS, images, and fonts. It allows you to crawl and download the source code from a target website.

License

Notifications You must be signed in to change notification settings

ahmed-alnassif/net-spider

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

29 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Net-Spdier

This tool scrapes the source code of an HTML page, including all the content within it, such as external files like JavaScript, CSS, images, fonts, and more. Additionally, it analyzes the CSS files and extracts any external links found within them. The tool is designed to assist developers in understanding the code used by large companies, focusing specifically on front-end development. It only retrieves front-end code and does not handle back-end or server-side code.

Net-Spider

Installation

Packages needed for installation

Arch Linux:

sudo pacman -Syu git python python-pip

Fedora:

sudo dnf update
sudo dnf install git python python-pip

Ubuntu or Debian:

sudo apt update
sudo apt install git python python-pip -y

Termux (Android):

apt update
apt install git python -y

Installation

git clone --depth=1 https://github.com/ahmed-alnassif/net-spider.git

Installation packages python

cd net-spider
python -m pip install -r requirements.txt
# OR
python setup.py install

Run

python main.py -u [url]
# Example
python main.py -u https://example.com

Usage

python main.py --help
  _____   __    _____     ________       ______________
___  | / /______  /_    __  ___/__________(_)_____  /____________
__   |/ /_  _ \  __/    _____ \___  __ \_  /_  __  /_  _ \_  ___/
_  /|  / /  __/ /_      ____/ /__  /_/ /  / / /_/ / /  __/  /
/_/ |_/  \___/\__/      /____/ _  .___//_/  \__,_/  \___//_/
                               /_/

Created By: Ahmed Al-Nassif  
Github: https://github.com/ahmed-alnassif  
E-mail: [email protected]  
Facebook: https://fb.me/ahm3d.nassif  
Program version: 24.6.4  

usage: python main.py -u [url]

This project designed to retrieve the source code for a web page, including front-end elements such as JavaScript, CSS, images, and fonts.

Net-Spider:
  --help          Show usage and help parameters
  -u              Target URL (e.g. http://example.com)
  -d, --domain    Pull links from the primary website address only
  --name NAME     The name of the folder in which to save the site files
  --hide          Hide the progress bar [----]
  -v              Give more output.
  --page RAW      Parse an HTML file and retrieve all files from it
  --update        Automatically update the tool

Requests settings:
  --random-agent  Random user agent
  --mobile        make requests as mobile default: PC, Note: this option will active random agent
  --cookie        Set cookie (e.g {"ID": "1094200543"})
  --header        Set header (e.g {"User-Agent": "Chrome Browser"})
  --proxy PROXY   Set proxy (e.g. {"https":"https://10.10.1.10:1080"})

Screenshots

screenshot tool net-spider
screenshot tool net-spider
net-spider WhatsApp

About

Net-Spider is a web scraping tool designed to retrieve the source code for a web page, including front-end elements such as JavaScript, CSS, images, and fonts. It allows you to crawl and download the source code from a target website.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages