Just some minor web crawlers.
Pull requests are always welcome.
- Required: License and copyright notice + State Changes + Disclose Source
- Permitted: Commercial Use + Modification + Distribution
- Forbidden: Hold Liable + Sublicensing
Get bill data from the National Assembly and structurize to json formats. (See attributes)
pip install -U celery-with-redis # Install dependencies
cd bills
cp settings.py.sample settings.py # Input data directory
python main.py
Get commentable bills from 국회입법예고
cd commentable_bills
python crawl.py # open and set datadir first
Get committee list data from the 위원회 현황
cd committee_list
python get.py # To get data files
Get Korean politicians' data from Korea Election Commission (중앙선거관리위원회).
This data contains the list of all people that have run for office in the National Asssmbly.
cd election_commission
python main.py -h
Get and merge data for POPONG Glossary from:
committee: Standing committee and Special Committee (국회상임위원회 및 특별위원회),
likms: Integrated Legislation Knowledge Management System (입법통합지식관리시스템),
nas: National Assembly Secretaritat (국회사무처).
python get.py # To get source data files
python merge.py # To create glossary.csv
Get Google search counts.
cd google
python ndocs.py
Get National Assembly meetings.
cd meetings
python crawl.py
Get National Assembly meetings calendar.
cd meetings_calendar
python get.py 2014-11-01 2014-11-11 # To get meetings schedule from 2014-11-01 to 2014-11-11 or
python get.py 2014-11-01 # To get meetings schedule at 2014-11-01
Get member information from the Korean National Assembly.
pip install Scrapy>=0.22.2
cd national_assembly
python crawl.py
Get news articles for recent bills from Naver News.
pip install psycopg2 lxml
cd naver_news
cp settings.py.sample settings.py
vi settings.py # fill in values
python crawl.py
Get People Power 21 (열려라국회) webpages. (Currently broken)
cd peoplepower
scrapy crawl peoplepower21
Get pledges from NEC (선거관리위원회) for 19th National Assembly officials.
cd pledges
python crawler.py
Get Korean politicians' data from ROKPS(헌정회).
cd rokps
python crawler.py
python parser.py
Get Korean lastnames from Wikipedia.
cd wikipedia
python wiki_lastnames.py
Get Wikipedia links for assembly members.
cd wikipedia
python assembly_members.py
Get Twitter follower lists for specified handles.
make twitter_setup
python twitter/followers.py