Skip to content

aiohttp-like interface to chromium. based on selenium_driverless to bypass cloudflare

License

Notifications You must be signed in to change notification settings

milahu/aiohttp_chromium

Repository files navigation

aiohttp_chromium

aiohttp-like interface to chromium

based on selenium_driverless to bypass cloudflare

status

working prototype

usage

aiohttp_chromium is a drop-in replacement for aiohttp

import asyncio

#import aiohttp
import aiohttp_chromium as aiohttp

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://httpbin.org/get') as resp:
            print(resp.status)
            print(await resp.text())

asyncio.run(main())

see also

why

handling file downloads with selenium is too verbose, and too complex to integrate into selenium, so this is a wrapper for selenium

i wanted a "stupid http client", so it has the same interface as aiohttp.client, and handling web pages has lower priority, so the selenium interface is hidden in response._driver

known issues

chromium window is stealing focus

when creating new tabs, or when switching between tabs, the chromium window is grabbing focus

this is an issue with the window manager

workaround for the KDE plasma desktop: move the chromium window to a different desktop, and focus some window

chromium seems to have no command line switch to disable this focus-grabbing

possible solutions

  • run chromium in a LD_PRELOAD wrapper
  • binary patching of the chromium executable
  • configure the window manager

todo

keywords

  • web scraper
  • chromium
  • aiohttp
  • web scraping
  • asyncio
  • bypass cloudflare
  • headful scraper
  • headful web scraper
  • headful chromium
  • gui scripting
  • headful webscraper
  • selenium driverless

similar projects