Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Selenium failing on Ubuntu #631

Open
digininja opened this issue Nov 29, 2023 · 26 comments
Open

Selenium failing on Ubuntu #631

digininja opened this issue Nov 29, 2023 · 26 comments

Comments

@digininja
Copy link
Contributor

OS Used - ALL Information (architecture, linux flavor, etc.)

Ubuntu 22.04.3 LTS - fully up to date

Pulled from repo and setup ran.

error you are encountering

When it runs, it sits for ages showing nothing then finally errors with:

Message: Failed to read marionette port

Hitting ctrl-c gives this stack trace.

^CTraceback (most recent call last):
  File "/tmp/EyeWitness/Python/./EyeWitness.py", line 425, in <module>
    single_mode(cli_parsed)
  File "/tmp/EyeWitness/Python/./EyeWitness.py", line 245, in single_mode
    driver = create_driver(cli_parsed)
  File "/tmp/EyeWitness/Python/modules/selenium_module.py", line 79, in create_driver
    driver = webdriver.Firefox(profile, capabilities=capabilities, options=options, service_log_path=cli_parsed.selenium_log_path)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/firefox/webdriver.py", line 201, in __init__
    super().__init__(command_executor=executor, options=options, keep_alive=True)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 286, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 378, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 438, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/remote_connection.py", line 290, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/remote_connection.py", line 311, in _request
    response = self._conn.request(method, url, body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/usr/lib/python3/dist-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 446, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 441, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt

Expected behavior (vs. what you encountered)

Ideally it should take a screenshot.

Any additional information

Silent whispers speak,
Eyes bear witness to truth,
Moments etched in gaze.
@0x6d6f7468
Copy link
Collaborator

0x6d6f7468 commented Jan 4, 2024 •
edited
Loading

Hey digi, are you still having this issue? I've not seen it yet running on Parrot.

Also out of curiosity, assuming it still happens, does anything change if you bump the geckodriver version from 0.32.0 to the latest (0.34.0)?

[EDIT] Crazy that I've gotten this far in my career without learning how to read... Not confident that the geckodriver version would interact with Selenium, but still curious if this issue persists on your setup.

@digininja
Copy link
Contributor Author

I've just checked and after an update it was still failing and updating to 0.34.0 it also fails.

@0x6d6f7468
Copy link
Collaborator

Hmm, interesting. Using Selenium 4.9.1? Just checking, I know the installer currently forces that version but wondering if Ubuntu possibly ships with Selenium of a different version.

@digininja
Copy link
Contributor Author

digininja commented Jan 5, 2024 via email

@0x6d6f7468
Copy link
Collaborator

Should be. I'm going to check on an Ubuntu system whether my hunch is correct.

The process would just be to uninstall the existing Selenium and then python3 -m pip install selenium==4.9.1. If you want to check the current Selenium version, run python3 -c "import selenium; print(selenium.__version__)".

As far as I know, Pip will not overwrite an existing package unless forced, and I don't think the setup script explicitly warns about that currently. This is mostly just a sanity check for me before I decide to start tearing into things.

@digininja
Copy link
Contributor Author

digininja commented Jan 5, 2024 via email

@digininja
Copy link
Contributor Author

I've just checked, there are no system Selenium packages installed:

$ dpkg --get-selections selenium
dpkg: no packages found matching selenium

Trying to install 4.9.1 says it is already installed:

$ sudo python3 -m pip install selenium==4.9.1
[sudo] password for robin: 
Requirement already satisfied: selenium==4.9.1 in /usr/local/lib/python3.10/dist-packages (4.9.1)
Requirement already satisfied: urllib3<3,>=1.26 in /usr/lib/python3/dist-packages (from urllib3[socks]<3,>=1.26->selenium==4.9.1) (1.26.5)
Requirement already satisfied: trio~=0.17 in /usr/local/lib/python3.10/dist-packages (from selenium==4.9.1) (0.22.0)
Requirement already satisfied: trio-websocket~=0.9 in /usr/local/lib/python3.10/dist-packages (from selenium==4.9.1) (0.9.2)
Requirement already satisfied: certifi>=2021.10.8 in /usr/local/lib/python3.10/dist-packages (from selenium==4.9.1) (2022.9.24)
Requirement already satisfied: attrs>=19.2.0 in /usr/lib/python3/dist-packages (from trio~=0.17->selenium==4.9.1) (21.2.0)
Requirement already satisfied: sortedcontainers in /usr/lib/python3/dist-packages (from trio~=0.17->selenium==4.9.1) (2.1.0)
Requirement already satisfied: async-generator>=1.9 in /usr/local/lib/python3.10/dist-packages (from trio~=0.17->selenium==4.9.1) (1.10)
Requirement already satisfied: idna in /usr/lib/python3/dist-packages (from trio~=0.17->selenium==4.9.1) (3.3)
Requirement already satisfied: outcome in /usr/local/lib/python3.10/dist-packages (from trio~=0.17->selenium==4.9.1) (1.2.0)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from trio~=0.17->selenium==4.9.1) (1.3.0)
Requirement already satisfied: exceptiongroup>=1.0.0rc9 in /usr/local/lib/python3.10/dist-packages (from trio~=0.17->selenium==4.9.1) (1.0.0rc9)
Requirement already satisfied: wsproto>=0.14 in /usr/lib/python3/dist-packages (from trio-websocket~=0.9->selenium==4.9.1) (1.0.0)
Requirement already satisfied: PySocks!=1.5.7,<2.0,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from urllib3[socks]<3,>=1.26->selenium==4.9.1) (1.7.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

I've uninstalled it:

$ sudo python3 -m pip uninstall selenium==4.9.1
Found existing installation: selenium 4.9.1
Uninstalling selenium-4.9.1:
  Would remove:
    /usr/local/lib/python3.10/dist-packages/selenium-4.9.1.dist-info/*
    /usr/local/lib/python3.10/dist-packages/selenium/*
Proceed (Y/n)? y
  Successfully uninstalled selenium-4.9.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

and reinstalled it:

$ sudo python3 -m pip install selenium==4.9.1
Collecting selenium==4.9.1
  Downloading selenium-4.9.1-py3-none-any.whl (6.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 13.8 MB/s eta 0:00:00
Requirement already satisfied: urllib3<3,>=1.26 in /usr/lib/python3/dist-packages (from urllib3[socks]<3,>=1.26->selenium==4.9.1) (1.26.5)
Requirement already satisfied: trio~=0.17 in /usr/local/lib/python3.10/dist-packages (from selenium==4.9.1) (0.22.0)
Requirement already satisfied: trio-websocket~=0.9 in /usr/local/lib/python3.10/dist-packages (from selenium==4.9.1) (0.9.2)
Requirement already satisfied: certifi>=2021.10.8 in /usr/local/lib/python3.10/dist-packages (from selenium==4.9.1) (2022.9.24)
Requirement already satisfied: attrs>=19.2.0 in /usr/lib/python3/dist-packages (from trio~=0.17->selenium==4.9.1) (21.2.0)
Requirement already satisfied: sortedcontainers in /usr/lib/python3/dist-packages (from trio~=0.17->selenium==4.9.1) (2.1.0)
Requirement already satisfied: async-generator>=1.9 in /usr/local/lib/python3.10/dist-packages (from trio~=0.17->selenium==4.9.1) (1.10)
Requirement already satisfied: idna in /usr/lib/python3/dist-packages (from trio~=0.17->selenium==4.9.1) (3.3)
Requirement already satisfied: outcome in /usr/local/lib/python3.10/dist-packages (from trio~=0.17->selenium==4.9.1) (1.2.0)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from trio~=0.17->selenium==4.9.1) (1.3.0)
Requirement already satisfied: exceptiongroup>=1.0.0rc9 in /usr/local/lib/python3.10/dist-packages (from trio~=0.17->selenium==4.9.1) (1.0.0rc9)
Requirement already satisfied: wsproto>=0.14 in /usr/lib/python3/dist-packages (from trio-websocket~=0.9->selenium==4.9.1) (1.0.0)
Requirement already satisfied: PySocks!=1.5.7,<2.0,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from urllib3[socks]<3,>=1.26->selenium==4.9.1) (1.7.1)
Installing collected packages: selenium
  Attempting uninstall: selenium
    Found existing installation: selenium 4.0.0a1
    Uninstalling selenium-4.0.0a1:
      Successfully uninstalled selenium-4.0.0a1
Successfully installed selenium-4.9.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

And it is still broken.

I've also tried removing the pip versions of the libraries and found I've already got all the apt versions of them installed, however EW can't see netaddr despite it being installed:

$ sudo apt install -y python3-fuzzywuzzy python3-pyvirtualdisplay  python3-netaddr python3-levenshtein
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3-netaddr is already the newest version (0.8.0-2).
python3-fuzzywuzzy is already the newest version (0.18.0-2).
python3-levenshtein is already the newest version (0.12.2-2build1).
python3-pyvirtualdisplay is already the newest version (2.2-1).
0 to upgrade, 0 to newly install, 0 to remove and 6 not to upgrade.

$ ./EyeWitness.py --single https://digi.ninja
Traceback (most recent call last):
  File "/home/robin/tools/web/EyeWitness/Python/./EyeWitness.py", line 13, in <module>
    from modules import db_manager
  File "/home/robin/tools/web/EyeWitness/Python/modules/db_manager.py", line 4, in <module>
    from modules.objects import HTTPTableObject
  File "/home/robin/tools/web/EyeWitness/Python/modules/objects.py", line 5, in <module>
    from modules.helpers import strip_nonalphanum
  File "/home/robin/tools/web/EyeWitness/Python/modules/helpers.py", line 12, in <module>
    from netaddr import IPAddress
ModuleNotFoundError: No module named 'netaddr'

Any ideas?

@digininja
Copy link
Contributor Author

By coincidence, I do a yearly test for a client from a Kali box on their network. This box hasn't been turned on for a year so I've just updated EW and it has ran fine.

There is definitely something different between Kali and Ubuntu.

@0x6d6f7468
Copy link
Collaborator

I'll have to get an Ubuntu machine up to test, I'll report back with what I find. Thanks for the additional information and troubleshooting efforts.

@0x6d6f7468
Copy link
Collaborator

0x6d6f7468 commented Jan 9, 2024 •
edited
Loading

Able to replicate on an Ubuntu 22.04 VM. The fix appears to be the same as in issue #604.

If you would when you have a moment, can you follow the instructions linked in that issue to uninstall the snap store Firefox and install the proper version from repo?

Assuming that fixes the problem, I don't think we should close this issue just yet until we address this in the setup.sh script, as this feels like something we should be accounting for during setup.

Thanks.

@digininja
Copy link
Contributor Author

digininja commented Jan 9, 2024 via email

@digininja
Copy link
Contributor Author

Worked first time!

Coming up with a fix for this is going to be tricky as forcing people off the snap version just to run this tool doesn't feel like it would go down very well.

@0x6d6f7468
Copy link
Collaborator

Agreed, though right now the solution appears to be either overwriting the snap version automatically or prompting the user to do it manually. I feel like if Ubuntu users are already going to be required to do it manually to use EW, it might not make much difference to do it in the setup, assuming the user is notified beforehand.

Alternatively, maybe we should compare the snap and apt Firefox versions to see if there's a possible fix or workaround so this wouldn't be necessary. Might take a look at that as time allows, though I'm happy enough that a fix exists for the time being.

@digininja
Copy link
Contributor Author

digininja commented Jan 9, 2024 via email

@0x6d6f7468
Copy link
Collaborator

I think that you're on to something there, but thankfully might be wrong that it's not fixable. Check this out:

https://stackoverflow.com/a/72237794

I just tested it. Doing TMPDIR=./ python3 EyeWitness.py --single http://127.0.0.1:8000 worked.

@digininja
Copy link
Contributor Author

digininja commented Jan 9, 2024 via email

@0x6d6f7468
Copy link
Collaborator

Ugh, but it still feels more like a workaround than a "fix", doesn't it. Changing the invocation does seem less invasive than installing a different version of Firefox, at least...

Perhaps we can just update the README and/or prompt in the setup.sh script that those are the two options? I'll keep looking and see if there's a good/clean way to manage this.

@digininja
Copy link
Contributor Author

digininja commented Jan 9, 2024 via email

@0x6d6f7468
Copy link
Collaborator

That's one of the things I'm considering, yeah. I'm also looking at whether Selenium has a clean way to directly specify the temporary directory.

@0x6d6f7468
Copy link
Collaborator

Setting os.environ["TMPDIR"] = "./" in EyeWitness.py seems to work.

Perhaps a good solution would be to set the TMPDIR variable to the EyeWitness.py directory by default and add a new CLI argument so people can overwrite it if they want? I wonder if people would find that agreeable.

@digininja
Copy link
Contributor Author

digininja commented Jan 9, 2024 via email

@digininja
Copy link
Contributor Author

digininja commented Jan 9, 2024 via email

@0x6d6f7468
Copy link
Collaborator

Fair, yeah. Just teasing out the solution. I'm also still looking for whether you can just give Snap applications the ability to write to /tmp, though like you said that might not be possible to work around.

@digininja
Copy link
Contributor Author

digininja commented Jan 9, 2024 via email

@0x6d6f7468
Copy link
Collaborator

Actually, from what I'm reading, you can't even run snap applications from root. The current user is required to have a home directory under /home in order to function. So that might be a non-starter right there. I'm wondering if that might be a reason to avoid using the snap Firefox, though it's hard to say how many users will be trying to run EW on Ubuntu under the root user.

@digininja
Copy link
Contributor Author

digininja commented Jan 9, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@digininja @0x6d6f7468