SFTP problem #199

adi- · 2016-09-08T11:52:08Z

I get this error:

File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/paramiko/client.py", line 70, in __init__
    self._system_host_keys = HostKeys()
RecursionError: maximum recursion depth exceeded

which really doesn't say much about the real problem. I know that there is a mkdir loop, but what causes it?

The text was updated successfully, but these errors were encountered:

ZuluPro · 2016-09-08T13:05:19Z

@adi- Thank for reporting this error, I'll look at it deeper this evening.
Can you give us more about how you configure it ?

adi- · 2016-09-08T13:10:07Z

The basic I have started with was:

DBBACKUP_STORAGE = 'storages.backends.sftpstorage.SFTPStorage'
DBBACKUP_STORAGE_OPTIONS = {
  'host': '__server_address__',
  'params': {
    'username': '__username__',
    'password': '__password__',
  }
}

I have also tried setting custom root_path, file_mode, 'dir_mode` but without any luck.

ZuluPro · 2016-09-08T13:16:28Z

Could we also have the full traceback ?

adi- · 2016-09-08T13:21:50Z

$ ./manage.py dbbackup

Backing Up Database: ***
Backup size: 1.1 KiB
Writing file to ***-2016-09-08-121004.psql
Writing file ***-2016-09-08-121004.psql
Writing file ***-2016-09-08-121004.psql
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)   
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/dbbackup/utils.py", line 99, in wrapper
    func(*args, **kwargs)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/dbbackup/management/commands/dbbackup.py", line 55, in handle
    self._save_new_backup(database)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/dbbackup/management/commands/dbbackup.py", line 81, in _save_new_backup
    self.write_to_storage(outputfile, filename)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/dbbackup/management/commands/_base.py", line 83, in write_to_storage
   self.storage.write_file(file, path)   
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/dbbackup/storage.py", line 82, in write_file
    self.storage.save(name=filename, content=filehandle)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 132, in _save
    self._mkdir(dirname)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)

[...]

  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 116, in _mkdir
    if not self.exists(parent):
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 153, in exists
    self.sftp.stat(remote_path)
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 85, in sftp
    self._connect()
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/storages/backends/sftpstorage.py", line 51, in _connect
    self._ssh = paramiko.SSHClient()
  File "/home/vagrant/.virtualenvs/gazex/lib/python3.5/site-packages/paramiko/client.py", line 70, in __init__
    self._system_host_keys = HostKeys()   
RecursionError: maximum recursion depth exceeded

jbittel · 2016-09-16T17:35:05Z

I've not discovered the exact cause, but I was able to resolve this problem by setting SFTP_KNOWN_HOST_FILE with a path to a file containing the host keys for the target server. If there's a better or simpler way to resolve this, I'd be interested.

Thanks!

adi- · 2016-09-17T09:13:14Z

Still doesn't work. Could you paste your settings here?

aheld · 2016-09-17T16:16:05Z

Using a a windows path I can replicate this defect. the code checks if the parent exists, and if not then it tries to create it.

Using a windows path the 'dirname' becomes and empty string, and then calling "exists" on an empty string says that it does not exist and tries to create it. -- infinite loop.

>>> posixpath.dirname('/home/user')
'/home'
>>> posixpath.dirname('home\user')
''
>>> sfpBackend.exists('')
False
>>> sf._mkdir('\home\aheld\test')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/aheld/.virtualenvs/low_income/src/django-storages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/Users/aheld/.virtualenvs/low_income/src/django-storages/storages/backends/sftpstorage.py", line 117, in _mkdir
    self._mkdir(parent)
  File "/Users/aheld/.virtualenvs/low_income/src/django-storages/storages/backends/sftpstorage.py", line 117, in _mkdir

...

RuntimeError: maximum recursion depth exceeded in __instancecheck__

there needs to be a sanity check somewhere around
https://github.com/jschneier/django-storages/blob/master/storages/backends/sftpstorage.py#L118

jschneier · 2016-09-19T02:51:33Z

@aheld interesting. @adi- can you confirm that the target system is a Windows box?

I noticed that we explicitly don't support it in the codebase but we could pretty easily add an optional dependency (for SFTP only) on backports.pathlib for the pre-3.4 versions.

adi- · 2016-09-19T06:21:13Z

Ubuntu 16.04 ;)

aheld · 2016-09-19T13:00:26Z

what is the path being set? If posixpath.dirname returns an empty string then you will get this, windows pathnames (backslashes) was just an initial guess

jbittel · 2016-09-19T15:04:56Z

Here's the settings I'm using, and both the source and target servers are Ubuntu:

SFTP_STORAGE_HOST = '<hostname>'
SFTP_STORAGE_ROOT = '/var/www/media/'
SFTP_STORAGE_PARAMS = {
    'username': '<username>',
    'password': '<password>',
    'allow_agent': False,
    'look_for_keys': False,
}
SFTP_KNOWN_HOST_FILE = '/var/www/data/known_hosts'
SFTP_STORAGE_INTERACTIVE = False

As I said above, if I remove the SFTP_KNOWN_HOST_FILE setting, or remove the file it points to, then I get the same infinite loop. Also note that the known_hosts file must have the correct restrictive permissions or it will not be used. I've verified that the path is correct and if you look at the full traceback you can see the recursion walking up the path one directory at a time until it reaches /.

ZuluPro · 2016-09-19T15:09:18Z

I think the real trouble is here: https://github.com/jschneier/django-storages/blob/master/storages/backends/sftpstorage.py#L78

We catch all exception without raise critical errors. Let me sometime to make a better try/except block

adi- · 2016-10-09T19:19:12Z

@jbittel How come you are using SFTP_ settings? Django-storages doesn't use those settings nor paramiko.

adi- · 2016-10-09T19:29:12Z

@jbittel ahhh, my bad. There is such an option. However, with your settings I cannot make it work either :(

adi- · 2016-10-09T19:51:13Z

Ok, make it work finally!

It will be good to mention about known_host_file in docs as in jazzband/django-dbbackup#220

adi- · 2016-10-09T20:23:44Z

So, this seems to be a duplicate of: #118

adi- · 2016-10-09T20:30:02Z

Closing...

ZuluPro added the bug label Sep 8, 2016

ZuluPro self-assigned this Sep 8, 2016

adi- mentioned this issue Oct 9, 2016

SFTP backup is buggy (fixable) jazzband/django-dbbackup#220

Open

adi- closed this as completed Oct 9, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SFTP problem #199

SFTP problem #199

adi- commented Sep 8, 2016

ZuluPro commented Sep 8, 2016

adi- commented Sep 8, 2016

ZuluPro commented Sep 8, 2016

adi- commented Sep 8, 2016

jbittel commented Sep 16, 2016

adi- commented Sep 17, 2016

aheld commented Sep 17, 2016 •

edited

Loading

jschneier commented Sep 19, 2016

adi- commented Sep 19, 2016

aheld commented Sep 19, 2016 •

edited

Loading

jbittel commented Sep 19, 2016

ZuluPro commented Sep 19, 2016

adi- commented Oct 9, 2016 •

edited

Loading

adi- commented Oct 9, 2016

adi- commented Oct 9, 2016

adi- commented Oct 9, 2016

adi- commented Oct 9, 2016

SFTP problem #199

SFTP problem #199

Comments

adi- commented Sep 8, 2016

ZuluPro commented Sep 8, 2016

adi- commented Sep 8, 2016

ZuluPro commented Sep 8, 2016

adi- commented Sep 8, 2016

jbittel commented Sep 16, 2016

adi- commented Sep 17, 2016

aheld commented Sep 17, 2016 • edited Loading

jschneier commented Sep 19, 2016

adi- commented Sep 19, 2016

aheld commented Sep 19, 2016 • edited Loading

jbittel commented Sep 19, 2016

ZuluPro commented Sep 19, 2016

adi- commented Oct 9, 2016 • edited Loading

adi- commented Oct 9, 2016

adi- commented Oct 9, 2016

adi- commented Oct 9, 2016

adi- commented Oct 9, 2016

aheld commented Sep 17, 2016 •

edited

Loading

aheld commented Sep 19, 2016 •

edited

Loading

adi- commented Oct 9, 2016 •

edited

Loading