ipfs datasource doesn't respect http.connection_timeout config field

[tezosmiami]

Steps to reproduce:
clone - https://github.com/tezosmiami/hicdex.git
run readme commands
What did you expect to happen:
for it to index
What actually happened:
it did index up to objkt #67865 - which appears to have lost metadata - not pinned - then dipdup jumps to the latest levels and keeps going from there without indexing - running all night. . .

Environment

on ubuntu 22.10 server

• Python version:
• 3.10.7
• DipDup version or commit hash:
• 6.5.5
• Database engine version:
• 14
• Hasura version:
• 2.23.0
• Are you in docker?
• yes
• Can you reproduce this issue on master?
• yes

logs.txt
config.txt
models.txt

[droserasprout]

Update datasource config to limit retries. . .

datasources:
  metadata:
    ...
    http:
      retry_count: 3

. . .than catch and process exceptions in the handler. Should help. . . HTTP gateway defaults will be updated in #677.

[tezosmiami]

no luck with changing retry count. . .

• see attached log + on_mint handler and utils files with exceptions
• maybe need some sort of timeout?
• is that what you mean by defaults?
  log.txt
  metadata_utils.txt
  on_mint.txt

[droserasprout]

Oh, it's ipfs datasource to blame, not metadata. Try:

try:
    coro = ipfs.get('asdfasdfasdf')
    data = await asyncio.wait_for(coro, 60)
except asyncio.TimeoutError:
    data = None

Normally all datasources should respect http.connection_timeout setting, defaulting to 60 seconds. If they don't it's possibly a bug.

[tezosmiami]

awesome - exactly it for this one - the metadata for that objkt is missing from all ipfs datasources - probably not pinned - was actually looking into async.wait_for() - this made it very clear - thanks again!

[droserasprout]

Nice. Leaving open to check retries and timeouts later.

[tezosmiami]

same thing seems to be happening sometimes with http request to tzkt_mainnet - sometimes goes back to indexing after hicdex container reboot. . .

log.txt