BUG: NFT transfer from Stargaze to IRISnet pended and didn't returned #329

heejin-github · 2023-03-08T06:32:06Z

Summary of Bug

NFT transfer from Stargaze to IRISnet pended, and didn't received in IRISnet, didn't returned to Stargaze also.

Environment

OS: Ubuntu 22.04.1 LTS
Software version:
- starsd: v8.1.0

Steps to Reproduce

This issue happened during the Task A19 (s --(1)-->i).

Tried to transfer NFTs from Stargaze to Irisnet. The settings were as follows.

Msg

# Json
{
  "receiver": "iaa1gtk6c7ft3mpp7cmsd39mw0ysqa6mf579gq67d5",
  "channel_id": "channel-207",
  "timeout": {
    "block": {
      "revision": 1,
      "height": 346000
    }
  }
}

# Encoded
ewogICJyZWNlaXZlciI6ICJpYWExZ3RrNmM3ZnQzbXBwN2Ntc2QzOW13MHlzcWE2bWY1NzlncTY3ZDUiLAogICJjaGFubmVsX2lkIjogImNoYW5uZWwtMjA3IiwKICAidGltZW91dCI6IHsKICAgICJibG9jayI6IHsKICAgICAgInJldmlzaW9uIjogMSwKICAgICAgImhlaWdodCI6IDM0NjAwMAogICAgfQogIH0KfQ==

Executed command

starsd tx wasm execute \
	stars1r9whurc74jqgdf7ftlfdxacngxv4z2u2ea6c9ph85rvp5qjsek4szdk7gy \
	'{"send_nft": {"contract": "stars1ve46fjrhcrum94c7d8yc2wsdz8cpuw73503e8qn9r44spr6dw0lsvmvtqh", "token_id": "dsrvNFTtokenA19", "msg": "ewogICJyZWNlaXZlciI6ICJpYWExZ3RrNmM3ZnQzbXBwN2Ntc2QzOW13MHlzcWE2bWY1NzlncTY3ZDUiLAogICJjaGFubmVsX2lkIjogImNoYW5uZWwtMjA3IiwKICAidGltZW91dCI6IHsKICAgICJibG9jayI6IHsKICAgICAgInJldmlzaW9uIjogMSwKICAgICAgImhlaWdodCI6IDM0NjAwMAogICAgfQogIH0KfQ=="}}' \
	--from=dsrvlabs \
	--gas-prices 1ustars \
    	--gas auto \
    	--gas-adjustment 1.3

The IBC Transfer Tx was successfully performed on Stargaze.

Success Tx: https://gon.ping.pub/stargaze/tx/7AD6FFCB159DB07C917DEE6DA4F8402DB9CDE9DD9121103CBD3241994C3BC732

Relayer tried to relay NFT transfer packets to IRISnet.
During the relay sequence, the timeout_height was reached and the corresponding Tx failed.

Failed Tx: https://gon.ping.pub/iris/tx/32A2E3AE1D0B00776BE364AD7F977885B5E1B1F375E9B6BE76CF99D8E22D147E

Expected and Actual Behavior

Expected Behavior : NFT owned by owner address. even if transfer Tx was failed.

$ starsd query wasm contract-state smart \
        stars1r9whurc74jqgdf7ftlfdxacngxv4z2u2ea6c9ph85rvp5qjsek4szdk7gy \
        '{"all_nft_info":{"token_id":"dsrvNFTtokenA19"}}' 
data:
  access:
    approvals: []
    owner: stars1gtk6c7ft3mpp7cmsd39mw0ysqa6mf579f7djy5
  info:
    extension: {}
    token_uri: https://www.dsrvlabs.com

Actual Behavior : NFT owned by contract.

$ starsd query wasm contract-state smart \
        stars1r9whurc74jqgdf7ftlfdxacngxv4z2u2ea6c9ph85rvp5qjsek4szdk7gy \
        '{"all_nft_info":{"token_id":"dsrvNFTtokenA19"}}' 
data:
  access:
    approvals: []
    owner: stars1ve46fjrhcrum94c7d8yc2wsdz8cpuw73503e8qn9r44spr6dw0lsvmvtqh
  info:
    extension: {}
    token_uri: https://www.dsrvlabs.com

Error Stack

https://gon.ping.pub/iris/tx/32A2E3AE1D0B00776BE364AD7F977885B5E1B1F375E9B6BE76CF99D8E22D147E
During the relay sequence, Tx failed with timeout error.

failed to execute message; message index: 1: receive packet verification failed: block height >= packet timeout height (1-346000 >= 1-346000): packet timeout

Additional Context

From the NFT user's perspective, if the NFT transfer fails as a result whether it's because the Tx failed or the timeout height was reached, they will expect it to returned back.
if it locked on contract, user will think that they've lost of the NFT.

The text was updated successfully, but these errors were encountered:

gjermundgaraba · 2023-03-09T08:18:10Z

I just want to confirm this problem. I am not sure if this happens because the cw721 contract is not handling the timeouts properly or where the exact problem lies (although I didn't find any actual tests of this in the cw721 repo, so unable to confirm). @shanev or @0xekez, would you be able to weigh in on this?

0xekez · 2023-03-09T15:59:58Z

hi, thanks for reporting this! the NFT won't get refunded unless the relayer commits the timeout message to Stargaze. do you perhaps have a link to that TX, or an error message if it failed?

0xekez · 2023-03-09T16:02:18Z

https://github.com/public-awesome/ics721/blob/main/e2e/suite_test.go#L682

here is a test that checks that NFTs are refunded on timeout that may be a good reference.

gjermundgaraba · 2023-03-09T16:41:39Z

Yes, I have the transaction on Stargaze (elgefar-1): 9A569DBF71A38CE3927E5EA05A24C585E39716AE31BAF21121960661C63D016D

0xekez · 2023-03-09T16:51:25Z

@gjermundgaraba - thanks for sending that over. it looks like that is the transaction that initiated the transfer (notice that this is the Send on the stargaze chain - ping.pub).

i'm curious if you have the timeout packet? you can see here in the test how getting the refund requires that a relayer sends a message on Stargaze informing the bridge that a timeout occured.

gjermundgaraba · 2023-03-09T17:07:35Z

Hmm, I am not able to find one, so perhaps it was never submitted.
So the question then becomes, why have no relayers commited a timeout message to stargaze? I have confirmed that relayers are running on these channels, but they are not picking up this.

gjermundgaraba · 2023-03-09T17:28:31Z

Aha! I was using the go relayer myself, and thinking perhaps it was a relayer software issue, I started up a hermes relayer as well, and it found the missing packets right away! My NFT is back!

0xekez · 2023-03-09T18:24:05Z

go hermes!

heejin-github · 2023-03-09T23:30:19Z

Hi, @gjermundgaraba and @0xekez
It seems that we've got similar issue on iris.
Tx was failed to relay from stargaze to iris because timeout reached.

https://gon.ping.pub/iris/tx/32A2E3AE1D0B00776BE364AD7F977885B5E1B1F375E9B6BE76CF99D8E22D147E

and I found the missing packets by hermes. but, it couldn't returned to stargaze because relayer couldn't find the NFT Class ID.
May I ask any idea for this?

failed to execute message; message index: 1: timeout packet callback failed: ibc/57316878F682A6541C5587C0A07BC345492C9FB0738A1619D1FF1649DE814F59: nft class does not exist

pending packets found.

# hermes query packet pending --chain gon-irishub-1 --port nft-transfer --channel channel-22
    Finished release [optimized] target(s) in 0.25s
     Running `/root/git/hermes/target/release/hermes query packet pending --chain gon-irishub-1 --port nft-transfer --channel channel-22`
2023-03-08T01:50:33.074950Z  INFO ThreadId(01) using default configuration from '/root/.hermes/config.toml'
SUCCESS Summary {
    src: PendingPackets {
        unreceived_packets: [
            Collated {
                start: Sequence(
                    74,
                ),
                end: Sequence(
                    74,
                ),
            },
        ],
        unreceived_acks: [],
    },
    dst: PendingPackets {
        unreceived_packets: [
            Collated {
                start: Sequence(
                    75,
                ),
                end: Sequence(
                    75,
                ),
            },
        ],
        unreceived_acks: [],
    },
}

trying to clear pending packets. but failed.

# hermes clear packets --chain gon-irishub-1 --channel channel-22 --port nft-transfer
    Finished release [optimized] target(s) in 0.24s
     Running `/root/git/hermes/target/release/hermes clear packets --chain gon-irishub-1 --channel channel-22 --port nft-transfer`
2023-03-08T02:48:29.811051Z  INFO ThreadId(01) using default configuration from '/root/.hermes/config.toml'
2023-03-08T02:48:33.420845Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=gon-irishub-1 src_port=nft-transfer src_channel=channel-22 dst_chain=elgafar-1}: 1 unreceived packets found: [ 74 ]
2023-03-08T02:48:33.918906Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=gon-irishub-1 src_port=nft-transfer src_channel=channel-22 dst_chain=elgafar-1}: pulled packet data for 1 events out of 1 sequences: 74..=74; events.total=1 events.left=0
2023-03-08T02:48:41.980326Z  INFO ThreadId(01) relay_recv_packet_and_timeout_messages{src_chain=gon-irishub-1 src_port=nft-transfer src_channel=channel-22 dst_chain=elgafar-1}:relay{odata=packet-recv ->Source @1-3756673; len=1}: assembled batch of 2 message(s)
2023-03-08T02:48:43.217676Z ERROR ThreadId(10) relay_recv_packet_and_timeout_messages{src_chain=gon-irishub-1 src_port=nft-transfer src_channel=channel-22 dst_chain=elgafar-1}:relay{odata=packet-recv ->Source @1-3756673; len=1}:send_messages_and_wait_commit{chain=gon-irishub-1 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=gon-irishub-1 account.sequence=0}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: timeout packet callback failed: ibc/57316878F682A6541C5587C0A07BC345492C9FB0738A1619D1FF1649DE814F59: nft class does not exist [cosmos/[email protected]/x/nft/keeper/nft.go:79] With gas wanted: '18446744073709551615' and gas used: '186328' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "356939"} }
2023-03-08T02:48:43.217739Z ERROR ThreadId(10) relay_recv_packet_and_timeout_messages{src_chain=gon-irishub-1 src_port=nft-transfer src_channel=channel-22 dst_chain=elgafar-1}:relay{odata=packet-recv ->Source @1-3756673; len=1}:send_messages_and_wait_commit{chain=gon-irishub-1 tracking_id=packet-recv}:send_tx_with_account_sequence_retry{chain=gon-irishub-1 account.sequence=0}: gas estimation failed or encountered another unrecoverable error error=gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: timeout packet callback failed: ibc/57316878F682A6541C5587C0A07BC345492C9FB0738A1619D1FF1649DE814F59: nft class does not exist [cosmos/[email protected]/x/nft/keeper/nft.go:79] With gas wanted: '18446744073709551615' and gas used: '186328' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "356939"} }
ERROR link error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: timeout packet callback failed: ibc/57316878F682A6541C5587C0A07BC345492C9FB0738A1619D1FF1649DE814F59: nft class does not exist [cosmos/[email protected]/x/nft/keeper/nft.go:79] With gas wanted: '18446744073709551615' and gas used: '186328' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "356939"} }

gjermundgaraba · 2023-03-10T14:55:21Z

This is very interesting! I am looking into it! :D

gjermundgaraba · 2023-03-10T20:09:36Z

I figured it out 🙏

This is the transaction in question: https://gon.ping.pub/iris/tx/99DE1DB6A917325B96EE8E073CBBC9F17B148FEE24AF5111595CC58D6F2B76C5

It's because the class-id has a slash in it, and the nft-transfer module incorrectly assumes it's an ibc class-id.

Proof: if you make a hash from the class-id in the tx above: ark/xxfs you get: 57316878F682A6541C5587C0A07BC345492C9FB0738A1619D1FF1649DE814F59, which is the class-id that the nft-transfer module does not recognize: timeout packet callback failed: ibc/57316878F682A6541C5587C0A07BC345492C9FB0738A1619D1FF1649DE814F59: nft class does not exist

heejin-github · 2023-03-13T07:13:04Z

I think I'm not that case.

This is the transaction from IRIS to Stargaze I've sent.
but, there's no slash in class-id. class-id is dsrvFirstAwesomeCollection01
https://gon.ping.pub/iris/tx/9E9F38E7352DEC169A84CDAE20634E046C661395CDCABA654A1A309515C25B6E

which class-id incorrectly assumed by nft-transfer module did you point out?

gjermundgaraba · 2023-03-13T09:06:16Z

It looks like yours has arrived succesfully?
I found the corresponding tx on stargaze: https://gon.ping.pub/stargaze/tx/6C76DD8FD8E90F25EB3FA2EC835D08CB7E7927A7A0EE9AFF32B3B34F1B503883

heejin-github · 2023-03-13T09:25:47Z

No, it doesn't arrived. Tx was failed.
https://gon.ping.pub/iris/tx/32A2E3AE1D0B00776BE364AD7F977885B5E1B1F375E9B6BE76CF99D8E22D147E

and it remained as a pending packets.
I'm trying to return it back to stargaze with hermes. but, it couldn't cleared these pending packets.

How did you get the NFT back with hermes?

# hermes query packet pending --chain gon-irishub-1 --port nft-transfer --channel channel-22
    Finished release [optimized] target(s) in 0.25s
     Running `/root/git/hermes/target/release/hermes query packet pending --chain gon-irishub-1 --port nft-transfer --channel channel-22`
2023-03-08T01:50:33.074950Z  INFO ThreadId(01) using default configuration from '/root/.hermes/config.toml'
SUCCESS Summary {
    src: PendingPackets {
        unreceived_packets: [
            Collated {
                start: Sequence(
                    74,
                ),
                end: Sequence(
                    74,
                ),
            },
        ],
        unreceived_acks: [],
    },
    dst: PendingPackets {
        unreceived_packets: [
            Collated {
                start: Sequence(
                    75,
                ),
                end: Sequence(
                    75,
                ),
            },
        ],
        unreceived_acks: [],
    },
}

error on hermes when trying to clear packets.

ERROR link error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: timeout packet callback failed: ibc/57316878F682A6541C5587C0A07BC345492C9FB0738A1619D1FF1649DE814F59: nft class does not exist

gjermundgaraba · 2023-03-13T10:04:28Z

That failure you have at the bottom with ibc/57316878F682A6541C5587C0A07BC345492C9FB0738A1619D1FF1649DE814F59 is unrelated to yours. That is the one with slash that I talked about earlier.

And the transaction you are linking, https://gon.ping.pub/iris/tx/32A2E3AE1D0B00776BE364AD7F977885B5E1B1F375E9B6BE76CF99D8E22D147E, is not related to yours either. If you look at the data sent it's, by chance, actually one of my NFTs:

{"classId":"wasm.stars1ve46fjrhcrum94c7d8yc2wsdz8cpuw73503e8qn9r44spr6dw0lsvmvtqh/channel-207/bugtest01","classUri":"https://indigo-wooden-hamster-81.mypinata.cloud/ipfs/QmZTeRokdPL8YMXTJuQNzBdCLqGNVTai3FGhqTUbJGbLNA","classData":"eyJjb21tdW5pdHkiOiJodHRwczovL2Rpc2NvcmQuZ2cvVVR4RXpGekhWWCIsImRpc2NvcmRfaGFuZGxlIjoiZ2plcm11bmQjMTU4NiIsImdpdGh1Yl91c2VybmFtZSI6ImdqZXJtdW5kZ2FyYWJhIiwiaXJpc21vZDpjcmVhdG9yIjp7InZhbHVlIjoiMjA0OTNjY2JhOTkzYjVjNTYzOTgwMDRhNDk0ZmI3MzQ5NzE2NWJmYSJ9LCJpcmlzbW9kOmRlc2NyaXB0aW9uIjp7InZhbHVlIjoiIn0sImlyaXNtb2Q6bWludF9yZXN0cmljdGVkIjp7InZhbHVlIjp0cnVlfSwiaXJpc21vZDpuYW1lIjp7InZhbHVlIjoiRW1wb3dlckNoYWluIFBsYXN0aWMgQ3JlZGl0IE5GVHMgY2xhc3MifSwiaXJpc21vZDpzY2hlbWEiOnsidmFsdWUiOiIifSwiaXJpc21vZDpzeW1ib2wiOnsidmFsdWUiOiJwY3JkbmZ0In0sImlyaXNtb2Q6dXBkYXRlX3Jlc3RyaWN0ZWQiOnsidmFsdWUiOnRydWV9LCJpcmlzbW9kOnVyaV9oYXNoIjp7InZhbHVlIjoiIn0sInRlYW1fbmFtZSI6IkVtcG93ZXJDaGFpbiJ9","tokenIds":["bg69"],"tokenUris":null,"tokenData":["eyJpcmlzbW9kOm5hbWUiOnsidmFsdWUiOiIifSwiaXJpc21vZDp1cmlfaGFzaCI6eyJ2YWx1ZSI6IiJ9fQ=="],"sender":"stars1ypynejafjw6u2cucqp9yjnahxjt3vkl66wa94m","receiver":"iaa1ypynejafjw6u2cucqp9yjnahxjt3vkl6ms2fum","memo":null}

Back to your NFT.

I checked the NFT, and it is there, on Stargaze, owned by you:

$ starsd q wasm contract-state smart stars1ve46fjrhcrum94c7d8yc2wsdz8cpuw73503e8qn9r44spr6dw0lsvmvtqh '{"nft_contract": {"class_id": "wasm.stars1ve46fjrhcrum94c7d8yc2wsdz8cpuw73503e8qn9r44spr6dw0lsvmvtqh/channel-207/dsrvFirstAwesomeCollection01"}}' $STARGAZE_Q_DEFAULT
data: stars1r9whurc74jqgdf7ftlfdxacngxv4z2u2ea6c9ph85rvp5qjsek4szdk7gy

$ starsd q wasm contract-state smart stars1r9whurc74jqgdf7ftlfdxacngxv4z2u2ea6c9ph85rvp5qjsek4szdk7gy '{"tokens": {"owner": "stars1gtk6c7ft3mpp7cmsd39mw0ysqa6mf579f7djy5"}}' $STARGAZE_Q_DEFAULT
data:
  tokens:
  - dsrvNFTtokenA19

heejin-github · 2023-03-13T14:50:34Z

Got it! maybe someone cleared my pended packets in iris. then returned my NFT back.
Thanks for your kindly help.

gjermundgaraba mentioned this issue Mar 9, 2023

BUG: Go relayer not picking up all timeouts #365

Open

gjermundgaraba mentioned this issue Mar 10, 2023

AWARD: EmpowerChain Claim 1: issue in nft-transfer might be possible to use to attack relaying capabilities #397

Closed

taramakage added the relayer Something wrong with the realyer label Mar 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: NFT transfer from Stargaze to IRISnet pended and didn't returned #329

BUG: NFT transfer from Stargaze to IRISnet pended and didn't returned #329

heejin-github commented Mar 8, 2023

gjermundgaraba commented Mar 9, 2023

0xekez commented Mar 9, 2023

0xekez commented Mar 9, 2023

gjermundgaraba commented Mar 9, 2023

0xekez commented Mar 9, 2023

gjermundgaraba commented Mar 9, 2023

gjermundgaraba commented Mar 9, 2023

0xekez commented Mar 9, 2023

heejin-github commented Mar 9, 2023

gjermundgaraba commented Mar 10, 2023 •

edited

Loading

gjermundgaraba commented Mar 10, 2023

heejin-github commented Mar 13, 2023

gjermundgaraba commented Mar 13, 2023

heejin-github commented Mar 13, 2023 •

edited

Loading

gjermundgaraba commented Mar 13, 2023 •

edited

Loading

heejin-github commented Mar 13, 2023

BUG: NFT transfer from Stargaze to IRISnet pended and didn't returned #329

BUG: NFT transfer from Stargaze to IRISnet pended and didn't returned #329

Comments

heejin-github commented Mar 8, 2023

Summary of Bug

Environment

Steps to Reproduce

Expected and Actual Behavior

Error Stack

Additional Context

gjermundgaraba commented Mar 9, 2023

0xekez commented Mar 9, 2023

0xekez commented Mar 9, 2023

gjermundgaraba commented Mar 9, 2023

0xekez commented Mar 9, 2023

gjermundgaraba commented Mar 9, 2023

gjermundgaraba commented Mar 9, 2023

0xekez commented Mar 9, 2023

heejin-github commented Mar 9, 2023

gjermundgaraba commented Mar 10, 2023 • edited Loading

gjermundgaraba commented Mar 10, 2023

heejin-github commented Mar 13, 2023

gjermundgaraba commented Mar 13, 2023

heejin-github commented Mar 13, 2023 • edited Loading

gjermundgaraba commented Mar 13, 2023 • edited Loading

heejin-github commented Mar 13, 2023

gjermundgaraba commented Mar 10, 2023 •

edited

Loading

heejin-github commented Mar 13, 2023 •

edited

Loading

gjermundgaraba commented Mar 13, 2023 •

edited

Loading