Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mass decoding non dynamic #6630

Open
wants to merge 97 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
99b0690
recover working version of mass decoding
0xBoxer Aug 9, 2024
f7f34ca
fix run errors
0xBoxer Aug 9, 2024
9ce028d
fix run error
0xBoxer Aug 9, 2024
3fa37a0
fix run error
0xBoxer Aug 9, 2024
c0337c4
add comma
0xBoxer Aug 9, 2024
0a0d53c
add missing column from upstream
0xBoxer Aug 9, 2024
23a2a4f
fix test columns
0xBoxer Aug 9, 2024
936db77
fix some logic errors and remove inner join
0xBoxer Aug 9, 2024
9c48f10
add missing column
0xBoxer Aug 9, 2024
7594d3e
add factory address
0xBoxer Aug 9, 2024
3565dd6
move join into downstream cte
0xBoxer Aug 9, 2024
47350af
fix col name
0xBoxer Aug 9, 2024
30e0010
add extra factory address to base_trades
0xBoxer Aug 9, 2024
f24a745
push down to dex.trades
0xBoxer Aug 9, 2024
0b69938
fix macro
0xBoxer Aug 9, 2024
39e1332
trigger dex.trades run
0xBoxer Aug 9, 2024
1f4f653
fix mapping
0xBoxer Aug 9, 2024
0e92b7f
build paralell dex trades
0xBoxer Aug 10, 2024
af3e3ba
change model name
0xBoxer Aug 10, 2024
5a0e4f0
fix blockchain column
0xBoxer Aug 10, 2024
31858cb
take out date filter
0xBoxer Aug 10, 2024
d68d231
Merge branch 'main' into on-the-fly-decoding-dex.trades
0xBoxer Aug 29, 2024
75a8a32
change approach to be more dynamic
0xBoxer Aug 29, 2024
fc2ebf7
add univ3
0xBoxer Aug 29, 2024
9afc3df
add to union
0xBoxer Aug 29, 2024
0a46155
fix compile error
0xBoxer Aug 29, 2024
fd1ca4a
fix columns
0xBoxer Aug 30, 2024
b3cc67e
fix lineage
0xBoxer Aug 30, 2024
504c1b8
add project column
0xBoxer Aug 30, 2024
493cb6a
fi
0xBoxer Aug 30, 2024
cb0f747
edit yml
0xBoxer Aug 30, 2024
450db87
remove factory_address from partition_by in uniswap_v2 and uniswap_v3…
0xBoxer Aug 30, 2024
d7ea43f
construct full lineage
0xBoxer Aug 30, 2024
7e301be
rename macro
0xBoxer Aug 30, 2024
7cbd330
rename enrich_dex_trades macro to log_decoded_enrich_dex_trades for c…
0xBoxer Aug 30, 2024
ae8d63f
add name to pair_index in Uniswap V2 factory mass decoding and join c…
0xBoxer Aug 30, 2024
549657a
trigger run
0xBoxer Aug 30, 2024
77f9038
Merge branch 'main' into mass-decoding-non-dynamic
0xBoxer Aug 30, 2024
30bef8e
add factory address
0xBoxer Aug 30, 2024
0245c65
Restored accidentally deleted folder
0xBoxer Aug 30, 2024
af18ba7
delete files too many
0xBoxer Aug 30, 2024
e388922
remove copies
0xBoxer Aug 30, 2024
e097619
fix dbt compile
0xBoxer Aug 30, 2024
f47df16
remove factory_address duplicate from mass_decoded_dex_trades_ethereu…
0xBoxer Aug 30, 2024
436dcb4
add final select to dex_trades
0xBoxer Aug 30, 2024
55a17cb
remove duplicate dex_trades
0xBoxer Aug 30, 2024
1ee90ad
add mode
0xBoxer Aug 30, 2024
4a326a1
add arbitrum files
0xBoxer Aug 30, 2024
e50e4e6
fixing creation traces
0xBoxer Aug 30, 2024
72d94ac
remove mode because no creation traces
0xBoxer Aug 30, 2024
43cd31b
fix
0xBoxer Aug 30, 2024
b5035a2
Merge branch 'main' into mass-decoding-non-dynamic
Hosuke Sep 3, 2024
033a9da
Merge branch 'main' into mass-decoding-non-dynamic
Hosuke Sep 10, 2024
7da7e8a
Merge branch 'main' into mass-decoding-non-dynamic
Hosuke Sep 11, 2024
63d39a6
Exclude unknown factory addresses
Hosuke Sep 11, 2024
5099ff8
Update schema
Hosuke Sep 11, 2024
2e99897
Fix typo
Hosuke Sep 12, 2024
2f1bf77
List all column
Hosuke Sep 12, 2024
ffc0679
Fix
Hosuke Sep 12, 2024
9cb0813
Update arbitrum decoding trades
Hosuke Sep 12, 2024
8bd225f
Recover dodo_ethereum_base_trades
Hosuke Sep 12, 2024
716b14d
Update uni-v3 mapping
Hosuke Sep 12, 2024
f528b41
Rename dex_automation_beta_trades
Hosuke Sep 12, 2024
9e6263e
Remove dex_info test
Hosuke Sep 12, 2024
b36b3e2
Fix unique_key
Hosuke Sep 12, 2024
04cf0c5
Fix unique_key
Hosuke Sep 12, 2024
5a3c8ff
Merge branch 'main' into mass-decoding-non-dynamic
Hosuke Sep 20, 2024
2ba354a
Remove inner join to show all uni-v2 forks
Hosuke Sep 20, 2024
5c30919
Exclude cetain factory addresses
Hosuke Sep 23, 2024
5a511d9
Update column name
Hosuke Sep 23, 2024
24cacf6
Update exlude logic
Hosuke Sep 23, 2024
439d520
Update exlude logic
Hosuke Sep 23, 2024
e8b18f4
Update exlude logic
Hosuke Sep 23, 2024
91855c8
Update base_trades
Hosuke Sep 24, 2024
b290648
Fix incremental parts
Hosuke Sep 24, 2024
cd1b9d1
Update _schema.yml
Hosuke Sep 24, 2024
760ec59
Update _schema.yml
Hosuke Sep 24, 2024
20ddaa2
Merge branch 'main' into mass-decoding-non-dynamic
Hosuke Sep 24, 2024
7c519eb
Try to fix incremental error
Hosuke Sep 24, 2024
9fbb852
Merge branch 'mass-decoding-non-dynamic' of ssh://github.com/duneanal…
Hosuke Sep 24, 2024
f59386a
Fix CTE name
Hosuke Sep 25, 2024
86a1fbd
Implement boxer's proposal
Hosuke Sep 25, 2024
6ee7fcc
Update filter condition and transfers source table
Hosuke Sep 25, 2024
974b002
Fix ref
Hosuke Sep 25, 2024
82a880a
Update source
Hosuke Sep 26, 2024
0afe9c4
Fix blockchain
Hosuke Sep 26, 2024
a873d8e
Merge branch 'main' into mass-decoding-non-dynamic
Hosuke Sep 26, 2024
d405e18
Fix source
Hosuke Sep 26, 2024
4d218d8
Update filter logics
Hosuke Sep 29, 2024
f70131e
Remove incremental predicates
Hosuke Sep 29, 2024
16b32f9
Merge branch 'main' into mass-decoding-non-dynamic
Hosuke Sep 29, 2024
f2deff7
Revert "Remove incremental predicates"
Hosuke Sep 30, 2024
9596e20
Add distinct incremental selection
Hosuke Oct 1, 2024
175c2c6
Fix incremental
Hosuke Oct 1, 2024
dff0105
Fix incremental in decode_evm_event function
Hosuke Oct 1, 2024
5ec5988
Remove distinct selection
Hosuke Oct 1, 2024
576d854
Merge branch 'main' into mass-decoding-non-dynamic
Hosuke Oct 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{% macro evm_event_decoding_base(
logs = null,
abi = null,
topic0 = null
)
%}

SELECT
*

FROM TABLE (
decode_evm_event (
abi => '{{abi}}',
input => TABLE (
SELECT l.*
FROM {{logs}} l
WHERE topic0 = {{topic0}}
{% if is_incremental() %}
AND {{ incremental_predicate('block_time') }}
{% else %}
AND block_date >= (SELECT MIN(block_date) FROM {{logs}} WHERE topic0 = {{topic0}})
{% endif %}
)
)
)
jeff-dude marked this conversation as resolved.
Show resolved Hide resolved

{% endmacro %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{% macro uniswap_v2_factory_mass_decoding(logs) %}

{% set abi = '{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "address",
"name": "token0",
"type": "address"
},
{
"indexed": true,
"internalType": "address",
"name": "token1",
"type": "address"
},
{
"indexed": false,
"internalType": "address",
"name": "pair",
"type": "address"
},
{
"indexed": false,
"internalType": "uint256",
"name": "pair_index",
"type": "uint256"
}
],
"name": "PairCreated",
"type": "event"
}' %}

{% set topic0 = '0x0d3648bd0f6ba80134a33ba9275ac585d9d315f0ad8355cddefde31afa28d0e9' %}

{{ evm_event_decoding_base(logs, abi, topic0) }}

{% endmacro %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{% macro uniswap_v2_pool_mass_decoding(logs) %}

{% set abi = '{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "address",
"name": "sender",
"type": "address"
},
{
"indexed": false,
"internalType": "uint256",
"name": "amount0In",
"type": "uint256"
},
{
"indexed": false,
"internalType": "uint256",
"name": "amount1In",
"type": "uint256"
},
{
"indexed": false,
"internalType": "uint256",
"name": "amount0Out",
"type": "uint256"
},
{
"indexed": false,
"internalType": "uint256",
"name": "amount1Out",
"type": "uint256"
},
{
"indexed": true,
"internalType": "address",
"name": "to",
"type": "address"
}
],
"name": "Swap",
"type": "event"
}' %}

{% set topic0 = '0xd78ad95fa46c994b6551d0da85fc275fe613ce37657fb8d5e3d130840159d822' %}

{{ evm_event_decoding_base(logs, abi, topic0) }}

{% endmacro %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
{% macro uniswap_v2_forks_trades(
blockchain = null
, dex_type = 'uni-v2'
, project = null
, version = null
, Pair_evt_Swap = null
, Factory_evt_PairCreated = null
, pair_column_name = 'pair'
)
%}

WITH evt_swap AS (
SELECT
block_number
, block_time
, to
, contract_address
, tx_hash
, index
, amount0In
, amount0Out
, amount1In
, amount1Out
FROM {{ Pair_evt_Swap }}
{% if is_incremental() %}
WHERE {{ incremental_predicate('block_time') }}
{% endif %}
jeff-dude marked this conversation as resolved.
Show resolved Hide resolved
)

, dexs AS
(
SELECT
t.block_number
, t.block_time
, t.to AS taker
, t.contract_address AS maker
, CASE WHEN amount0Out = UINT256 '0' THEN amount1Out ELSE amount0Out END AS token_bought_amount_raw
, CASE WHEN amount0In = UINT256 '0' OR amount1Out = UINT256 '0' THEN amount1In ELSE amount0In END AS token_sold_amount_raw
, CASE WHEN amount0Out = UINT256 '0' THEN f.token1 ELSE f.token0 END AS token_bought_address
, CASE WHEN amount0In = UINT256 '0' OR amount1Out = UINT256 '0' THEN f.token1 ELSE f.token0 END AS token_sold_address
, t.contract_address AS project_contract_address
, t.tx_hash
, t.index AS evt_index
, f.contract_address as factory_address
FROM
evt_swap t
INNER JOIN
{{ Factory_evt_PairCreated }} f
ON f.{{ pair_column_name }} = t.contract_address
--some scammers emitted events with established pair addresses, joining in the the creation trace to resolve correct factory
INNER JOIN {{ source(blockchain, 'creation_traces') }} ct
ON f.{{ pair_column_name }} = ct.address
AND f.contract_address = ct."from"
)

SELECT
'{{ blockchain }}' AS blockchain
jeff-dude marked this conversation as resolved.
Show resolved Hide resolved
, '{{project}}' AS project
, '{{ version }}' AS version
, '{{dex_type}}' AS dex_type
, CAST(date_trunc('month', dexs.block_time) AS date) AS block_month
, CAST(date_trunc('day', dexs.block_time) AS date) AS block_date
, dexs.block_time
, dexs.block_number
, dexs.token_bought_amount_raw
, dexs.token_sold_amount_raw
, dexs.token_bought_address
, dexs.token_sold_address
, dexs.taker
, dexs.maker
, dexs.project_contract_address
, dexs.tx_hash
, dexs.evt_index
, dexs.factory_address
FROM
dexs

{% endmacro %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{% macro uniswap_v3_factory_mass_decoding(logs) %}

{% set abi = '
{"name":"PoolCreated","type":"event","inputs":[{"name":"token0","type":"address","indexed":true,"internalType":"address"},{"name":"token1","type":"address","indexed":true,"internalType":"address"},{"name":"fee","type":"uint24","indexed":true,"internalType":"uint24"},{"name":"tickSpacing","type":"int24","indexed":false,"internalType":"int24"},{"name":"pool","type":"address","indexed":false,"internalType":"address"}],"anonymous":false}
' %}

{% set topic0 = '0x783cca1c0412dd0d695e784568c96da2e9c22ff989357a2e8b1d9b2b4e6b7118' %}

{{ evm_event_decoding_base(logs, abi, topic0) }}

{% endmacro %}

Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{% macro uniswap_v3_pool_mass_decoding(logs) %}

{% set abi = '
{"name":"Swap","type":"event","inputs":[{"name":"sender","type":"address","indexed":true,"internalType":"address"},{"name":"recipient","type":"address","indexed":true,"internalType":"address"},{"name":"amount0","type":"int256","indexed":false,"internalType":"int256"},{"name":"amount1","type":"int256","indexed":false,"internalType":"int256"},{"name":"sqrtPriceX96","type":"uint160","indexed":false,"internalType":"uint160"},{"name":"liquidity","type":"uint128","indexed":false,"internalType":"uint128"},{"name":"tick","type":"int24","indexed":false,"internalType":"int24"}],"anonymous":false}
' %}

{% set topic0 = '0xc42079f94a6350d7e6235f29174924f928cc2ac818eb64fed8004e115fbcca67' %}

{{ evm_event_decoding_base(logs, abi, topic0) }}

{% endmacro %}

Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@

{% macro uniswap_v3_forks_trades(
blockchain = null
, dex_type = 'uni-v3'
, project = null
, version = null
, Pair_evt_Swap = null
, Factory_evt_PoolCreated = null
, pair_column_name = 'pool'
, taker_column_name = 'recipient'
, maker_column_name = null
)
%}

WITH evt_swap AS (
SELECT
block_number
, block_time
, {{ taker_column_name }}
{% if maker_column_name %}
, {{ maker_column_name }}
{% endif %}
, amount0
, amount1
, contract_address
, tx_hash
, index
FROM {{ Pair_evt_Swap }}
{% if is_incremental() %}
WHERE {{ incremental_predicate('block_time') }}
{% endif %}
)

, dexs AS
(
SELECT
t.block_number
,t.block_time
, t.{{ taker_column_name }} AS taker
, {% if maker_column_name %}
t.{{ maker_column_name }}
{% else %}
cast(null as varbinary)
{% endif %} as maker
, CASE WHEN amount0 < INT256 '0' THEN abs(amount0) ELSE abs(amount1) END AS token_bought_amount_raw -- when amount0 is negative it means trader_a is buying token0 from the pool
, CASE WHEN amount0 < INT256 '0' THEN abs(amount1) ELSE abs(amount0) END AS token_sold_amount_raw
, CASE WHEN amount0 < INT256 '0' THEN f.token0 ELSE f.token1 END AS token_bought_address
, CASE WHEN amount0 < INT256 '0' THEN f.token1 ELSE f.token0 END AS token_sold_address
, t.contract_address as project_contract_address
, t.tx_hash
, t.index as evt_index
, f.contract_address as factory_address
FROM
evt_swap t
INNER JOIN
{{ Factory_evt_PoolCreated }} f
ON f.{{ pair_column_name }} = t.contract_address
--some scammers emitted events with established pair addresses, joining in the the creation trace to resolve to correct factory
INNER JOIN {{ source(blockchain, 'creation_traces') }} ct
ON f.{{ pair_column_name }} = ct.address
AND f.contract_address = ct."from"
)

SELECT
'{{ blockchain }}' AS blockchain
, '{{ project }}' AS project
, '{{ version }}' AS version
, '{{dex_type}}' AS dex_type
, CAST(date_trunc('month', dexs.block_time) AS date) AS block_month
, CAST(date_trunc('day', dexs.block_time) AS date) AS block_date
, dexs.block_time
, dexs.block_number
, CAST(dexs.token_bought_amount_raw AS UINT256) AS token_bought_amount_raw
, CAST(dexs.token_sold_amount_raw AS UINT256) AS token_sold_amount_raw
, dexs.token_bought_address
, dexs.token_sold_address
, dexs.taker
, dexs.maker
, dexs.project_contract_address
, dexs.tx_hash
, dexs.evt_index
, dexs.factory_address
FROM
dexs
{% endmacro %}
Loading
Loading