Penunumbra integration

.pre-commit-config.yaml

@@ -2,9 +2,6 @@ repos:
 -   repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v2.3.0
     hooks:
-    - id: flake8
-      types: ['file']
-      files: \.(py|pyx|pxd)$
     - id: detect-private-key
 -   repo: https://github.com/pre-commit/mirrors-eslint
     rev: v8.10.0

README_Penumbra_Osiris.md

@@ -0,0 +1,88 @@
+## Penumbra Osiris
+The [penumbra osiris bot](https://github.com/Se7en-Seas/hummingbot/blob/penunumbra-integration/scripts/penumbra_osiris.py) aims to be an approximate parity to its [rust counterpart](https://github.com/penumbra-zone/osiris/) and function as an abstraction to allow new users to interact with the penumbra protocol without needing to understand the underlying mechanics.
+
+## First time setup
+1. Clone this [repo and checkout this branch](https://github.com/Se7en-Seas/hummingbot/tree/penunumbra-integration/)
+```bash
+git clone https://github.com/Se7en-Seas/hummingbot/tree/penunumbra-integration/
+cd hummingbot
+git fetch --all
+git checkout penunumbra-integration
+```
+2. Set up pclientd and have it running with any releveant test funds. You'll know you're good to go when you can run the following command and get a response
+```bash
+grpcurl -plaintext -d '{"account_filter": {"account": 0}}'     127.0.0.1:8081 penumbra.view.v1.ViewService/Balances
+```
+3. Install Anaconda w/ Python 3.7+ along with relevant dependencies, this is a requirement to compile the hummingbot repo
+```bash
+apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6
+wget https://repo.anaconda.com/archive/Anaconda3-2023.09-0-Linux-x86_64.sh
+bash Anaconda3-2023.09-0-Linux-x86_64.sh # Accept the lisence agreement and install
+# install miniconda
+mkdir -p ~/miniconda3
+wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
+bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
+rm -rf ~/miniconda3/miniconda.sh
+~/miniconda3/bin/conda init bash
+```
+
+3.5 Set up conda for your terminal, here is a minimal bash example
+```bash
+export PATH=/root/anaconda3/bin:$PATH
+source ~/.bashrc
+conda init bash
+conda activate hummingbot
+```
+
+4. Compile hummingbot (this will take a while) & activate the conda environment (if you haven't already)
+```bash
+make install
+./compile # Assuming you're still in the base hummingbot repo directory
+```
+
+5. Start hummingbot and run the penumbra osiris bot :3
+```bash 
+./start # If it's your first time running hummingbot, you'll need to follow the on screen instructions to set up your configurations
+# Now that you're in the terminal, you'll need to set up the penumbra connector by specifying the pclientd url (typically localhost:8081) and the gateway url (typically localhost:15888 note the gateway is currently not used and so the URL is a formality but not relevant to the osiris script)
+connect penumbra # Note this is only necessary for the first time setup
+start --script penumbra_osiris.py # This will start the penumbra osiris bot
+# Note your terminal may appear to hang, this is normal as the bot is submitting orders in the background and cannot render logging at the same time
+# To see your account's status
+status
+# If you wish to stop the bot, just type in stop, or exit to exit the hummingbot terminal
+```
+
+
+Notes:
+
+To check the logs you can look at the logs directory
+```bash
+cat ./logs/logs_penumbra_osiris.log 
+```
+
+If you wish to add more logging in the log file, you must do:
+```python
+logging.getLogger().info("<TEXT_HERE>")
+```
+however if you wish to print to the terminal, you must do:
+```python
+print("<TEXT_HERE>")
+```
+
+
+## How to refresh the Penumbra Protos
+Note the last proto update was on version 0.75.0
+1. Grab all of the relevant [protos](https://github.com/penumbra-zone/penumbra/tree/main/proto)
+2. Move everything out of rust-vendored to top level dir so everything is in proto/penumbra and appears as [so](https://github.com/Se7en-Seas/hummingbot/tree/penunumbra-integration/hummingbot/connector/gateway/clob_spot/data_sources/penumbra/proto/penumbra)
+3. Wipe the [generated directory](https://github.com/Se7en-Seas/hummingbot/tree/penunumbra-integration/hummingbot/connector/gateway/clob_spot/data_sources/penumbra)
+4. Run the following command to generate the protos **from the base repo directory**
+```bash 
+pip install grpcio-tools
+
+find <ABSOLUTE_PATH_TO_REPO>/hummingbot/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/penumbra -name "*.proto" | xargs python3.10 -m grpc_tools.protoc  \
+    --python_out=<ABSOLUTE_PATH_TO_REPO>/hummingbot/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/penumbra/generated \
+    --grpc_python_out=<ABSOLUTE_PATH_TO_REPO>/hummingbot/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/penumbra/generated \
+    -I<ABSOLUTE_PATH_TO_REPO>/hummingbot/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/penumbra/proto/penumbra
+```
+5. Fix all of the import paths to be absolute. Namely, fix all instances of `from penumbra.` imports to be prefixed with `from hummingbot.connector.gateway.clob_spot.data_sources.penumbra.generated.`
+6. Update all files that use `grpc.experimental` to contain the import `import grpc.experimental` 

hummingbot/client/command/connect_command.py

@@ -51,7 +51,7 @@ async def connect_exchange(self,  # type: HummingbotApplication
                     f"Would you like to replace your existing {connector_name} API key {api_key} (Yes/No)? >>> "
                 )
             else:
-                prompt = f"Would you like to replace your existing {connector_name} key (Yes/No)? >>> "
+                prompt = f"Would you like to replace your existing {connector_name} config (Yes/No)? >>> "
             answer = await self.app.prompt(prompt=prompt)
             if self.app.to_stop_config:
                 self.app.to_stop_config = False

hummingbot/client/settings.py

@@ -613,3 +613,7 @@ def gateway_connector_trading_pairs(connector: str) -> List[str]:
 
 STRATEGIES: List[str] = get_strategy_list()
 GATEWAY_CONNECTORS: List[str] = []
+
+
+if __name__ == "__main__":
+    print(AllConnectorSettings.create_connector_settings())

hummingbot/connector/derivative/binance_perpetual/binance_perpetual_constants.py

@@ -8,11 +8,11 @@
 DOMAIN = EXCHANGE_NAME
 TESTNET_DOMAIN = "binance_perpetual_testnet"
 
-PERPETUAL_BASE_URL = "https://fapi.binance.com/fapi/"
-TESTNET_BASE_URL = "https://testnet.binancefuture.com/fapi/"
+PERPETUAL_BASE_URL = "https://fapi.binance.us/fapi/"
+TESTNET_BASE_URL = "https://testnet.binancefuture.us/fapi/"
 
-PERPETUAL_WS_URL = "wss://fstream.binance.com/"
-TESTNET_WS_URL = "wss://stream.binancefuture.com/"
+PERPETUAL_WS_URL = "wss://fstream.binance.us/"
+TESTNET_WS_URL = "wss://stream.binancefuture.us/"
 
 PUBLIC_WS_ENDPOINT = "stream"
 PRIVATE_WS_ENDPOINT = "ws"

hummingbot/connector/exchange/binance/binance_constants.py

@@ -1,7 +1,7 @@
 from hummingbot.core.api_throttler.data_types import LinkedLimitWeightPair, RateLimit
 from hummingbot.core.data_type.in_flight_order import OrderState
 
-DEFAULT_DOMAIN = "com"
+DEFAULT_DOMAIN = "us"
 
 HBOT_ORDER_ID_PREFIX = "x-XEKWYICX"
 MAX_ORDER_ID_LEN = 32

hummingbot/connector/exchange/penumbra/penumbra_api_order_book_data_source.py

@@ -0,0 +1,89 @@
+import asyncio
+from collections import defaultdict
+from typing import TYPE_CHECKING, Any, Dict, List, Optional
+
+from hummingbot.connector.exchange.penumbra import penumbra_utils as utils
+from hummingbot.core.api_throttler.async_throttler import AsyncThrottler
+from hummingbot.core.data_type.order_book_message import OrderBookMessage
+from hummingbot.core.data_type.order_book_tracker_data_source import OrderBookTrackerDataSource
+from hummingbot.core.web_assistant.web_assistants_factory import WebAssistantsFactory
+from hummingbot.core.web_assistant.ws_assistant import WSAssistant
+
+if TYPE_CHECKING:
+    from hummingbot.connector.exchange.penumbra.penumbra_exchange import PenumbraExchange
+
+
+class PenumbraAPIOrderBookDataSource(OrderBookTrackerDataSource):
+    def __init__(
+        self,
+        trading_pairs: List[str],
+        connector: "PenumbraExchange",
+        api_factory: Optional[WebAssistantsFactory] = None,
+        domain: str = 'localhost:8081',
+        throttler: Optional[AsyncThrottler] = None,
+    ):
+        super().__init__(trading_pairs)
+        self._connector = connector
+        self._domain = domain
+        self._throttler = throttler
+        self._api_factory = api_factory or utils.build_api_factory(
+            throttler=self._throttler,
+        )
+        self._message_queue: Dict[str, asyncio.Queue] = defaultdict(asyncio.Queue)
+        self._last_ws_message_sent_timestamp = 0
+        self._ping_interval = 0
+
+    # TODO: Need to actually implement
+
+    async def get_last_traded_prices(self, trading_pairs: List[str], domain: Optional[str] = None) -> Dict[str, float]:
+        print("get_last_traded_prices")
+        raise NotImplementedError
+
+    async def _order_book_snapshot(self, trading_pair: str) -> OrderBookMessage:
+        print("_order_book_snapshot")
+        raise NotImplementedError
+
+    async def _request_order_book_snapshot(self, trading_pair: str) -> Dict[str, Any]:
+        """
+        Retrieves a copy of the full order book from the exchange, for a particular trading pair.
+
+        :param trading_pair: the trading pair for which the order book will be retrieved
+
+        :return: the response from the exchange (JSON dictionary)
+        """
+        print("_request_order_book_snapshot")
+        raise NotImplementedError
+
+    async def _parse_trade_message(self, raw_message: Dict[str, Any], message_queue: asyncio.Queue):
+        print("_parse_trade_message")
+        raise NotImplementedError
+
+    async def _parse_order_book_diff_message(self, raw_message: Dict[str, Any], message_queue: asyncio.Queue):
+        print("_parse_order_book_diff_message")
+        raise NotImplementedError
+
+    async def _subscribe_channels(self, websocket_assistant: WSAssistant):
+        """
+        Subscribes to the trade events and diff orders events through the provided websocket connection.
+
+        :param websocket_assistant: the websocket assistant used to connect to the exchange
+        """
+        print("_subscribe_channels")
+
+        raise NotImplementedError
+
+    def _channel_originating_message(self, event_message: Dict[str, Any]) -> str:
+        print("_channel_originating_message")
+        raise NotImplementedError
+
+    async def _process_websocket_messages(self, websocket_assistant: WSAssistant):
+        """
+        Connects to the trade events and order diffs websocket endpoints and listens to the messages sent by the
+        exchange. Each message is stored in its own queue.
+        """
+        print("_process_websocket_messages")
+        raise NotImplementedError
+
+    async def _connected_websocket_assistant(self) -> WSAssistant:
+        print("_connected_websocket_assistant")
+        raise NotImplementedError

hummingbot/connector/exchange/penumbra/penumbra_api_user_stream_data_source.py

@@ -0,0 +1,53 @@
+import asyncio
+from typing import TYPE_CHECKING, Any, Dict, List, Optional
+
+from hummingbot.connector.exchange.penumbra import penumbra_utils as utils
+from hummingbot.core.api_throttler.async_throttler import AsyncThrottler
+from hummingbot.core.data_type.user_stream_tracker_data_source import UserStreamTrackerDataSource
+from hummingbot.core.web_assistant.auth import AuthBase
+from hummingbot.core.web_assistant.web_assistants_factory import WebAssistantsFactory
+from hummingbot.core.web_assistant.ws_assistant import WSAssistant
+
+if TYPE_CHECKING:
+    from hummingbot.connector.exchange.penumbra.penumbra_exchange import PenumbraExchange
+
+
+class PenumbraAPIUserStreamDataSource(UserStreamTrackerDataSource):
+    def __init__(
+        self,
+        auth: AuthBase,
+        trading_pairs: List[str],
+        connector: "PenumbraExchange",
+        domain: str = 'localhost:8081',
+        api_factory: Optional[WebAssistantsFactory] = None,
+        throttler: Optional[AsyncThrottler] = None,
+    ):
+        super().__init__()
+        self._connector = connector
+        self._auth: AuthBase = auth
+        self._trading_pairs = trading_pairs
+        self._last_recv_time: float = 0
+        self._domain = domain
+        self._throttler = throttler
+        self._api_factory = api_factory or utils.build_api_factory(throttler=self._throttler, auth=self._auth)
+        self._ping_interval = 0
+        self._last_ws_message_sent_timestamp = 0
+
+    # TODO: These are all stubs, unnecessary for Avellenda Strategy, implement as warranted
+
+    async def _connected_websocket_assistant(self) -> WSAssistant:
+        return
+
+    async def _subscribe_channels(self, websocket_assistant: WSAssistant):
+        """
+        Subscribes to the trade events and diff orders events through the provided websocket connection.
+
+        :param websocket_assistant: the websocket assistant used to connect to the exchange
+        """
+        return
+
+    async def _process_websocket_messages(self, websocket_assistant: WSAssistant, queue: asyncio.Queue):
+        return
+
+    async def _process_event_message(self, event_message: Dict[str, Any], queue: asyncio.Queue):
+        return

hummingbot/connector/exchange/penumbra/penumbra_constants.py

@@ -0,0 +1,11 @@
+import sys
+
+from hummingbot.core.api_throttler.data_types import RateLimit
+
+EXCHANGE_NAME = "penumbra"
+
+NO_LIMIT = sys.maxsize
+
+RATE_LIMITS = [
+    RateLimit(limit_id='/*', limit=NO_LIMIT, time_interval=1),
+]