Skip to content
This repository has been archived by the owner on Sep 9, 2023. It is now read-only.

Refactor Research

Jump to bottom
Brian Mancini edited this page Sep 11, 2020 · 17 revisions

Document per-exchange requirements

  • authentication required
  • single socket / multi-socket / per-socket limits
  • subscription limits

The goal of this research will be to determine how we can use a more generic base client and we can use composition to enable the required capabilities for an exchange.

  • authenticate
  • getSocket
  • sendMessage
  • message handling
  • composite messages
  • restRequests
  • watcher (include ping/pong and heartbeat messages, allow better tune-ability)

Some features:

  • emit raw message (possibly as third parameter) -> this will be useful for debugging
  • emit unhandled message for debuugging -> this will be useful for seeing if anything is wrong

Poloniex

  • performs REST request on client initialization for market identifiers.
  • treats some markets like rooms, subscribing to ticker, trades, and orderbook in a single call to the market feed

JEX

  • would benefit from event based pattern matching for subscribe as well as handling messages. It requires subscribing to spotTrade, optionTrade, or contractTrade. These emit similarly named events

Bittrex

  • uses SignalR, requires an asynchronous connection
  • batched tickers, subscribe to all markets
  • supports heartbeat

CEX

  • Single socket per market
  • Instantiates BasicClient

Coinbase

  • heartbeat is per market

Coinex

  • Single Socket per subscription
  • Instantiates BasicClient

Gateio

  • uses debounces all subscriptions -> TODO should use the debounce helper though
  • requires sending all subscribed markets to the subscription

Gemini

  • does not use BasicClient
  • room based implementation
  • has different URLs for tickers and other feeds
  • constructs a subscription object (thanks Cam!) for each market that gets subscribed to

HitBTC

  • uses send throttling of 25ms

Huobi

  • has ping
  • otherwise super straight forward client

Kraken

  • requires loading a symbol map at start
  • uses send debouncing and batched to all active symbols -> TODO refactor to use flow control helpers

Kucoin

  • limit of 100 subscriptions per socket
  • asynchronous connection, REST provides a socket token
  • socket is "ready" after welcome message has been received
  • initiate L3 REST snapshot AFTER successful l3 update subscription to prevent race-condition, see #207

Liquid

  • ping
  • loads symbol maps at client start

OKEx

  • uses ping
  • throttling
  • limit of 1 socket per second
  • limit of 240 subscriptions per hour (this is a soft limit, looks like you can subscribe to ~400 without it failing)

Poloniex

  • loads symbol maps at start
  • uses room subscriptions for symbols

Upbit

  • uses debounced and batched sends

Huobi Futures

  • Uses simplified symbols for subscription adn requires a mapping from the contract_code to the simplified code. For example, the contract BTC201225 is actually BTC_NQ as a socket subscription.
  • Upon reconnection, the maps need to be reloaded and changed...

Ping/Pong

  • Coinex, Gateio, Kucoin, Liquid, OKEx
    • Coinex sends a ping every 30s
    • Gateio sends a pint every 30s
    • Kucoin sends a ping every 50s
    • Liquid sends a ping every 60s
    • OKEx sends a ping every 15s
    • uses _beforeConnect to attached event handlers for connected, disconnected and closed event handlers for startPing and stopPing
  • Bibox, CEX,
    • Bibox replies with a pong with identifier
    • CEX replies with an empty pong
    • Huobi replies with a pong with identifier

Authentication

  • CEX requires an API key for market data
    • Overrides _onConnected and calls _onAuthorized which triggers the authorized event
    • Then calls super._onConnected() once authorized

Heartbeat

  • Bittrex
    • subscribe to heartbeat
    • sends heartbeat every 5s, resets watcher
  • LedgerX
    • no subscribe required
    • sends heartbeat every 5s, resets watcher

Subscription Confirmation

  • Bittrex
    • subscription request has unique identifier
    • subscription response has unique identifier, success boolean, error code for failures

General Thoughts

  • convert this thing to typescript to make the refactor easier

  • subscribe pushes subscriptions into central event subscriber for the client

  • asynchronously acquire a connection which can be customized to meet the needs of the exchange

    • limits on subscriptions per socket
    • connection throttling
    • authentication (based on feed type)

  • once a connection is provided for the subscription it can be placed into a "send" queue for the socket

  • the send queue can customize behavior based on how the exchange works

    • throttling
    • batching
    • grouping

  • event hooks??

  • event handling is similar to a router

  • subscription package should have unique identifier (for look up of failures)

    • subscription should be a state machine

  • _sendSub* method

    • receive market only (no longer just receive remote_id)
    • receive list of all markets in the sub type > this allows for easy batched sends

_ _toRemoveId, _fromRemoteId methods

  • facilitates any mapping that needs to be done between the received subscribing method and the received version.

  • this can be used in the huobi futures or poloniex exchanges

  • post subscription hooks

    • kucoin, binance order books
      • need to fire off REST request AFTER the update stream has started, we end up with a race condition in #207 if we attempt to do this in a time based manner
Clone this wiki locally