Based on the proposal for a standalone “Ledger Adapter” in the Settlement Architecture thread, I’m going to try to outline some of the options we need to decide on for that component.
At a high level, the goal is to have a standalone component that abstracts away the differences between ledgers / settlement systems and that we can reuse across connector implementations.
For payment channel-based settlement systems (or anything that is more stateful) we may need additional APIs for opening / closing / depositing / withdrawing from settlement system accounts, although those may be hard or impossible to standardize across ledgers.
API Components
Each of these are necessary pieces of the ledger adapter API that we need to agree upon:
API Transport and Encoding
What type of API does the ledger adapter expose and expect from the connector or settlement engine?
HTTP + JSON
gRPC
HTTP + OER
Flexible Transport + ILP Packets
Flexible Transport + OER
Flexible Transport and Encoding
Unix socket + OER?
ZeroMQ + OER?
Authentication
How should the connector or settlement engine authenticate with the ledger adapter and vice versa?
Static token
JWT
TLS certificates
No auth (private port for API)
Idempotency
Should some or all of the ledger adapter’s API endpoints/methods be idempotent (meaning that if they are called twice with the same parameters they will not trigger a duplicate settlement)?
No API calls are idempotent (ledger adapter may handle retries)
Referencing accounts
How should the API refer to the different accounts the ledger adapter can send/receive to/from?
Settlement system addresses encoded as strings
Connector / settlement engine-defined identifiers (the ledger adapter would store the mapping from that to the settlement system addresses)
Flexible
Nice-to-have recommendations
The following are things that we do not necessarily need standardization on but that may be useful to try to agree upon so that the way we set up and run different settlement engines is more consistent:
Database
How should the ledger adapter store state?
PostgreSQL
SQLite
LevelDB / RocksDB
Redis
Doesn’t matter as long as it lives in a single file or directory
Anything goes
The ledger adapter shouldn’t need state
Also, should the storage be encrypted?
Packaging
How do you download and run the ledger adapter?
Docker container
Run it directly, dockerize it if you want
Configuration
How should the operator configure the settlement engine?
Environment variables
Command line arguments
API calls (that are saved to the database)
Next Steps
What do you think? Are there other high level categories we need to agree upon or should we get into the pros/cons of the different options?
I’d be down with HTTP + JSON initially. It’s by far the most universal (RIP gRPC).
We can always add other transports in the future (in the same way that LND exposes a REST API and a gRPC server, or Ethereum nodes support HTTP, websockets and IPC).
I propose connector-defined identifiers linked to the accounting relationship, since those are initialized in the connector via BTP or the open signup API.
If necessary, the settlement engine should be responsible for linking that identifier to a particular on-ledger identity through a handshake with the peer’s settlement engine (that’s how it’s currently implemented, and it works well). That way, there’s no settlement system -specific logic in the connector.
Also, settlement system addresses would limit the account space to 1:1 with ledger addresses, but some settlement engines but may want to support multiple accounts for a particular on-ledger address.
Far from an expert on this, but a few comments:
A simple key-value abstraction provides the simplest migration path, both for porting the existing plugins over, and for connector operators that need to migrate their DB (while still allowing multiple DBs to be supported, since JS settlement engines can use the existing store abstraction)
I would lean towards a high-performance, in-memory DB (e.g. Redis, which we’re already using)
Both?
Also, for settlement engines written in JS, I think they should be able to run as a module in JS, since that allows them to be used in a web context without Node APIs.
API calls would be nice, since some configuration it might be useful to change dynamically.
I have been experimenting with ZeroMQ + OER using PUB/SUB sockets, and while it offers a lot of flexibility, I think HTTP is likely the path of least resistance. In the long-term, it might make sense to transition towards Flexible Transport + OER, but HTTP + JSON is likely the lowest common denominator.
I would prefer using environment variables, but it might make sense to use API calls for permissionless ledgers since economic security changes over time.
Perhaps this isn’t much of an issue, but how will confirmation times be handled? For example, how should block confirmations be handled for a base-layer transaction?
We spent some time debating this at Coil and we propose the following API:
(We assume that the asset code and asset scale are implicit per ledger adaptor)
Settlement Engine -> Ledger Adaptor:
createAccount(accountId, counterpartyDetails)
increaseLiquidity(accountId, amount)
getLiquidity(accountId)
decreaseLiquidity(accountId, amount)
sendSettlement(accountId, amount)
sendMessage(accountId, message)
incomingMessage(accountId, message)
Ledger Adaptor -> Settlement Engine:
sendMessage(accountId, message)
incomingMessage(accountId, message)
liquidityIncrease(accountId, amount)
liquidityDecrease(accountId, amount)
The goal here is settlement system abstraction but also flexibility to support a variety of settlement systems. Also, that the adaptor doesn’t have any business logic related to liquidity management, i.e. the Settlement Engine decides when it needs more liquidity in the settlement account because it knows when it anticipates making a settlement and for how much etc.
Examples:
Below is an illustrative example of what might happen under the hood with different adaptors:
Create Account
The connector provides an account id (probably the same one they use internally for tracking balances) and some settlement system specific counter-party details.
The adaptor may create an actual account on the settlement system or simply start tracking a new account internally. Note that many adaptors will be using a pooled account so they’ll maintain a local ledger that tracks what portion of that pool is ring-fenced for settlement with each counter-party. This is needed by the Settlement Engine to do appropriate liquidity management.
The adaptor will also likely do some validation of the counter-party details.
XRP Paychan: The adaptor creates an outgoing payment channel for a default size or simply records in its own DB the need to track an account using this ID and the counter-party details.
XRP on Ledger: The adaptor makes an entry in its own DB to track this new account and links to the counter-party details. It validates that the counter-party XRP ledger address exists and begins watching for incoming payments from that address.
Increase Liquidity
XRP Paychan: If the adaptor hasn’t already it creates an outgoing payment channel of the specified size or it adds funding to the existing channel.
XRP on Ledger: The adaptor may move money from some cold wallet into a warm wallet or simply allocate a portion of some pooled funds to this account.
It’s entirely acceptable for an adaptor to respond to this request indicating that it failed in the case where getting more liquidity for settlement requires manual intervention. This tells the settlement engine to log an error or do something that would notify the operator to top up the wallet.
Get Liquidity
XRP Paychan: Check how much liquidity exists in the channel (less claims sent to the counter-party)
XRP on Ledger: Check how much liquidity has been ring-fenced for this counter-party
Decrease Liquidity
(Normally a safety measure or required to rebalance between accounts)
XRP Paychan: Close the channel (and possibly open a new smaller one if this is not a decrease to zero)
XRP on Ledger: Reduce the portion of the pool that is ring-fenced for this counterparty and possibly move money into a cold-wallet.
Send Settlement
XRP Paychan: Create a claim and send it to the settlement engine using the send message API. Wait for the response and then respond to original send settlement call.
In this proposal, the ledger adapter has an API call for sendMessage and the settlement engine has one for incomingMessage. Is the intention to make the ledger adapter responsible for communication with the other party’s ledger adapter / settlement engine?
Why would money be “ring-fenced” in the on-ledger settlement case?
Would the settlement engine be configured with details like how long the various operations on the ledger adapter are expected to take (especially the increase/decrease liquidity) or how much each of them cost?
For a payment channel based system, would you want/need a way to check the wallet balance (distinct from checking the balance of a particular channel)?
What do you think about @kincaid’s proposal for having the ledger adapters do a handshake between themselves (the messages are proxied through other components) to exchange the settlement-related details like addresses? I initially didn’t like it but after thinking it through found it interesting because it would remove the need to have the settlement engine keep track of the different parameters that the account on each type of ledger adapter needs to be configured with.
No. sendMessage on the ledger adaptor is a message from the counter-party sent via the SE.
E.g. Alice and Bob using XRP PayChan.
Alice’s SE determines it’s time to send a settlement to Bob.
Alice SE -> Alice LA: sendSettlement(Bob)
Alice LA: Sign claim
Alice LA -> Alice SE: sendMessage(Bob, Claim)
Alice SE -> Alice Connector: sendMessage(Bob, Claim)
Alice Connector -> Bob Connector: ILP Prepare (peer.settle.*, Claim)
Bob Connector -> Bob SE: sendMessage(Alice, Claim)
Bob SE -> Bob LA: sendMessage(Alice, Claim)
Maybe the SE -> LA call should be called receiveMessage?
It doesn’t HAVE to be but it makes it hard for the SE to do a good job of managing liquidity if the liquidity is shared between multiple accounts that may be managed by different SEs.
No, it should be able to figure that out over time. It could be configured with good defaults but that’s up to the SE implementation.
I don’t think you need it but it’s a possible future enhancement. What would the equivalent be for other settlement systems?
I like it. The flow for “create account” may be similar to the flow for “send settlement” in the paychan case (i.e. triggers some message exchanges)
I like most of this, save for the liquidity operations
Miscellaneous Comments
Does “ledger adapter” -> “settlement engine” need a “receiveSettlement” endpoint?
I think we need to support the balance service and “ledger adapter” using different asset scales, and I don’t think it’s implicit/can be universal between all instances. What that means is: the unit used for settlements may be smaller or larger than the unit used in ILP packets and for accounting.
Even in the current plugins, there are multiple examples of this:
Strata/Coil use nano XRP (-9) units for payments, but the plugins themselves send claims in drops, because that’s the smallest denomination on ledger
In the Ethereum plugin, ILP packets / accounting use units of gwei (-9), since the smallest denomination, wei (-18) caused issues with Stream probing the exchange rate (but settlements are denominated in wei)
ERC-20s have varying numbers of decimal places for their smallest on-ledger denomination, and I have no idea what scales connector operators will actually want/need to use (which makes it hard to standardize)
While I would strongly encourage peers to set their ILP packet asset scale equal to the smallest ledger denomination, clearly there are use cases for some peers using scales both smaller and larger than the ledger’s smallest unit. Peers may have different reasons for using different scales.
For this, I propose an assetScale configuration option for the “ledger adapter” to define the units the amounts in sendSettlement and receiveSettlement calls are denominated in. Then, the “ledger adapter” would be responsible for converting this amount to the unit the ledger actually uses. When it receives it a settlement, it’d convert that amount unit to that assetScale and call receiveSettlement with the converted amount.
To be safe when converting from smaller units to larger ones, it would always round them down. If two peers are configured to use the same assetScale at the ILP layer, this shouldn’t become an issue.
(assetCode as a configuration option probably doesn’t need to be standardized, but other configuration options, such as setting the ERC-20 token contract address, may trigger the “ledger adapter” to use a particular symbol, if it doesn’t already have a default such as “XRP” or “BTC”).
Liquidity Operations
I strongly do not think operations related to liquidity management can be cleanly abstracted to increaseLiquidity / decreaseLiquidity, and that the associated business logic will need to exist within the “ledger adapter” itself.
Unlocking Outgoing Liquidity?
For unidirectional channels, one important part of “unlocking” liquidity is asking the peer to claim the outgoing channel from you to them, since you can’t. (e.g., even if you claim the incoming channel, you might have a lot locked up in the outgoing side that you want back)
How would that be handled in this framework?
Both XRP and ETH paychans have mechanisms “dispute” or trigger an uncooperative close. In the XRP plugins, this timeout is set to an hour. In the ETH plugin, this is set to 6 days for security, since by contrast to XRP, there’s no way to cheaply checkpoint a claim on the ledger. Also on ETH, disputing the channel may cost non-negligible transaction fees. On both, disputing the channel is supposed to trigger the peer to claim/close it, but that only works if they’re online.
The nicer approach is for the node to politely ask its peer to claim the channel; and, only if they don’t, uncooperatively close.
Thus, the timing of the liquidity actually being unlocked is very very settlement system -specific, and the cost can vary significantly. How would decreaseLiquidity account for those differences?
Request Timeout
On ETH, channel opening could (at worst) take two minutes, since–in the worst case–it requires two transactions, and assuming the blocks took 1 minute each to mine. Would the request timeout? Or would increaseLiquidity respond immediately, and then would the “settlement engine” poll getLiquidity until it was updated?
On Lightning, it’d be useful to have the functionality to open a direct channel with your connector. Since the time to finality is so long (60 minutes), how would that work with this API?
Fees
For example, for more expensive ledgers, I think fees will be a major factor in how often additional collateral is deposited, or when / how often collateral is unlocked. This becomes very important as a DoS protection against open servers, so you can’t continually trigger them to burn their funds in fees.
And, fees cannot be generalized to a single interface, because for some ledgers (cough ETH), they’re really complex:
Fees need to be estimated before the channel open/close actually happens in order to have logic on whether to do it or not
ERC-20 tokens don’t pay fees in that asset, but pay fees in ETH (!!!)
Some transactions require fee x, but only after the tx is performed, some of the fee is refunded
The unit for fees is may be a different scale than the accounting (in ETH, fees may need precision of wei, rather than the gwei, if the gas price used is more precise than gwei)
Fees may play an even bigger role in business logic surrounding on-ledger settlements. If fees are expensive, we may want to add a behavior so the sending peer deducts the fee from its peer’s balance. But, the receiving peer may want to explicitly authorize each settlement. If not, from the receiver’s perspective, it’s like, “Why did you charge me $2 so you could send a settlement to me right now? I’m extending you plenty of credit, no settlement was necessary!”
By contrast, the receiver may want to authorize a faster settlement with a higher fee to execute it faster (in the same way Venmo optionally charges you 3% to instantly transfer money to your bank account, or you can wait a few days for free transfers).
I think these examples of fee-related business logic will prove to be more imperative than simply generic logic watching changes in liquidity, and at the very least, I think generic liquidity management can’t be properly implemented without such fee logic.
TL;DR
I strongly think the business logic and automated liquidity management cannot reasonably be generalized to all integrations, and probably not even all payment channel -based integrations. This is a concern within that “ledger adapter” itself.
I do think “ledger adapters” need additional RPCs to e.g. allow manual opening and closing of channels, but these probably can’t be uniformly standardized (at least without being very, very opinionated, and not very flexible).
This proposal simply requires the SE to understand WHEN it should request liquidity not to know HOW to make it available.
The goal with this proposal is for the SE to (over time through observation or through explicit config) know with some level of accuracy how long it will take to free up liquidity and so request this if it forecasts a need to perform a settlement that may not have the necessary liquidity.
It is then up to the LA to “make it happen”, even if this means sending a message to the counter-party requesting a collaborative rebalancing of unidirectional channels.
We anticipated all requests from the SE to the LA being async. I.e. You request that something is done and get an ACK. In parallel, notifications about changes of balance will let you know when they have been completed.
Our thinking here was that the most complex part of this process sits at the SE so, if possible, we should abstract everything else out as far as possible and only write this part once.
I recognise, especially when looking at the fees, this may be unnecessarily complex.
In that case, does an SE/LA per ledger make more sense?
Said differently, maybe we should standardise the Connector <-> SE interface?
If so I’d strongly advocate for an interface where that component simply tracks the balances on the connector(s) per account and performs settlements as necessary and then updates that balance.
i.e. getBalance is complemented by a debounced stream of balance changes from the connector to the SE.
If we are happy to abandon the idea of a Ledger Adaptor interface for now then I’d advocate that we take the conversation back to Settlement Architecture so it’s not split between 2 threads.
I’m not convinced that streaming balance updates to the SE/LA is better than sending specific messages to trigger settlement. Trying to figure out a way to write up the options that gets at the heart of the differences between them.
Whether it’s called a Settlement Engine or a Ledger Adapter, we’re talking about the API from the connector to this external system. The main sticking points are:
Whether the API is a stream of balance updates or calls to send money
Whether the logic for tracking account balances and triggering settlements lives in the connector or this external component
As far as I can see, the stream of balance updates is strictly worse than explicit send money calls on a number of dimensions and I am having trouble seeing any benefits to it:
@kincaid brought up an important security consideration about the balance stream API: before the settlement engine would trigger a settlement, it would need to first make an API call to the connector to “claim” or set aside whatever portion of the balance it was going to try to settle, and wait until that API call was resolved before settling. If you don’t, the account holder could send ILP packets that would be deducted from the balance at the same time that the SE is sending an outgoing settlement, thus getting paid out twice what they owe. It seems more straightforward to have the connector check whether it needs to settle, deduct the balance, and then send a send money call to the external system.
Too many and too few messages. Using a debounced stream would mean that the connector sends many completely unnecessary messages to the external system (i.e. when it does not need to settle) and too few messages if the account experiences a sudden burst of activity. If the connector instead send messages only when the account actually needs to settle, it would send no messages when the external system doesn’t need to take any action and it would send more in quick succession if the account has a lot of activity and needs to settle more than once.
If the balance logic lives in the external shared component, that means that we will have a drawn out debate like this one every time we need to change it. If we keep the logic out of this shared component, we can change the logic the connector (or business logic thing that’s attached to the connector) uses to determine when it should call send money whenever and however we want.
The balance stream API and logic behind it is more different from what’s already implemented in the plugins and so would take more work to convert the existing integrations.
No. As already discussed in Settlement Architecture the external system either includes a “Settlement Engine” (the component that decides when to settle and by how much, as defined in that same thread) or not.
This changes how you design the API.
The purpose of this thread was to discuss the API for an external system that DOES NOT include settlement business logic (i.e. it’s just a ledger adaptor). If you are suggesting that this is a bad design then let’s abandon this thread and go back to Settlement Architecture where we have all of the option properly laid out.
I’ve addressed some of your points above in that thread
I’m a little confused. I’m not suggesting having an external settlement engine, I prefer the Ledger Adapter that only abstracts the send money call. It sounded like the API we were discussing in this thread didn’t support the more complex logic you wanted to implement. To you, that seemed to suggest that we should go back to having an external settlement engine instead of a ledger adapter. To me, that suggested only that the settlement logic you wanted to implement was overly complicated but did not suggest that we should scrap the ledger adapter.
