Settlement Architecture

[ekrenzke]

Awesome work so far, @adrianhopebailie. I am glad this is getting more attention, because the previous plug-in architecture has been quite frustrating to work with at times.

This SHOULD be in the form of an ILP packet addressed to peer.settle.* hence I have proposed that this would be “injected” by the Settlement Engine into the outgoing middleware pipeline but really this is up to implementors.

That’s pretty slick.

I see the difference between the two proposals as the question of where the “settlement engine” component should exist (but not whether it’s an external service or an internal service). Does it exist between the balance service and the “ledger adapter,” interfacing between the two, or does its logic exist within the “ledger adapter” itself?

I believe the ledger adapter just exposes generic functionality to the ledger from the settlement engine in this proposal. The role of the ledger adapter is to provide a mapping from settlement logic (e.g. paying or requesting payment for outstanding debt obligations at an arbitrary time) to the specific-ledger functionality (e.g. signing base-layer transactions or handling payment channel claims). In other words, the settlement engine is used for maintaining relationships on the network with generic business logic operations, but the ledger adapter abstracts all of the low-level ledger-specific operations away.

Suppose we form a connection and issue a bunch of trades back and forth. You have effectively credited me 0.01 BTC in the process, which I must pay at settlement time. The terms of our credit relationship say that we must settle any time the credited amount exceeds 0.01 BTC, or approximately every hour – whichever comes first.

If I do not fulfill my debt obligation, then you start sending an appropriate reject message whenever I try to send a payment through the connection we share. The settlement engine allows us to (independently?) define business logic. In this example, I know why you are unwilling to route my packets – we have to settle up. My settlement engine uses the ledger adapter to construct and broadcast a Bitcoin transaction, and then your settlement engine uses the ledger adapter for confirmation.

The settlement engine allows us to implement business logic based on value and time without worrying about ledger-specific functionality, afforded by the adapter.

If it has very complicated ledger-specific logic around triggering settlement, per what you and Matt were suggesting, then does it have to be reimplemented for different “ledger adapters”?

For a minimal ledger adapter, I do not think so. A minimal implementation would probably only need to include sending money, checking balance, and possibly sending data. If we think of a ledger as a language, the ledger adapter is acting as the Rosetta Stone for business logic within the settlement engine. The business logic within the settlement engine is effectively a domain-specific language that can be extended to include ledger-specific functionality.

One thing I’m unclear about: what is the “settlement engine” doing?

The settlement engine is used for handling out-standing debt obligations. It serves as the conditional logic for packet forwarding, based on the credit relationships of incoming and outgoing connections. It is the semantic framework for packet forwarding, whereas the syntactic framework (e.g. packet malformation, fulfillment checks, octet encoding rules) is carried out by the base protocol.

If it has very complicated ledger-specific logic around triggering settlement, per what you and Matt were suggesting, then does it have to be reimplemented for different “ledger adapters”?

In most cases, I think the ledger adapter simply provides an interface for value and time. Hypothetically, it could be extended to build business logic using ledger-specific events (e.g. EVM or block height), but I believe this is a non-goal for minimal adapter implementations.

Also, could you provide specific examples of what logic you see the settlement engine might implement in both simple use cases and more advanced use cases? I think it might help better isolate what problem we’re trying to solve

Simple:

• Time-based Settlement (e.g. hourly )
• Value-based Settlement (e.g. credit threshold )

Advanced:

• Stop-Limit Order-based Settlement (e.g. price oracle )
• Event-sourced Settlement (e.g. EVM, block height )
• Demand-based Settlement (e.g. discounted settlement triggered by insufficient liquidity )

Please feel free to correct my interpretation. (@adrianhopebailie, @emschwartz)

[emschwartz]

@seanrowan the “settlement system” or ledger is the possibly global or at least multi-party thing that tracks a lot of different participants’ balances (think XRP, Bitcoin, etc). Each of those works in a completely different way, which makes integrating with many of them difficult.

We need some abstraction for those different settlement systems and the key questions are:

1. Does the settlement abstraction live in the same process as the connector (plugins), on its own (@adrianhopebailie called this the “ledger adapter”, though @kincaid disagrees with that term), or in the same process with the logic that dictates when to settle based on the ILP credit balance
2. What is the API for that abstraction?

We would like to be able to reuse the same abstraction component for the different implementations of Interledger, which means we need to agree on how it works and what the interface is.

To be clear though, the “settlement engine” / “ledger adapter” is a component that would be run by a single party and would manage their money on the underlying settlement system. Byzantine fault tolerance may be a feature of the underlying settlement system, but not the engine/adapter/abstraction on top of it.

[austin_king]

Thanks @adrianhopebailie @emschwartz @kincaid for all of the input and discussion, appreciate you guys moving this forward. I wanted to offer up my perspective given what Strata is seeing in the network today in hopes that giving some main net context could help us choose the best solution.

For the more complex, business level relationships that we are helping coordinate, the “business logic” for settlement is still very simple. The settleThreshold configuration is totally sufficient for the present time, and we anticipate it will not be an issue for a very long time to come. This makes me think that if this is the case for businesses that are handling higher volume between one another, that it is likely also the case for an open source implementation of the connector that we want to make available for others to run.

Given this, I think that designing the settlement engine as an entire stand alone component is an over optimization for the current time. I’m not strongly opinionated about where it should exist (in the connector vs. in the ledger adapter) but I think that designing the settlement engine as a stand alone piece introduces unnecessary complexity.

Happy to discuss further, and thanks again for all of the time you guys have spent developing this crucial part of the ecosystem.

[adrianhopebailie]

Based on comments from @ekrenzke, @kincaid and @austin_king both here and on the other Ledger Adapter Interface thread, I think we should not try to abstract the Settlement Engine -> Ledger Adaptor interface.

1. It’s complex and the complexity is probably not warranted at this time
2. It’s hard to get the abstraction right
3. Worst case scenario is that some code is re-used between different ledger specific settlement engines and this is extracted into stand-alone libraries

As I said on that thread I’d advocate for standardising on the interface I proposed above as:

Settlement Engine -> Connector

• getBalance (get the current balance of the clearing account, i.e. ILP balance)
• setBalance (update the current balance of the clearing account, apply a delta)
• sendData (send a message to the Settlement Engine on the peer)
• receiveData (accept a message from the Settlement Engine on the other peer)

This can be optimised (now or when it’s deemed to be required) by supplementing getBalance with a subscribeToBalanceChanges that allows the SE to get a stream of balance updates from the connector instead of polling.

I am confident that concerns about the volume of updates that will be sent can be alleviated using a debounced stream however I think this is a premature optimisation.

[adrianhopebailie]

Addressing some comments from @emschwartz in another thread:

Ledger Adapter API

@kincaid brought up an important security consideration about the balance stream API: before the settlement engine would trigger a settlement, it would need to first make an API call to the connector to “claim” or set aside whatever portion of the balance it was going to try to settle, and wait until that API call was resolved before settling. If you don’t, the account holder could send ILP packets that would be deducted from the balance at the same time that the SE is sending an outgoing settlement, thus getting paid out twice what they owe. It seems more straightforward to have the connector check whether it needs to settle, deduct the balance, and then send a send money call to the external system.

This applies however you do settlement. In fact it happens today in effect when you do a sendMoney, adjust the ILP balance and then the settlement fails.

Settlement is orchestration of atomic state changes to different systems so there is always a risk that one change happens before the other or that you need to roll-back changes on one to cover that they failed on the other.

The safest way to do this is to put the liquidity on hold at the connector, attempt the settlement and then release it if the settlement fails.

Ledger Adapter API

Too many and too few messages. Using a debounced stream would mean that the connector sends many completely unnecessary messages to the external system (i.e. when it does not need to settle) and too few messages if the account experiences a sudden burst of activity. If the connector instead send messages only when the account actually needs to settle, it would send no messages when the external system doesn’t need to take any action and it would send more in quick succession if the account has a lot of activity and needs to settle more than once.

This is not really an argument agains the interface design, it’s an argument against moving the settlement business logic out of the connector because it’s too difficult to get consensus on the interface design.

This is a fair argument and I can see that it may be the pragmatic way forward. We have implemented the settlement logic in a different component to the routing/balance checking logic in Rafiki so that we have the flexibility to scale these independently in time. We imagine running a single settlement engine per settlement system that is designed to be robust, stateful and always up so it can do things like watching settlement ledger for closing channels and incoming payments in contrast to the connector which will be stateless and can scale quickly to serve single or multiple peers.

Right now most Rafiki instances will run the connector and SE in the same process but at least by splitting this out logically now we can split them into their own process in future.

Ledger Adapter API

If the balance logic lives in the external shared component, that means that we will have a drawn out debate like this one every time we need to change it. If we keep the logic out of this shared component, we can change the logic the connector (or business logic thing that’s attached to the connector) uses to determine when it should call send money whenever and however we want.

We had envisioned this differently actually. We’ll continue to build out our SE implementation and if anyone wants to use it they can simply stream balance updates to it. Our expectation was that we’d develop an SE for a single settlement system and then re-use non-ledger specific code for others.

We MIGHT try to separate out the SE and Ledger Adaptor as proposed in that thread but probably not for now.

Ledger Adapter API

The balance stream API and logic behind it is more different from what’s already implemented in the plugins and so would take more work to convert the existing integrations.

I wouldn’t bet on any of this being a small amount of work unless we just stick with the plugins as they are

[kincaid]

It’s a game of telephone!

My point to Evan was that in order to do it safely, the external balance service approach (even assuming it uses a debounced stream), requires a lot more messages.

1. Connector sends balance update to balance service
2. Balance service notifies settlement engine of update
3. Settlement engine decides to settle, then sends message to balance service with new balance
4. Balance service ACKs that update, which is important because the SE can’t safely settle until it gets the ACK

(1.5 roundtrips, excluding connector -> balance service)

Whereas, with the sendMoney/receiveMoney approach:

1. Connector update balances, decides to settle, then sends sendMoney message to settlement engine. The SE can safely start settling immediately.

(0.5 roundtrips)

This applies however you do settlement. In fact it happens today in effect when you do a sendMoney, adjust the ILP balance and then the settlement fails.

Not with the ETH plugin, Lightning plugin, or Kava’s XRP plugin!

They all persist a “payoutAmount” for failed settlements, and will retry it later.

I wouldn’t bet on any of this being a small amount of work unless we just stick with the plugins as they are

Which…kinda?

We already migrated our XRP payment channel plugin and added a sendMoney/receiveMoney HTTP API: https://github.com/Kava-Labs/ilp-plugin-xrp-paychan/tree/ko-http-api (refer to tests)

On the Rust connector side, since most of the balance logic is already implemented, I imagine interfacing with this should be relatively straightforward.

(I would emphasize that this branch is something I just threw together for testing, but it should do the thing, allowing two peers to open channels, and send and receive paychan claims, using whatever transport the connector provides).

The larger amount of work, as we see it, is getting these settlement engines (sorry Adrian) to a production-grade state for connector operators. Right now, they’re mostly a blackbox: there’s either no admin management interface, or no management interface that works reliably. That means no way to get your money out, and certainly no good way to understand what’s actually happening. And then ultimately, building out more robust automatic liquidity management.

Switch is more or less a much simplified “admin management” interface for end users. But building the same thing for connector operators will probably be more challenging

[adrianhopebailie]

They all persist a “payoutAmount” for failed settlements, and will retry it later.

So the balance in the connector and the settled amount get out of sync. That’s just a way of reducing the number of messages required to stay in sync by allowing the synchronization to be delayed isn’t it?

[kincaid]

So the balance in the connector and the settled amount get out of sync. That’s just a way of reducing the number of messages required to stay in sync by allowing the synchronization to be delayed isn’t it?

Yes, they get temporarily out of sync.

AFAIK, the only real operational disadvantage of that is after a sendMoney call, if the settlement fails or otherwise is not sent immediately, that owed balance cannot be netted against incoming packets. In your approach, it could be refunded to the balance shared with the connector, enabling it to be netted.

I think that’s a reasonable tradeoff for the ease of implementation, lesser “API surface” required to get consensus on, and other benefits, but it’s worth digging into further.

[adrianhopebailie]

I think that’s a reasonable tradeoff for the ease of implementation, lesser “API surface” required to get consensus on, and other benefits, but it’s worth digging into further.

Don’t disagree at all. As I’ve said before we should just be clear when we make a design decision what the trade-offs are. This is really helpful, especially in the future, when we or others want to re-assess the design or understand why we took the path we did.

[sappenin]

So where does that leave us? Reading through both this thread and the discussion in Ledger Adaptor API, it’s unclear to me where there’s consensus and where there is not.

Here’s how things appear to stand from may vantage point (I’m guessing I’ve missed a few things, or am possibly simply wrong in certain areas, so please chime-in if you disagree):

Consensus

1. Settlement Adaptor: We want a non-Connector process that can interact with an underlying settlement system (e.g., Bitcoin, XRPL, etc) that does not run in the Connector process. This will allow us to scale Connectors and Settlement systems independently. I’m calling this thing the Settlement Adaptor (shout-out to @kincaid for good rationale around why to drop the word “ledger” here).
2. Settlement Engine: We all agree that ILP-account balance-tracking (i.e., “when to settle based on the ILP credit balance”) should live in a Settlement Engine, but for now this component will just be an internal service/function of each Connector implementation.
   1. Based upon how implementations get built, it may make sense in the future to extract this into a standalone system and/or shared-libraries, but this is out of scope for now.
3. Connector/SE <-> SA Transport: To aid adoption and for simplicity, we’re all OK with the Connector/SE communicating with the Settlement Adaptor using an HTTP+JSON API as a first-attempt. We may explore other techniques later if it makes sense.
4. Settlement Adaptor API Endpoints
   1. sendMoney: When called by an external system (e.g., a Connector), instructs the Settlement Adaptor to send a payment via the underlying settlement system (e.g., XRPL).
   2. getBalance: Returns the current balance of the underlying settlement account (i.e. available liquidity for settlements).
   3. sendData: When called by an external system (e.g., a Connector), allows an ILP packet to be transmitted to the Settlement Adaptor for further processing, such as to exchange a Payment Channel claim.
5. Connector/SE API Endpoints
   1. sendData: Allows the local Settlement Adaptor to communicate with the other Settlement Adaptor (used by the peer Connector) by transiting the local Connector using ILP link-layer technologies and peer.settle. addresses.
   2. sendMoney: Allows the Settlement Adaptor to indicate that a settlement payment was received on the underlying settlement system. For example, the account 123 received 10 XRP. The Connector will react to this callback and update its balances appropriately.

Open Questions

1. Are the above API endpoints correct?
2. If you get into a taxi cab, and ask the driver to drive backwards to your destination, will the cab driver owe you money?

[adrianhopebailie]

I think you have it close enough for us to just get on and build it!

Where I think we are not entirely in agreement (but this is immaterial if we put the SE in the connector) is on where the liquidity management logic should sit.

@kincaid suggests that it is too hard to abstract this so it should live in the Settlement Adaptor.

I don’t agree that it’s impossible and think it should be in the SE but agree that abstracting it now may be adding unnecessary complexity to the SA interface.

My main concern is that we are splitting two pieces of business logic that I think belong together, the settlement logic (when should I settle) and the liquidity management (how much liquidity have I got available for settlements) should both sit in the SE.

The current proposal puts the settlement logic in the connector and liquidity management in the SA.

Are the above API endpoints correct?

Correct enough to start with in my opinion.

If you get into a taxi cab, and ask the driver to drive backwards to your destination, will the cab driver owe you money?

So, let’s just be clear on the assumptions at play here…

[adrianhopebailie]

Conclusion

Thanks to all who contributed to this and joined the Community Call on 15 May to discuss. Here is my summary of the outcome (thanks @sappenin for your notes in Slack, they helped jog my memory):

Components

• Connector Balance Logic:
  Logic in the connector that decides whether or not to forward packets based on the Interledger balance of a peer.
  Adjusts Interledger balance based on exchange of packets with peer.

• Accounting System:
  Tracks the Interledger balance and determines when to make a settlement.
  Communicates with the settlement engine and adjusts the Interledger balance based on settlement events.

• Settlement Engine:
  Performs settlements on a settlement system.
  Manages liquidity in the settlement system.
  Communicates with the settlement engines at peers using a transport provided by the connector

Design Decisions

What is the Settlement Engine?

The terminology we came to consensus on is not exactly what was described earlier in this thread so take not of the differences. The reasons are less important than the results, it’s just terminology after all.

Notably we have agreed that the system that runs outside the connector and interfaces with the settlement system is called the Settlement Engine and this DOES NOT contain the logic for deciding WHEN to settle (which is in the accounting system).

In effect this consists of two components, a settlement system liquidity manager (manages the available liquidity for settlement) and a settlement system adaptor (interfaces with the settlement system to, primarily, perform settlements).

Liquidity management of Settlement System is handled in the Settlement Engine

We have settled (pun intended) on a design that does not attempt to abstract the liquidity management functions of the Settlement Engine out of that component as the majority view was that this is quite specific to the underlying settlement system and difficult to abstract.

As such, the Accounting System has no view of the connector’s liquidity in the settlement system. Instead the AS simply requests a settlement and assumes this will be done eventually if the request is acknowledged by the SE.

The AS adjusts its own records to this effect and, in the case that the SE is unable to complete the settlement, the connector may find that packets forwarded to the peer start to be rejected due to a lack of liquidity.

This was accepted as the lowest risk compromise for the sake of keeping the design simple and reducing the API surface between the AS and the SE.

Management of this scenario is then left to operators to ensure they can respond to this eventuality. In reality, this will manifest as failed settlements which should be handled with high priority by an operator anyway.

The alternative to this proposal was to include the liquidity management logic in the accounting system which has the benefit of allowing the accounting system to develop more sophisticated logic around management of liquidity in the settlement system and in the Interledger accounts however this comes at the cost of needing to abstract liquidity management functions.

In time, data from the settlement process (the time taken and rate of success performing settlements) may be useful to the AS in planning future settlements and we may extend the API to enable this.

REST

For simplicity we have agreed to use a REST and JSON based API

Architecture

We have agreed that the standard API we want to define is between the Accounting System (which will likely run inside most connectors for now) and the Settlement Engine.

Both sides must run an HTTP server to accept requests as the interface is bilateral.

All interactions follow a request/reply pattern however some replies will be responses and others will simply be ACKs.

Settlement Engines have an implicit asset code and scale (i.e. a different SE is required for different assets) so the AS must apply any scale conversions before calling the SE API or when receiving API calls from the SE.

The Accounting System (Connector) must host the following interface:

• ReceiveSettlement(accountId, amount)
  Notification from the settlement system that an incoming settlement has been received.
  The response to this is an ACK.
  The AS will ACK the request and adjust the Interledger balance of the peer accordingly.
  The SE should retry until this is ACK’d so it needs to be idempotent.

• SendRequest(accountId, message)
  Deliver a message to the peer’s settlement system and return the response.
  The response is the response from the peer’s settlement system.
  If this is used for some form of RPC between settlement systems then they should define that protocol separately (i.e. how to handle failures and errors).
  An error response from the other SE should not be carried in an error at the API level.
  (i.e. The response is always a 2XX)
  The SE should retry until it gets a response so this needs to be idempotent.

Both calls can return an “Unknown Account Id” response (Maybe a 404?)

The Settlement Engine must host the following interface:

• CreateAccount(accountId)
  Be prepared to send and receive settlements for this account.
  Connector provides the accountId and SE .

• SendSettlement(accountId, amount)
  Perform an outgoing settlement on the underlying settlement system.
  The response to this is an ACK.
  The SE will ACK the request and then attempt to perform the settlement. It will continue attempting the settlement until it succeeds (i.e. There is no mechanism to notify the AS to reverse the balance change)
  The AS should retry until it gets a response so this needs to be idempotent.

• ReceiveRequest(accountId, message)
  Accept a message from the peer’s settlement system and return a response.
  The SE should never return an error response unless the accountId is unknown.
  (i.e. The response is always a 2XX even if the response is carrying an error message back to the peer’s SE)
  The AS should retry until it gets a response so this needs to be idempotent.

Discussion on how to spec this out is being done in the #settlement channel on Slack

Kava have an implementation that is already pretty close to this: https://github.com/Kava-Labs/ilp-plugin-xrp-paychan

UPDATE: Added amount as a param to ReceiveSettlement. This was a mistake in the original post.

[Lucian_Trestioreanu]

I think there might be a small flaw in the JS XRP settlement engine https://github.com/interledgerjs/settlement-xrp.

In file: .nvm/versions/node/v10.16.3/lib/node_modules/ilp-settlement-xrp/dist/index.js

The SE of the receiving party checks the tx result immediately after the transaction is submitted.

handleTransaction(tx) {
            log(`MY_TRACE: Went through handleTransaction(tx)`);
            log(`MY_TRACE: tx`, tx);
            if (!tx.validated ||
                tx.meta.TransactionResult !== 'tesSUCCESS' ||
                tx.transaction.TransactionType !== 'Payment' ||
                tx.transaction.Destination !== xrpAddress) {
                log(`MY_TRACE: Returning: !tx.validated, !==tesSUCCESS`);
                return;
            }
            log(`tx amount`, tx.transaction.Amount);
            //const amount = new bignumber_js_1.default(tx.meta.delivered_amount).shiftedBy(-6);
            const amount = new bignumber_js_1.default(tx.transaction.Amount).shiftedBy(-6); 

In case the validation will be slow, the tx result will be the intermediary one - non-validated.
As such, the field tx.meta.delivered_amount is not found, and although the money are already on the ledger, the ILP balance between the parties will de-sync, because the receiver still believes the balance is not settled yet.

I tried a quick fix by using the field tx.transaction.Amount, as shown on the last line of code above, and like this, it works.

[sappenin]

Hi @Lucian_Trestioreanu thanks for pointing that out. Can you make an issue in the Github project?

interledgerjs/settlement-xrp

[Lucian_Trestioreanu]

@sappenin @kincaid
Hi, I put it here:

settlement tx not checked correctly in case of slow validating network

[Lucian_Trestioreanu]

What if two connectors want to peer over e.g. three currencies/ledgers, i.e. there would be three settlement engines between the two connectors?

How would settlement proceed?

If, at some moment the two connectors become part of a payment chain between 2 parties, how would payment routing proceed?
Alice -> Connector 0 -> Connector 1 (3 SE with Conn 2) -> Connector 2 (3 SE with Conn1) -> Connector 3 -> Bob

[sappenin]

What if two connectors want to peer over e.g. three currencies/ledgers, i.e. there would be three settlement engines between the two connectors?

Each connector needs a Settlement Engine for each currency/ledger that it wants to peer using. So, if two connectors are peering over two different ledgers, there would be 4 settlement engine instances (2 for each Connector).

[Lucian_Trestioreanu]

Hi, @sappenin Sorry for the misunderstanding. This is what I meant:

How would settlement proceed between Conn 2 and Conn 3?

[matdehaast]

Currently not supported on any connector implementations, but you would have Business Logic tied into Routing on the connector to make that decision on which SE it would want to use. This could be based on various factors such as liquidity, price etc.

We have discussed this functionality being available in all connectors but it just hasn’t been prioritized yet.

[Lucian_Trestioreanu]

Thank you very much, Matt!