Settlement Architecture

[adrianhopebailie]

I think you have it close enough for us to just get on and build it!

Where I think we are not entirely in agreement (but this is immaterial if we put the SE in the connector) is on where the liquidity management logic should sit.

@kincaid suggests that it is too hard to abstract this so it should live in the Settlement Adaptor.

I don’t agree that it’s impossible and think it should be in the SE but agree that abstracting it now may be adding unnecessary complexity to the SA interface.

My main concern is that we are splitting two pieces of business logic that I think belong together, the settlement logic (when should I settle) and the liquidity management (how much liquidity have I got available for settlements) should both sit in the SE.

The current proposal puts the settlement logic in the connector and liquidity management in the SA.

@sappenin

Are the above API endpoints correct?

Correct enough to start with in my opinion.

@sappenin

If you get into a taxi cab, and ask the driver to drive backwards to your destination, will the cab driver owe you money?

So, let’s just be clear on the assumptions at play here…

[adrianhopebailie]

Conclusion

Thanks to all who contributed to this and joined the Community Call on 15 May to discuss. Here is my summary of the outcome (thanks @sappenin for your notes in Slack, they helped jog my memory):

Components

• Connector Balance Logic:
  Logic in the connector that decides whether or not to forward packets based on the Interledger balance of a peer.
  Adjusts Interledger balance based on exchange of packets with peer.

• Accounting System:
  Tracks the Interledger balance and determines when to make a settlement.
  Communicates with the settlement engine and adjusts the Interledger balance based on settlement events.

• Settlement Engine:
  Performs settlements on a settlement system.
  Manages liquidity in the settlement system.
  Communicates with the settlement engines at peers using a transport provided by the connector

Design Decisions

What is the Settlement Engine?

The terminology we came to consensus on is not exactly what was described earlier in this thread so take not of the differences. The reasons are less important than the results, it’s just terminology after all.

Notably we have agreed that the system that runs outside the connector and interfaces with the settlement system is called the Settlement Engine and this DOES NOT contain the logic for deciding WHEN to settle (which is in the accounting system).

In effect this consists of two components, a settlement system liquidity manager (manages the available liquidity for settlement) and a settlement system adaptor (interfaces with the settlement system to, primarily, perform settlements).

Liquidity management of Settlement System is handled in the Settlement Engine

We have settled (pun intended) on a design that does not attempt to abstract the liquidity management functions of the Settlement Engine out of that component as the majority view was that this is quite specific to the underlying settlement system and difficult to abstract.

As such, the Accounting System has no view of the connector’s liquidity in the settlement system. Instead the AS simply requests a settlement and assumes this will be done eventually if the request is acknowledged by the SE.

The AS adjusts its own records to this effect and, in the case that the SE is unable to complete the settlement, the connector may find that packets forwarded to the peer start to be rejected due to a lack of liquidity.

This was accepted as the lowest risk compromise for the sake of keeping the design simple and reducing the API surface between the AS and the SE.

Management of this scenario is then left to operators to ensure they can respond to this eventuality. In reality, this will manifest as failed settlements which should be handled with high priority by an operator anyway.

The alternative to this proposal was to include the liquidity management logic in the accounting system which has the benefit of allowing the accounting system to develop more sophisticated logic around management of liquidity in the settlement system and in the Interledger accounts however this comes at the cost of needing to abstract liquidity management functions.

In time, data from the settlement process (the time taken and rate of success performing settlements) may be useful to the AS in planning future settlements and we may extend the API to enable this.

REST

For simplicity we have agreed to use a REST and JSON based API

Architecture

We have agreed that the standard API we want to define is between the Accounting System (which will likely run inside most connectors for now) and the Settlement Engine.

Both sides must run an HTTP server to accept requests as the interface is bilateral.

All interactions follow a request/reply pattern however some replies will be responses and others will simply be ACKs.

Settlement Engines have an implicit asset code and scale (i.e. a different SE is required for different assets) so the AS must apply any scale conversions before calling the SE API or when receiving API calls from the SE.

The Accounting System (Connector) must host the following interface:

• ReceiveSettlement(accountId, amount)
  Notification from the settlement system that an incoming settlement has been received.
  The response to this is an ACK.
  The AS will ACK the request and adjust the Interledger balance of the peer accordingly.
  The SE should retry until this is ACK’d so it needs to be idempotent.

• SendRequest(accountId, message)
  Deliver a message to the peer’s settlement system and return the response.
  The response is the response from the peer’s settlement system.
  If this is used for some form of RPC between settlement systems then they should define that protocol separately (i.e. how to handle failures and errors).
  An error response from the other SE should not be carried in an error at the API level.
  (i.e. The response is always a 2XX)
  The SE should retry until it gets a response so this needs to be idempotent.

Both calls can return an “Unknown Account Id” response (Maybe a 404?)

The Settlement Engine must host the following interface:

• CreateAccount(accountId)
  Be prepared to send and receive settlements for this account.
  Connector provides the accountId and SE .

• SendSettlement(accountId, amount)
  Perform an outgoing settlement on the underlying settlement system.
  The response to this is an ACK.
  The SE will ACK the request and then attempt to perform the settlement. It will continue attempting the settlement until it succeeds (i.e. There is no mechanism to notify the AS to reverse the balance change)
  The AS should retry until it gets a response so this needs to be idempotent.

• ReceiveRequest(accountId, message)
  Accept a message from the peer’s settlement system and return a response.
  The SE should never return an error response unless the accountId is unknown.
  (i.e. The response is always a 2XX even if the response is carrying an error message back to the peer’s SE)
  The AS should retry until it gets a response so this needs to be idempotent.

Discussion on how to spec this out is being done in the #settlement channel on Slack

Kava have an implementation that is already pretty close to this: https://github.com/Kava-Labs/ilp-plugin-xrp-paychan

UPDATE: Added amount as a param to ReceiveSettlement. This was a mistake in the original post.

[Lucian_Trestioreanu]

I think there might be a small flaw in the JS XRP settlement engine https://github.com/interledgerjs/settlement-xrp.

In file: .nvm/versions/node/v10.16.3/lib/node_modules/ilp-settlement-xrp/dist/index.js

The SE of the receiving party checks the tx result immediately after the transaction is submitted.

handleTransaction(tx) {
            log(`MY_TRACE: Went through handleTransaction(tx)`);
            log(`MY_TRACE: tx`, tx);
            if (!tx.validated ||
                tx.meta.TransactionResult !== 'tesSUCCESS' ||
                tx.transaction.TransactionType !== 'Payment' ||
                tx.transaction.Destination !== xrpAddress) {
                log(`MY_TRACE: Returning: !tx.validated, !==tesSUCCESS`);
                return;
            }
            log(`tx amount`, tx.transaction.Amount);
            //const amount = new bignumber_js_1.default(tx.meta.delivered_amount).shiftedBy(-6);
            const amount = new bignumber_js_1.default(tx.transaction.Amount).shiftedBy(-6); 

In case the validation will be slow, the tx result will be the intermediary one - non-validated.
As such, the field tx.meta.delivered_amount is not found, and although the money are already on the ledger, the ILP balance between the parties will de-sync, because the receiver still believes the balance is not settled yet.

I tried a quick fix by using the field tx.transaction.Amount, as shown on the last line of code above, and like this, it works.

[sappenin]

Hi @Lucian_Trestioreanu thanks for pointing that out. Can you make an issue in the Github project?

interledgerjs/settlement-xrp

[Lucian_Trestioreanu]

@sappenin @kincaid
Hi, I put it here:

settlement tx not checked correctly in case of slow validating network

[Lucian_Trestioreanu]

What if two connectors want to peer over e.g. three currencies/ledgers, i.e. there would be three settlement engines between the two connectors?

How would settlement proceed?

If, at some moment the two connectors become part of a payment chain between 2 parties, how would payment routing proceed?
Alice -> Connector 0 -> Connector 1 (3 SE with Conn 2) -> Connector 2 (3 SE with Conn1) -> Connector 3 -> Bob

[sappenin]

What if two connectors want to peer over e.g. three currencies/ledgers, i.e. there would be three settlement engines between the two connectors?

Each connector needs a Settlement Engine for each currency/ledger that it wants to peer using. So, if two connectors are peering over two different ledgers, there would be 4 settlement engine instances (2 for each Connector).

[Lucian_Trestioreanu]

Hi, @sappenin Sorry for the misunderstanding. This is what I meant:

How would settlement proceed between Conn 2 and Conn 3?

[matdehaast]

Currently not supported on any connector implementations, but you would have Business Logic tied into Routing on the connector to make that decision on which SE it would want to use. This could be based on various factors such as liquidity, price etc.

We have discussed this functionality being available in all connectors but it just hasn’t been prioritized yet.

[Lucian_Trestioreanu]

Thank you very much, Matt!