Settlement Architecture

[adrianhopebailie]

Addressing some comments from @emschwartz in another thread:

@emschwartz

@kincaid brought up an important security consideration about the balance stream API: before the settlement engine would trigger a settlement, it would need to first make an API call to the connector to “claim” or set aside whatever portion of the balance it was going to try to settle, and wait until that API call was resolved before settling. If you don’t, the account holder could send ILP packets that would be deducted from the balance at the same time that the SE is sending an outgoing settlement, thus getting paid out twice what they owe. It seems more straightforward to have the connector check whether it needs to settle, deduct the balance, and then send a send money call to the external system.

This applies however you do settlement. In fact it happens today in effect when you do a sendMoney, adjust the ILP balance and then the settlement fails.

Settlement is orchestration of atomic state changes to different systems so there is always a risk that one change happens before the other or that you need to roll-back changes on one to cover that they failed on the other.

The safest way to do this is to put the liquidity on hold at the connector, attempt the settlement and then release it if the settlement fails.

@emschwartz

Too many and too few messages. Using a debounced stream would mean that the connector sends many completely unnecessary messages to the external system (i.e. when it does not need to settle) and too few messages if the account experiences a sudden burst of activity. If the connector instead send messages only when the account actually needs to settle, it would send no messages when the external system doesn’t need to take any action and it would send more in quick succession if the account has a lot of activity and needs to settle more than once.

This is not really an argument agains the interface design, it’s an argument against moving the settlement business logic out of the connector because it’s too difficult to get consensus on the interface design.

This is a fair argument and I can see that it may be the pragmatic way forward. We have implemented the settlement logic in a different component to the routing/balance checking logic in Rafiki so that we have the flexibility to scale these independently in time. We imagine running a single settlement engine per settlement system that is designed to be robust, stateful and always up so it can do things like watching settlement ledger for closing channels and incoming payments in contrast to the connector which will be stateless and can scale quickly to serve single or multiple peers.

Right now most Rafiki instances will run the connector and SE in the same process but at least by splitting this out logically now we can split them into their own process in future.

@emschwartz

If the balance logic lives in the external shared component, that means that we will have a drawn out debate like this one every time we need to change it. If we keep the logic out of this shared component, we can change the logic the connector (or business logic thing that’s attached to the connector) uses to determine when it should call send money whenever and however we want.

We had envisioned this differently actually. We’ll continue to build out our SE implementation and if anyone wants to use it they can simply stream balance updates to it. Our expectation was that we’d develop an SE for a single settlement system and then re-use non-ledger specific code for others.

We MIGHT try to separate out the SE and Ledger Adaptor as proposed in that thread but probably not for now.

@emschwartz

The balance stream API and logic behind it is more different from what’s already implemented in the plugins and so would take more work to convert the existing integrations.

I wouldn’t bet on any of this being a small amount of work unless we just stick with the plugins as they are