Settlement Architecture

[adrianhopebailie]

How exactly would you like to proceed on this?

The goal is just to level-set and have a discussion about what we need to consider in designing a settlement architecture and propose some ideas. Mapping the Solution Space as @emschwartz suggested.

The current architecture looks like this:

All of the Settlement Business Logic and the Settlement System Interface are in the plugins AND all ILP packets go through the plugins too. @emschwartz gave a nice breakdown of the pros and cons of this approach in Plugin Architecture and Ledger Integrations. To summarise; the plugin has all the info it needs to implement sophisticated Settlement Business Logic BUT it is all tightly coupled with the interface to the Settlement System, the interface to the other connector and must be implemented in Javascript (or at least wrapped in a Javascript adaptor).

NOTE: Not shown here, but adding significant complexity is the lifecycle management aspects of the current architecture. A plugin can be in numerous lifecycle states (connecting, connected, disconnected, terminated etc) which are not granular enough to indicate the state of the multiple connections the plugin maintains (to the peer, to the settlement system etc).

The Settlement Engine

We’ve talked a lot about a “Settlement Engine” but I don’t believe we all agree on what that is. Below are my understandings of the two proposals on the table and an attempt to come to agreement on some definitions.

I’ve given the “Settlement Engine” a specific definition which I think is consistent with the use of the descriptor “Engine” in other architectures; it is responsible for orchestrating complex business logic.

I’ve also defined what I call the “Ledger Adaptor” as a separate component that simply holds settlement system-specific integration logic. i.e. The Ledger Adaptor knows how to speak to a settlement system but it only performs operations on that system when instructed to do so by the Settlement Engine.

The interface to the Ledger Adaptor is unsurprisingly familiar (sendMoney, receiveMoney, sendData, receiveData) but in contrast to the Javascript Ledger Interface, calling these APIs is not expected to have side-effects.

E.g. Calling sendMoney on the Ledger Adaptor results in a payment being sent over the settlement system. This may succeed or fail and the Settlement Engine decides how to proceed once the result of this operation is known.

The Ledger Adaptor is also able to call sendData on the Settlement Engine which is simply a way of requesting that a message is passed to the corresponding Ledger Adaptor on the peer. (e.g. To exchange a Payment Channel claim)

This SHOULD be in the form of an ILP packet addressed to peer.settle.* hence I have proposed that this would be “injected” by the Settlement Engine into the outgoing middleware pipeline but really this is up to implementors.

Note that the Settlement Engine could also initiate a message to the other Settlement Engine.

In both proposed architectures the Settlement Engine tracks the various balances and balance change events in both the clearing accounts (Peer ILP balances) and settlement accounts (Settlement system balances) and decides when to make a settlement on the settlement system (call sendMoney on the Ledger Adaptor).

The Settlement Engine also receives notifications of incoming settlements from the Ledger Adaptor. Not shown, but required is a mechanism for the Settlement Engine to query the Ledger Adaptor for balances on the settlement system.

The major difference between the two proposals is where the Settlement Engine is implemented.

Proposal 1 - Internal Settlement Engine

In this architecture the Settlement Engine is internal to the connector.

The interface that needs to be defined is between connectors and Ledger Adaptors. It likely includes:

• sendMoney (perform a settlement)
• receiveMoney (acknowledge an incoming settlement)
• sendData (send a message to the Ledger Adaptor on the peer)
• receiveData (accept a message from the Ledger Adaptor on the other peer)
• getBalance (get the current balance of the settlement account, i.e. available liquidity for settlements)

Proposal 2 - External Settlement Engine

In this architecture the Settlement Engine is external to the connector.

An interface needs to be defined between the Settlement Engine and the Connector AND between the Ledger Adaptor and the Settlement Engine:

Settlement Engine -> Ledger Adaptor:

• sendMoney (perform a settlement)
• receiveMoney (acknowledge an incoming settlement)
• receiveData (accept a message from the Ledger Adaptor on the other peer)
• getBalance (get the current balance of the settlement account, i.e. available liquidity for settlements)

Settlement Engine -> Connector

• getBalance (get the current balance of the clearing account, i.e. ILP balance)
• setBalance (update the current balance of the clearing account, apply a delta)
• sendData (send a message to the Settlement Engine on the peer)
• receiveData (accept a message from the Settlement Engine on the other peer)

Next steps…
If we agree on what is being proposed then I think we can proceed to discussing the pros and cons of the two designs and decide which we should use (or even find a way to support both).

[emschwartz]

This is an excellent summary. Thanks a lot @adrianhopebailie for writing it up!

A couple of clarifying questions:

• Are the APIs between the various components you’re describing assumed to be HTTP/RPC APIs (network calls), internal APIs, or is that to be determined?
• Would a combined settlement engine + ledger adapter be a 3rd option? (It seemed like that was what we were discussing before)

If we agree on having a ledger adapter with an HTTP/RPC API, then we actually don’t need to agree on the internal / external settlement engine design. I think the main question to this point has been how to abstract away different settlement systems in such a way that we don’t need to reimplement the integration in every programming language. If different people start working on external settlement engines that require different APIs from the connector, it may be nice to try to standardize those but it doesn’t seem like we need as broad agreement on that design as for the ledger adapter.

Update: I started a new thread on the Ledger Adapter API to work on designing that component.

[ernesteze]

Great post @adrianhopebailie!

In my opinion external Settlement Engine + combining the Ledger Adapter is a good solution. Like @emschwartz already mentioned that’s how I also had it mind.

Cheers

[adrianhopebailie]

If we agree on having a ledger adapter with an HTTP/RPC API, then we actually don’t need to agree on the internal / external settlement engine design.

Agree. We should focus on the ledger adaptor first. I think we could re-use a lot of the code in existing plugins for these.

Are the APIs between the various components you’re describing assumed to be HTTP/RPC APIs (network calls), internal APIs, or is that to be determined?

For max re-use it would be best if they were network APIs so that we can write the adaptors once and re-use them widely.

That said, and in line with your previous comment, we can probably leave the integration between the connector and settlement engine to evolve as people develop stand-alone engines. It sounds like a lot of connectors plan to do a simple engine in the connector.

I anticipate we’ll externalise this for Rafiki and as a result may have a proposal for this interface soon.

Would a combined settlement engine + ledger adapter be a 3rd option?

Sure, but this changes the API that we should prioritise standardising (Connector to SE) and I agree with the idea of focusing on the ledger adaptor first.

[ernesteze]

What are the advantages of starting with the Ledger Adapter (LA) over the Settlement Engine (SE)?

The SE seems is easier to standardized than a ledger adapter as the LA is too ledger-specific. Or am I missing something?

How do you plan on dealing with ledger-specific integrations?

[emschwartz]

Using @adrianhopebailie’s terminology, the Ledger Adapter is purely an abstraction over the ledger’s capability to send and receive money. In contrast, the Settlement Engine implements the logic for when those functions should be called.

How do you plan on dealing with ledger-specific integrations?

The Ledger Adapter is the ledger-specific integration. Each ledger would have its own Ledger Adapter that implements the common functions needed by the Settlement Engine / connector.

[kincaid]

Nice explanation! I think I broadly agree with the analysis between the two proposals, but I still have some open questions.

I don’t think the diagram in proposal 1 entirely accurately describes it. I see “sendMoney” and “receiveMoney” as a much more direct API between the “ledger adapter” and the “balance service.”

I see the difference between the two proposals as the question of where the “settlement engine” component should exist (but not whether it’s an external service or an internal service). Does it exist between the balance service and the “ledger adapter,” interfacing between the two, or does its logic exist within the “ledger adapter” itself?

Proposal 1 has the flexibility to:

• Implement a thin “settlement engine” in between the balance service and “ledger adapter,” which likely only uses basic settleTo/settleThreshold logic (so simple that it probably doesn’t need to be a separate service), or,
• Implement a complex “settlement engine” with ledger-specific orchestration logic that exists within the “ledger adapter” itself.

The sendMoney/receiveMoney primitive can be used in both cases (subject to a minor tradeoff(s), in each, which I can elaborate on).

I’ve given the “Settlement Engine” a specific definition which I think is consistent with the use of the descriptor “Engine” in other architectures; it is responsible for orchestrating complex business logic.

The Settlement Engine also receives notifications of incoming settlements from the Ledger Adaptor. Not shown, but required is a mechanism for the Settlement Engine to query the Ledger Adaptor for balances on the settlement system.

One thing I’m unclear about: what is the “settlement engine” doing?

If it has very complicated ledger-specific logic around triggering settlement, per what you and Matt were suggesting, then does it have to be reimplemented for different “ledger adapters”?

Also, could you provide specific examples of what logic you see the settlement engine might implement in both simple use cases and more advanced use cases? I think it might help better isolate what problem we’re trying to solve

---

As an aside, I have some differences on the proposed terminology (and maybe they go beyond nomenclature):

• I think “settlement” as a descriptor is clearer and more generic than “ledger.” For example, while payment channels are technically their own type of “ledger,” people don’t typically think of them that way, and calling the integration a “ledger adapter” could create confusion as to whether it integrates with the base layer ledger, such as Bitcoin or Ethereum.
  • I see this as an abstraction to integrate with any mechanism to settle outside of the credit relationship on Interledger. What if I implemented a custom settlement integration that, e.g., mailed you cash? Is there a “ledger” in that case?
• Your description of the intermediary component is that it “orchestrates complex business logic,” which fits the definition of “engine.”
  1. While addressed earlier, with respect to logic triggering settlement, some of this is tied directly to the settlement system, and may need to coexist with it.
  2. What is the best definition of “engine”? This definition seems more consistent with other sources I was able to find online: “An ‘engine’ is a self-contained, but externally-controllable, piece of code that encapsulates powerful logic designed to perform a specific type of work” (which to me aligns much more with the settlement system integration than its orchestration logic!)
• “Adapter” sounds boring and “settlement engine” sounds pretty awesome

(i.e. we should call the settlement system integration a “settlement engine” rather than “ledger adapter”)

[seanrowan]

I’m currently working through the RFC’s and codebase to fully understand the existing system and the current design progression to the settlement engine. I have a few questions that may have obvious answers, but I just want to be certain that I’m understanding things correctly:

• Is a unique settlement engine expected to exist between every pair of connectors that do business with each other? Or is the settlement engine expected to leverage many network participants to contribute to the safety and liveness of a global settlement engine state that many connectors can leverage? And in either case, why is that the current thinking?
• If it’s the latter, what is the current thought about how this system could achieve Byzantine fault tolerance?
• Also if it’s the latter, what is the thinking about how this system can avoid the Sybil attack?

Thank you for reading my questions.

[ekrenzke]

Awesome work so far, @adrianhopebailie. I am glad this is getting more attention, because the previous plug-in architecture has been quite frustrating to work with at times.

This SHOULD be in the form of an ILP packet addressed to peer.settle.* hence I have proposed that this would be “injected” by the Settlement Engine into the outgoing middleware pipeline but really this is up to implementors.

That’s pretty slick.

I see the difference between the two proposals as the question of where the “settlement engine” component should exist (but not whether it’s an external service or an internal service). Does it exist between the balance service and the “ledger adapter,” interfacing between the two, or does its logic exist within the “ledger adapter” itself?

I believe the ledger adapter just exposes generic functionality to the ledger from the settlement engine in this proposal. The role of the ledger adapter is to provide a mapping from settlement logic (e.g. paying or requesting payment for outstanding debt obligations at an arbitrary time) to the specific-ledger functionality (e.g. signing base-layer transactions or handling payment channel claims). In other words, the settlement engine is used for maintaining relationships on the network with generic business logic operations, but the ledger adapter abstracts all of the low-level ledger-specific operations away.

Suppose we form a connection and issue a bunch of trades back and forth. You have effectively credited me 0.01 BTC in the process, which I must pay at settlement time. The terms of our credit relationship say that we must settle any time the credited amount exceeds 0.01 BTC, or approximately every hour – whichever comes first.

If I do not fulfill my debt obligation, then you start sending an appropriate reject message whenever I try to send a payment through the connection we share. The settlement engine allows us to (independently?) define business logic. In this example, I know why you are unwilling to route my packets – we have to settle up. My settlement engine uses the ledger adapter to construct and broadcast a Bitcoin transaction, and then your settlement engine uses the ledger adapter for confirmation.

The settlement engine allows us to implement business logic based on value and time without worrying about ledger-specific functionality, afforded by the adapter.

If it has very complicated ledger-specific logic around triggering settlement, per what you and Matt were suggesting, then does it have to be reimplemented for different “ledger adapters”?

For a minimal ledger adapter, I do not think so. A minimal implementation would probably only need to include sending money, checking balance, and possibly sending data. If we think of a ledger as a language, the ledger adapter is acting as the Rosetta Stone for business logic within the settlement engine. The business logic within the settlement engine is effectively a domain-specific language that can be extended to include ledger-specific functionality.

One thing I’m unclear about: what is the “settlement engine” doing?

The settlement engine is used for handling out-standing debt obligations. It serves as the conditional logic for packet forwarding, based on the credit relationships of incoming and outgoing connections. It is the semantic framework for packet forwarding, whereas the syntactic framework (e.g. packet malformation, fulfillment checks, octet encoding rules) is carried out by the base protocol.

If it has very complicated ledger-specific logic around triggering settlement, per what you and Matt were suggesting, then does it have to be reimplemented for different “ledger adapters”?

In most cases, I think the ledger adapter simply provides an interface for value and time. Hypothetically, it could be extended to build business logic using ledger-specific events (e.g. EVM or block height), but I believe this is a non-goal for minimal adapter implementations.

Also, could you provide specific examples of what logic you see the settlement engine might implement in both simple use cases and more advanced use cases? I think it might help better isolate what problem we’re trying to solve

Simple:

• Time-based Settlement (e.g. hourly )
• Value-based Settlement (e.g. credit threshold )

Advanced:

• Stop-Limit Order-based Settlement (e.g. price oracle )
• Event-sourced Settlement (e.g. EVM, block height )
• Demand-based Settlement (e.g. discounted settlement triggered by insufficient liquidity )

Please feel free to correct my interpretation. (@adrianhopebailie, @emschwartz)

[emschwartz]

@seanrowan the “settlement system” or ledger is the possibly global or at least multi-party thing that tracks a lot of different participants’ balances (think XRP, Bitcoin, etc). Each of those works in a completely different way, which makes integrating with many of them difficult.

We need some abstraction for those different settlement systems and the key questions are:

1. Does the settlement abstraction live in the same process as the connector (plugins), on its own (@adrianhopebailie called this the “ledger adapter”, though @kincaid disagrees with that term), or in the same process with the logic that dictates when to settle based on the ILP credit balance
2. What is the API for that abstraction?

We would like to be able to reuse the same abstraction component for the different implementations of Interledger, which means we need to agree on how it works and what the interface is.

To be clear though, the “settlement engine” / “ledger adapter” is a component that would be run by a single party and would manage their money on the underlying settlement system. Byzantine fault tolerance may be a feature of the underlying settlement system, but not the engine/adapter/abstraction on top of it.

[austin_king]

Thanks @adrianhopebailie @emschwartz @kincaid for all of the input and discussion, appreciate you guys moving this forward. I wanted to offer up my perspective given what Strata is seeing in the network today in hopes that giving some main net context could help us choose the best solution.

For the more complex, business level relationships that we are helping coordinate, the “business logic” for settlement is still very simple. The settleThreshold configuration is totally sufficient for the present time, and we anticipate it will not be an issue for a very long time to come. This makes me think that if this is the case for businesses that are handling higher volume between one another, that it is likely also the case for an open source implementation of the connector that we want to make available for others to run.

Given this, I think that designing the settlement engine as an entire stand alone component is an over optimization for the current time. I’m not strongly opinionated about where it should exist (in the connector vs. in the ledger adapter) but I think that designing the settlement engine as a stand alone piece introduces unnecessary complexity.

Happy to discuss further, and thanks again for all of the time you guys have spent developing this crucial part of the ecosystem.

[adrianhopebailie]

Based on comments from @ekrenzke, @kincaid and @austin_king both here and on the other Ledger Adapter Interface thread, I think we should not try to abstract the Settlement Engine -> Ledger Adaptor interface.

1. It’s complex and the complexity is probably not warranted at this time
2. It’s hard to get the abstraction right
3. Worst case scenario is that some code is re-used between different ledger specific settlement engines and this is extracted into stand-alone libraries

As I said on that thread I’d advocate for standardising on the interface I proposed above as:

Settlement Engine -> Connector

• getBalance (get the current balance of the clearing account, i.e. ILP balance)
• setBalance (update the current balance of the clearing account, apply a delta)
• sendData (send a message to the Settlement Engine on the peer)
• receiveData (accept a message from the Settlement Engine on the other peer)

This can be optimised (now or when it’s deemed to be required) by supplementing getBalance with a subscribeToBalanceChanges that allows the SE to get a stream of balance updates from the connector instead of polling.

I am confident that concerns about the volume of updates that will be sent can be alleviated using a debounced stream however I think this is a premature optimisation.

[adrianhopebailie]

Addressing some comments from @emschwartz in another thread:

Ledger Adapter API

@kincaid brought up an important security consideration about the balance stream API: before the settlement engine would trigger a settlement, it would need to first make an API call to the connector to “claim” or set aside whatever portion of the balance it was going to try to settle, and wait until that API call was resolved before settling. If you don’t, the account holder could send ILP packets that would be deducted from the balance at the same time that the SE is sending an outgoing settlement, thus getting paid out twice what they owe. It seems more straightforward to have the connector check whether it needs to settle, deduct the balance, and then send a send money call to the external system.

This applies however you do settlement. In fact it happens today in effect when you do a sendMoney, adjust the ILP balance and then the settlement fails.

Settlement is orchestration of atomic state changes to different systems so there is always a risk that one change happens before the other or that you need to roll-back changes on one to cover that they failed on the other.

The safest way to do this is to put the liquidity on hold at the connector, attempt the settlement and then release it if the settlement fails.

Ledger Adapter API

Too many and too few messages. Using a debounced stream would mean that the connector sends many completely unnecessary messages to the external system (i.e. when it does not need to settle) and too few messages if the account experiences a sudden burst of activity. If the connector instead send messages only when the account actually needs to settle, it would send no messages when the external system doesn’t need to take any action and it would send more in quick succession if the account has a lot of activity and needs to settle more than once.

This is not really an argument agains the interface design, it’s an argument against moving the settlement business logic out of the connector because it’s too difficult to get consensus on the interface design.

This is a fair argument and I can see that it may be the pragmatic way forward. We have implemented the settlement logic in a different component to the routing/balance checking logic in Rafiki so that we have the flexibility to scale these independently in time. We imagine running a single settlement engine per settlement system that is designed to be robust, stateful and always up so it can do things like watching settlement ledger for closing channels and incoming payments in contrast to the connector which will be stateless and can scale quickly to serve single or multiple peers.

Right now most Rafiki instances will run the connector and SE in the same process but at least by splitting this out logically now we can split them into their own process in future.

Ledger Adapter API

If the balance logic lives in the external shared component, that means that we will have a drawn out debate like this one every time we need to change it. If we keep the logic out of this shared component, we can change the logic the connector (or business logic thing that’s attached to the connector) uses to determine when it should call send money whenever and however we want.

We had envisioned this differently actually. We’ll continue to build out our SE implementation and if anyone wants to use it they can simply stream balance updates to it. Our expectation was that we’d develop an SE for a single settlement system and then re-use non-ledger specific code for others.

We MIGHT try to separate out the SE and Ledger Adaptor as proposed in that thread but probably not for now.

Ledger Adapter API

The balance stream API and logic behind it is more different from what’s already implemented in the plugins and so would take more work to convert the existing integrations.

I wouldn’t bet on any of this being a small amount of work unless we just stick with the plugins as they are

[kincaid]

It’s a game of telephone!

My point to Evan was that in order to do it safely, the external balance service approach (even assuming it uses a debounced stream), requires a lot more messages.

1. Connector sends balance update to balance service
2. Balance service notifies settlement engine of update
3. Settlement engine decides to settle, then sends message to balance service with new balance
4. Balance service ACKs that update, which is important because the SE can’t safely settle until it gets the ACK

(1.5 roundtrips, excluding connector -> balance service)

Whereas, with the sendMoney/receiveMoney approach:

1. Connector update balances, decides to settle, then sends sendMoney message to settlement engine. The SE can safely start settling immediately.

(0.5 roundtrips)

This applies however you do settlement. In fact it happens today in effect when you do a sendMoney, adjust the ILP balance and then the settlement fails.

Not with the ETH plugin, Lightning plugin, or Kava’s XRP plugin!

They all persist a “payoutAmount” for failed settlements, and will retry it later.

I wouldn’t bet on any of this being a small amount of work unless we just stick with the plugins as they are

Which…kinda?

We already migrated our XRP payment channel plugin and added a sendMoney/receiveMoney HTTP API: https://github.com/Kava-Labs/ilp-plugin-xrp-paychan/tree/ko-http-api (refer to tests)

On the Rust connector side, since most of the balance logic is already implemented, I imagine interfacing with this should be relatively straightforward.

(I would emphasize that this branch is something I just threw together for testing, but it should do the thing, allowing two peers to open channels, and send and receive paychan claims, using whatever transport the connector provides).

The larger amount of work, as we see it, is getting these settlement engines (sorry Adrian) to a production-grade state for connector operators. Right now, they’re mostly a blackbox: there’s either no admin management interface, or no management interface that works reliably. That means no way to get your money out, and certainly no good way to understand what’s actually happening. And then ultimately, building out more robust automatic liquidity management.

Switch is more or less a much simplified “admin management” interface for end users. But building the same thing for connector operators will probably be more challenging

[adrianhopebailie]

They all persist a “payoutAmount” for failed settlements, and will retry it later.

So the balance in the connector and the settled amount get out of sync. That’s just a way of reducing the number of messages required to stay in sync by allowing the synchronization to be delayed isn’t it?

[kincaid]

So the balance in the connector and the settled amount get out of sync. That’s just a way of reducing the number of messages required to stay in sync by allowing the synchronization to be delayed isn’t it?

Yes, they get temporarily out of sync.

AFAIK, the only real operational disadvantage of that is after a sendMoney call, if the settlement fails or otherwise is not sent immediately, that owed balance cannot be netted against incoming packets. In your approach, it could be refunded to the balance shared with the connector, enabling it to be netted.

I think that’s a reasonable tradeoff for the ease of implementation, lesser “API surface” required to get consensus on, and other benefits, but it’s worth digging into further.

[adrianhopebailie]

I think that’s a reasonable tradeoff for the ease of implementation, lesser “API surface” required to get consensus on, and other benefits, but it’s worth digging into further.

Don’t disagree at all. As I’ve said before we should just be clear when we make a design decision what the trade-offs are. This is really helpful, especially in the future, when we or others want to re-assess the design or understand why we took the path we did.

[sappenin]

So where does that leave us? Reading through both this thread and the discussion in Ledger Adaptor API, it’s unclear to me where there’s consensus and where there is not.

Here’s how things appear to stand from may vantage point (I’m guessing I’ve missed a few things, or am possibly simply wrong in certain areas, so please chime-in if you disagree):

Consensus

1. Settlement Adaptor: We want a non-Connector process that can interact with an underlying settlement system (e.g., Bitcoin, XRPL, etc) that does not run in the Connector process. This will allow us to scale Connectors and Settlement systems independently. I’m calling this thing the Settlement Adaptor (shout-out to @kincaid for good rationale around why to drop the word “ledger” here).
2. Settlement Engine: We all agree that ILP-account balance-tracking (i.e., “when to settle based on the ILP credit balance”) should live in a Settlement Engine, but for now this component will just be an internal service/function of each Connector implementation.
   1. Based upon how implementations get built, it may make sense in the future to extract this into a standalone system and/or shared-libraries, but this is out of scope for now.
3. Connector/SE <-> SA Transport: To aid adoption and for simplicity, we’re all OK with the Connector/SE communicating with the Settlement Adaptor using an HTTP+JSON API as a first-attempt. We may explore other techniques later if it makes sense.
4. Settlement Adaptor API Endpoints
   1. sendMoney: When called by an external system (e.g., a Connector), instructs the Settlement Adaptor to send a payment via the underlying settlement system (e.g., XRPL).
   2. getBalance: Returns the current balance of the underlying settlement account (i.e. available liquidity for settlements).
   3. sendData: When called by an external system (e.g., a Connector), allows an ILP packet to be transmitted to the Settlement Adaptor for further processing, such as to exchange a Payment Channel claim.
5. Connector/SE API Endpoints
   1. sendData: Allows the local Settlement Adaptor to communicate with the other Settlement Adaptor (used by the peer Connector) by transiting the local Connector using ILP link-layer technologies and peer.settle. addresses.
   2. sendMoney: Allows the Settlement Adaptor to indicate that a settlement payment was received on the underlying settlement system. For example, the account 123 received 10 XRP. The Connector will react to this callback and update its balances appropriately.

Open Questions

1. Are the above API endpoints correct?
2. If you get into a taxi cab, and ask the driver to drive backwards to your destination, will the cab driver owe you money?

[adrianhopebailie]

I think you have it close enough for us to just get on and build it!

Where I think we are not entirely in agreement (but this is immaterial if we put the SE in the connector) is on where the liquidity management logic should sit.

@kincaid suggests that it is too hard to abstract this so it should live in the Settlement Adaptor.

I don’t agree that it’s impossible and think it should be in the SE but agree that abstracting it now may be adding unnecessary complexity to the SA interface.

My main concern is that we are splitting two pieces of business logic that I think belong together, the settlement logic (when should I settle) and the liquidity management (how much liquidity have I got available for settlements) should both sit in the SE.

The current proposal puts the settlement logic in the connector and liquidity management in the SA.

Are the above API endpoints correct?

Correct enough to start with in my opinion.

If you get into a taxi cab, and ask the driver to drive backwards to your destination, will the cab driver owe you money?

So, let’s just be clear on the assumptions at play here…

[adrianhopebailie]

Conclusion

Thanks to all who contributed to this and joined the Community Call on 15 May to discuss. Here is my summary of the outcome (thanks @sappenin for your notes in Slack, they helped jog my memory):

Components

• Connector Balance Logic:
  Logic in the connector that decides whether or not to forward packets based on the Interledger balance of a peer.
  Adjusts Interledger balance based on exchange of packets with peer.

• Accounting System:
  Tracks the Interledger balance and determines when to make a settlement.
  Communicates with the settlement engine and adjusts the Interledger balance based on settlement events.

• Settlement Engine:
  Performs settlements on a settlement system.
  Manages liquidity in the settlement system.
  Communicates with the settlement engines at peers using a transport provided by the connector

Design Decisions

What is the Settlement Engine?

The terminology we came to consensus on is not exactly what was described earlier in this thread so take not of the differences. The reasons are less important than the results, it’s just terminology after all.

Notably we have agreed that the system that runs outside the connector and interfaces with the settlement system is called the Settlement Engine and this DOES NOT contain the logic for deciding WHEN to settle (which is in the accounting system).

In effect this consists of two components, a settlement system liquidity manager (manages the available liquidity for settlement) and a settlement system adaptor (interfaces with the settlement system to, primarily, perform settlements).

Liquidity management of Settlement System is handled in the Settlement Engine

We have settled (pun intended) on a design that does not attempt to abstract the liquidity management functions of the Settlement Engine out of that component as the majority view was that this is quite specific to the underlying settlement system and difficult to abstract.

As such, the Accounting System has no view of the connector’s liquidity in the settlement system. Instead the AS simply requests a settlement and assumes this will be done eventually if the request is acknowledged by the SE.

The AS adjusts its own records to this effect and, in the case that the SE is unable to complete the settlement, the connector may find that packets forwarded to the peer start to be rejected due to a lack of liquidity.

This was accepted as the lowest risk compromise for the sake of keeping the design simple and reducing the API surface between the AS and the SE.

Management of this scenario is then left to operators to ensure they can respond to this eventuality. In reality, this will manifest as failed settlements which should be handled with high priority by an operator anyway.

The alternative to this proposal was to include the liquidity management logic in the accounting system which has the benefit of allowing the accounting system to develop more sophisticated logic around management of liquidity in the settlement system and in the Interledger accounts however this comes at the cost of needing to abstract liquidity management functions.

In time, data from the settlement process (the time taken and rate of success performing settlements) may be useful to the AS in planning future settlements and we may extend the API to enable this.

REST

For simplicity we have agreed to use a REST and JSON based API

Architecture

We have agreed that the standard API we want to define is between the Accounting System (which will likely run inside most connectors for now) and the Settlement Engine.

Both sides must run an HTTP server to accept requests as the interface is bilateral.

All interactions follow a request/reply pattern however some replies will be responses and others will simply be ACKs.

Settlement Engines have an implicit asset code and scale (i.e. a different SE is required for different assets) so the AS must apply any scale conversions before calling the SE API or when receiving API calls from the SE.

The Accounting System (Connector) must host the following interface:

• ReceiveSettlement(accountId, amount)
  Notification from the settlement system that an incoming settlement has been received.
  The response to this is an ACK.
  The AS will ACK the request and adjust the Interledger balance of the peer accordingly.
  The SE should retry until this is ACK’d so it needs to be idempotent.

• SendRequest(accountId, message)
  Deliver a message to the peer’s settlement system and return the response.
  The response is the response from the peer’s settlement system.
  If this is used for some form of RPC between settlement systems then they should define that protocol separately (i.e. how to handle failures and errors).
  An error response from the other SE should not be carried in an error at the API level.
  (i.e. The response is always a 2XX)
  The SE should retry until it gets a response so this needs to be idempotent.

Both calls can return an “Unknown Account Id” response (Maybe a 404?)

The Settlement Engine must host the following interface:

• CreateAccount(accountId)
  Be prepared to send and receive settlements for this account.
  Connector provides the accountId and SE .

• SendSettlement(accountId, amount)
  Perform an outgoing settlement on the underlying settlement system.
  The response to this is an ACK.
  The SE will ACK the request and then attempt to perform the settlement. It will continue attempting the settlement until it succeeds (i.e. There is no mechanism to notify the AS to reverse the balance change)
  The AS should retry until it gets a response so this needs to be idempotent.

• ReceiveRequest(accountId, message)
  Accept a message from the peer’s settlement system and return a response.
  The SE should never return an error response unless the accountId is unknown.
  (i.e. The response is always a 2XX even if the response is carrying an error message back to the peer’s SE)
  The AS should retry until it gets a response so this needs to be idempotent.

Discussion on how to spec this out is being done in the #settlement channel on Slack

Kava have an implementation that is already pretty close to this: https://github.com/Kava-Labs/ilp-plugin-xrp-paychan

UPDATE: Added amount as a param to ReceiveSettlement. This was a mistake in the original post.