Formalize the use of the `peer.` address space (define some addressing best practice)

Protocol Design
1

From @adrianhopebailie on Fri Nov 16 2018 13:16:27 GMT+0000 (UTC)

We have started to use peer. addresses for various peer protocols.

Let’s use this thread to capture some ideas on formalizing this and then we can update the address spec if necessary. IL-RFC 15 currently defines this address space as:

“Addresses for exchange of packets only with a direct peer. Connectors MUST NOT forward packets with peer. addresses. Packets exchanged between peers to pass routing and config information will use peer. addresses.”

One of the great features of the ILP address space is the extensible nature and (almost) infinite sub-addressing that is possible. This means we can do convenient things like add a sub-address space per application or protocol and then still further divide the space up for per session or per transaction addresses.

So, one could take the peer.config address space and say that packets addressed to that address are directed at the ‘config’ module/application on the peer.

We could (should?) go further and be more specific. E.g. to describe the logical service that a message is addressed to, like: peer.router.config

Or describe the sub-protocol being exchanged via ILP: peer.ccp.config

We could also add some specific suffixes to allow more efficient filtering, like: peer.router.config~20c6e98c-b5ff-4bdc-aca2-85541c694907

Another thought is to define a standard for using the condition and fulfill fields to provide some message validation between peers rather than simply using hard-coded values. Given that peer. packets never traverse the network this should have no impact.

As @sappenin has pointed out in the past, it’s not clear why we do some direct peer messaging at the “ledger” layer (i.e. via BTP) and some at the ILP layer (i.e. using peer. addressed ILP packets.

Should we encourage both? Are the use cases different enough?

One advantage to doing everything in ILP packets is that we dramatically simplify the bi-lateral transport layer requirements.

We still have a single reference connector implementation (that is used widely) so changes are possible now without much ecosystem impact but that window will soon pass. I feel like we’ve done enough experimenting to look at the various use cases and start extracting abstractions that can be re-used (enforced) going forward.

Please provide your thoughts here!


Copied from original issue: Formalize the use of the `peer.` address space (define some addressing best practice) · Issue #496 · interledger/rfcs · GitHub

2

From @adrianhopebailie on Mon Dec 03 2018 02:42:47 GMT+0000 (UTC)

Related to #484 I suggest we can do away with the special Transfer message type and simply use an ILP prepare packet addressed to peer.settlement.

As previously suggested by @sappenin we have a mix of peer-to-peer messages using peer.* addressed ILP packets and transport layer sub-protocols.

In looking at designs for a new, faster simpler alternative to BTP it seems simplifying the transport to ONLY send ILP packets is a significant improvement.

  1. The request-response semantic is built into ILP so there is no need to define message types at the ledger layer
  2. ILP packets already have an expiry so there is no need to deal with this separately in the ledger layer either.
1 Like
3

From @adrianhopebailie on Mon Dec 03 2018 02:44:59 GMT+0000 (UTC)

Building on this, it would be trivial to switch BTP implementations to do an auth handshake using ILP packets addressed to peer.auth.

This should only be needed if the underlying transport is no already considered authenticated.

4

From @kincaidoneil on Mon Dec 03 2018 12:58:21 GMT+0000 (UTC)

Would this do away with the BTP packet format entirely and just use HTTP for request-response correlation, per https://github.com/emschwartz/ilp-rs/issues/26? (at least between connectors)

5

From @adrianhopebailie on Mon Dec 03 2018 13:48:45 GMT+0000 (UTC)

> Would this do away with the BTP packet format entirely and just use HTTP for request-response correlation, per emschwartz/ilp-rs#26? (at least between connectors)

It would mean you COULD do that. Whatever protocol you use to exchange ILP packets could assume that it ONLY exchanges ILP packets which significantly simplifies the protocol.

I think HTTP may still be too much overhead for some use cases but it would mean you could use a simpler TCP/UDP/WebSockets based protocol too

6

From @emschwartz on Mon Dec 03 2018 16:42:52 GMT+0000 (UTC)

I’m not a fan of the idea of using ILP packets to send settlement-related messages like payment channel claims. ILP is built on your ability to settle, rather than settlements being built on ILP.

I think the idea behind using ILP packets for ILDCP and route broadcasts is that you might want those to be paid for using ILP. With settlements, you’re doing the opposite of paying for those messages (but we also wouldn’t have a good way to express that in an ILP packet because they can’t carry negative amounts).

7

From @adrianhopebailie on Mon Dec 03 2018 19:15:42 GMT+0000 (UTC)

> I’m not a fan of the idea of using ILP packets to send settlement-related messages like payment channel claims.

The purpose of the messages isn’t really relevant. What is relevant is which subsystems are producing and handling them.

It makes as much sense to have a settlement subsystem as it does to have an ILDCP subsystem.

ILP is built on your ability to settle, rather than settlements being built on ILP.

ILP makes no assumptions about you ability to settle. Settlements are a bilateral concern.

Anything sent in peer.* addressed packets is, by definition, not built on ILP because those packets are always bilateral. i.e. Sending messages between peers is exactly what that address-space is for.

I was also uncertain about it initially, but if you look at how clean it makes the ledger layer protocol it makes sense.

8

From @kincaidoneil on Mon Dec 03 2018 21:19:29 GMT+0000 (UTC)

What are the opinions on using protobufs for settlement-related messages? How does the performance of protobufs compare to ILP packet or BTP + ILP packet serialization? (e.g. if plugins defined their own ledger-specific gRPC services. I’m thinking along the lines of GitHub - interledgerjs/ilp-transport-grpc: gRPC-based protocol for exchanging ILP packets, but without any standardized framing)

I think HTTP may still be too much overhead for some use cases but it would mean you could use a simpler TCP/UDP/WebSockets based protocol too

How does the overhead of HTTP/2 streaming compare to websocket messages?

I ask about HTTP because on clients that are stuck using websockets, the performance of serializing ILP packets inside of BTP packets (for settlement messages) might actually be worse than straight BTP packets (though I haven’t done any benchmarking).

9

From @sappenin on Tue Dec 04 2018 18:31:48 GMT+0000 (UTC)

> How does the performance of protobufs compare to ILP packet or BTP + ILP packet serialization?

This doesn’t specifically address HTTP/2 vs Websockets messages, but I’ve noticed a lot of redundant encoding/decoding while using ILP+BTP+GRPC.

I think if gRPC is going to be employed at the bilateral layer, we’ll want to explore representing ILP packet data natively in protobuf as opposed to trying to encode an ILP packet inside of a BTP packet inside of a ByteString.

10

From @adrianhopebailie on Tue Dec 04 2018 18:41:55 GMT+0000 (UTC)

> What are the opinions on using protobufs for settlement-related messages?

I think that is something that is outside the scope of what we need to standardize. Part of the motivation for this change is that the standard is simpler. i.e. Just exchange ILP packets.

If you want to exchange bilateral messages in a different way for other functions (e.g. settlement) you can but that would mean you need to get others to implement that too to be interoperable.

E.g. You could decide that there is NO bilateral settlement message. The plugin/middleware could simply trigger settlements as required and the other side simply monitors the settlement system to know when these occur.

Or you could do settlement entirely outside the flow of the transactions where the balance middleware simply enforces balance checking but updates are pushed into it from an external system that settles as required. (Technically possible today by setting very high settlement threshholds and doing balance updates via the Admin API).

I think if gRPC is going to be employed at the bilateral layer, we’ll want to explore representing ILP packet data natively in protobuf as opposed to trying to encode an ILP packet inside of a BTP packet inside of a ByteString.

Agreed. I think if you want to optimize and are exploring other ways of doing bilateral communication there is a gain to be achieved by eliminating layers. Flattening everything out into a single message will be more efficient.

11

From @sharafian on Thu Dec 06 2018 21:58:03 GMT+0000 (UTC)

> > I think if gRPC is going to be employed at the bilateral layer, we’ll want to explore representing ILP packet data natively in protobuf as opposed to trying to encode an ILP packet inside of a BTP packet inside of a ByteString.

Agreed. I think if you want to optimize and are exploring other ways of doing bilateral communication there is a gain to be achieved by eliminating layers. Flattening everything out into a single message will be more efficient.

I don’t see why gRPC deserialization would be faster than deserializing an ILP packet. They’ve probably spent more time fine-tuning their code but protobufs are almost certainly a more complicated format than our OER packets.

Switching websockets to something faster, though, sounds totally reasonable.

12

From @adrianhopebailie on Fri Dec 07 2018 21:28:16 GMT+0000 (UTC)

> I don’t see why gRPC deserialization would be faster than deserializing an ILP packet.

That’s not what I was suggesting. Sorry for not being clear. My point is that the layering we use is probably unnecessary to a large extent.

If you reduced bi-lateral messages to simply an ILP packet plus correlation id then I think you save a lot of serialization effort. I’m working on a PoC of something like this.

I actually think Websockets are pretty fast. They’re a simple framing protocol on top of sockets so they don’t add much overhead. The weird part for a host-to-host connection when using Websockets is the HTTP server you have to host to accept incoming connections.

Technically this could be trimmed down to a very minimal Websocket-handshake-only capable server but it’s still more than you need, especially if you do all the auth after the handshake.

13

From @sappenin on Thu Feb 28 2019 04:40:29 GMT+0000 (UTC)

I don’t think the discussion ever came to a conclusion here - specifically, is there anything we need to standardize here? For example, my Connector implementation currently supports ‘peer.config’, ‘peer.route’ (and eventually ‘peer.settle’) but it seems like IL-RFC-15 already specifies the purpose of the ‘peer.’ allocation scheme - so, is it fair to say that each of the specific usages can be specified in their respective RFCs? (For example, the CCP spec can standardize ‘peer.route’.)

14

From @adrianhopebailie on Thu Feb 28 2019 06:53:13 GMT+0000 (UTC)

> each of the specific usages can be specified in their respective RFCs?

I agree. We should probably do the following before we close this:

  • [ ] Ensure peer.route is defined in the PR that is documenting the routing protocol
  • [ ] Ensure peer.config is properly specified in the ILDCP spec
  • [ ] Document peer.settle (in progress as part of new connector implementation work)
1 Like
15

From @adrianhopebailie on Thu Feb 28 2019 06:54:24 GMT+0000 (UTC)

Also, it is common for a registry to exist that keeps track of these somewhere. should that be IL-RFC 15?

1 Like