JS Connector Middleware: `deduplicate.ts`

In the JS implementation, there is a “middleware” called deduplicate.ts that creates a cache of incoming packets and (occasionally) returns a promise for the result of a previously encountered packet instead of processing the actual packet.

  1. What is the theoretical rationale for this middleware? Is it intending to protect the sender? Or is this middleware actually meant to protect the Connector itself? Some other purpose? (It seems like the current design of ILP is meant to tolerate the occasional duplicate/lost packet…)

  2. Does anyone have experience with this middleware in production to know:

    1. If this functionality worth the overhead? I would expect it to be rarely useful (duplicate packets should be rare), but the cache inside of this middleware consumes memory on every packet call.
    2. Is the performance hit very noticable? I would expect it to be a moderate hit because there is a fair amount of byte-array parsing and a hash-per-packet.
  3. This middleware seems to “step on” identical packets that have larger amounts and/or expiries, which is surprising. Is there any reason that an otherwise identical 2nd Prepare packet having a larger amount than the previous one shouldn’t be treated as a different packet?

1 Like
  1. I believe the original rationale for the deduplicate middleware was preventing routing loops.
  2. The performance hit is noticeable (more the cache’s memory usage than the hashing). I would recommend disabling the middleware in production. It will probably be removed from ilp-connector at some point.
  3. I don’t know about this. My guess is that it accounts for acceptable variation in path/rate of the duplicate prepare.
4 Likes

@sentientwaffle Thanks, that’s really helpful.

@sappenin

  1. The original motivation for this middleware was simply: If you get two packets with the same condition, there is no reason to forward both of them because you’re already going to get the fulfillment for the first one, which will allow you to fulfill them both and claim the money for both packets. The connector pays once but gets paid twice. Seemed like a sensible optimization. I couldn’t really think of a situation where anyone would ever send two packets with the same condition but it still seemed that if that situation did occur, a connector would want to take advantage of it.

And yes, as DJ mentioned, it had a nice side benefit that packets can never go in a loop because the second time they reach a given connector, they would be filtered out.

  1. Sticking with the “game theory” viewpoint: If somebody sends you a packet and then sends you another packet with the same condition but either a higher amount or a later expiry date, you could drop the second one hoping that the first one would succeed. But maybe a packet with a higher amount or a later expiry would have a better chance of succeeding. So the rationale was: If this whole thing is a minor selfish little optimization on behalf of the connector, then we want to make sure we at least don’t make any packets fail because of it. So in the case where the second packet may have a better chance of success than the first one, we still forward it.

So that’s why the deduplicate middleware exists and why it works the way it does.

Now let me address the question whether this was a good idea in hindsight.

While true that theoretically any connector would have an incentive to deduplicate, in practice it turns out that at least in the JavaScript implementation, the deduplicate middleware is inefficient enough to have a net negative value. It should be disabled by default and it should be turned on by no one.

The cascading expiry time (each connector subtracts one second from the expiry) is entirely sufficient for loop prevention imho, so there is no reason to use the deduplicate middleware.

3 Likes

Thanks @justmoon that’s really helpful, especially the theory behind the feature.