In the JS implementation, there is a “middleware” called deduplicate.ts that creates a cache of incoming packets and (occasionally) returns a promise for the result of a previously encountered packet instead of processing the actual packet.
What is the theoretical rationale for this middleware? Is it intending to protect the sender? Or is this middleware actually meant to protect the Connector itself? Some other purpose? (It seems like the current design of ILP is meant to tolerate the occasional duplicate/lost packet…)
Does anyone have experience with this middleware in production to know:
- If this functionality worth the overhead? I would expect it to be rarely useful (duplicate packets should be rare), but the cache inside of this middleware consumes memory on every packet call.
- Is the performance hit very noticable? I would expect it to be a moderate hit because there is a fair amount of byte-array parsing and a hash-per-packet.
This middleware seems to “step on” identical packets that have larger amounts and/or expiries, which is surprising. Is there any reason that an otherwise identical 2nd Prepare packet having a larger amount than the previous one shouldn’t be treated as a different packet?
@sentientwaffle Thanks, that’s really helpful.
- The original motivation for this middleware was simply: If you get two packets with the same condition, there is no reason to forward both of them because you’re already going to get the fulfillment for the first one, which will allow you to fulfill them both and claim the money for both packets. The connector pays once but gets paid twice. Seemed like a sensible optimization. I couldn’t really think of a situation where anyone would ever send two packets with the same condition but it still seemed that if that situation did occur, a connector would want to take advantage of it.
And yes, as DJ mentioned, it had a nice side benefit that packets can never go in a loop because the second time they reach a given connector, they would be filtered out.
- Sticking with the “game theory” viewpoint: If somebody sends you a packet and then sends you another packet with the same condition but either a higher amount or a later expiry date, you could drop the second one hoping that the first one would succeed. But maybe a packet with a higher amount or a later expiry would have a better chance of succeeding. So the rationale was: If this whole thing is a minor selfish little optimization on behalf of the connector, then we want to make sure we at least don’t make any packets fail because of it. So in the case where the second packet may have a better chance of success than the first one, we still forward it.
So that’s why the deduplicate middleware exists and why it works the way it does.
Now let me address the question whether this was a good idea in hindsight.
The cascading expiry time (each connector subtracts one second from the expiry) is entirely sufficient for loop prevention imho, so there is no reason to use the deduplicate middleware.
Thanks @justmoon that’s really helpful, especially the theory behind the feature.