Question: multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <del> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </del> 7.1. <del> If the multipath option is negotiated to use one packet number space for all paths, the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control. When a packet is acknowledged, the state of the congestion control MUST be updated for the path where the acknowledged packet was originally sent.  The RTT is calculated based on the delay between the transmission of that packet and its first acknowledgement (see compute-rtt) and is used to update the RTT statistics for the sending path. Also loss detection MUST be adapted to allow for different RTTs on different paths.  For example, timer computations should take into account the RTT of the path on which a packet was sent.  Detections based on packet numbers shall compare a given packet number to the highest packet number received for that path. </del> <ins> If a zero-length connection ID is used, one packet number space for all paths.  That means the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. In this case, ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control, as explained in zero-length-cid-loss-and-congestion. Further, the receiver of packets with zero-length connection IDs should implement handling of acknowledgements as defined in sending- acknowledgements-and-handling-ranges. ECN handing is specified in ecn-handling, and mitigation of the RTT measurement is further explained in ack-delay-and-zero-length-cid- considerations. If a node does not want to implement this logic, it MAY instead limit its use of multiple paths as explained in restricted-sending-to-zero- length-cid-peer. </ins> 7.1.1. <ins> If zero-length CID and therefore also a single packet number space is used by the sender, the receiver MAY send ACK frames instead of ACK_MP frames to reduce overhead as the additional path ID field will anyway always carry the same value. </ins> If senders decide to send packets on paths with different transmission delays, some packets will very likely be received out of order.  This will cause the ACK frames to carry multiple ranges of

This derives from the discussions on issue
The commit applies the changes suggested in the previous reviews. With those changes, I think we have a good basis for discussing a unified solution in the working group.
Thanks NAME for the review. I have applied the editorial changes. You are asking for something more regarding the "specific logic" required when supporting zero-length CID. There are in fact two pieces to that: logic at the receiver, i.e., the node that chose to receive packets with zero-length CID; and logic at the sender, i.e., the node that accepts to send data on multiple paths towards a node that uses zero-length CID. This is explained in details in the section "Using Zero-Length connection ID", so I guess what we need is a reference to that section.
Comment by NAME (on PR ): I think we expand this section a bit as we found getting the delay right was non-trivial. QUIC time-stamp would work for sure. But if one decides not to use QUIC timestamp, we probably want to explain what is needed to be done here.
I have submitted PR for this one.
Closing as got merged.
In section 3.1, we wrote When the multipath option is negotiated, clients that want to use an additional path MUST first initiate the Address Validation procedure with PATHCHALLENGE and PATHRESPONSE frames described in of ]. After receiving packets from the client on the new paths, the servers MAY in turn attempt to validate these paths using the same mechanisms. With multipath, we cannot simply write that the server MAY validate the new path. There is a risk of a malicious client that operates as follows: client creates path from its own address, A. Path is validate by server client starts long download from server client creates a second path to server using a spoofed address, B client starts address validation for B but does not care about the response from the server client stops acknowledging packets over the first path to force the server to move packets to the second one and then opportunistically acknowledges packets that should have been sent by the server over that second packet (SPNS would probably make such an attack simpler than MPNS) We probably need to start working on the security considerations section of the draft.
This is already covered in the QUIC threat model. The server is supposed to validate new paths by sending a "path challenge" on that path, then waiting for a challenge response.


Question: multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <del> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </del> 7.1. <del> If the multipath option is negotiated to use one packet number space for all paths, the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control. When a packet is acknowledged, the state of the congestion control MUST be updated for the path where the acknowledged packet was originally sent.  The RTT is calculated based on the delay between the transmission of that packet and its first acknowledgement (see compute-rtt) and is used to update the RTT statistics for the sending path. Also loss detection MUST be adapted to allow for different RTTs on different paths.  For example, timer computations should take into account the RTT of the path on which a packet was sent.  Detections based on packet numbers shall compare a given packet number to the highest packet number received for that path. </del> <ins> If a zero-length connection ID is used, one packet number space for all paths.  That means the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. In this case, ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control, as explained in zero-length-cid-loss-and-congestion. Further, the receiver of packets with zero-length connection IDs should implement handling of acknowledgements as defined in sending- acknowledgements-and-handling-ranges. ECN handing is specified in ecn-handling, and mitigation of the RTT measurement is further explained in ack-delay-and-zero-length-cid- considerations. If a node does not want to implement this logic, it MAY instead limit its use of multiple paths as explained in restricted-sending-to-zero- length-cid-peer. </ins> 7.1.1. <ins> If zero-length CID and therefore also a single packet number space is used by the sender, the receiver MAY send ACK frames instead of ACK_MP frames to reduce overhead as the additional path ID field will anyway always carry the same value. </ins> If senders decide to send packets on paths with different transmission delays, some packets will very likely be received out of order.  This will cause the ACK frames to carry multiple ranges of

If the client spoofs a path, it will never receive the challenge, and thus the server will never receive the response and will not validate the path. See section 8.2 of RFC 9000, path validation. Security section 21.1.3. Connection Migration states that "Path validation () establishes that a peer is both willing and able to receive packets sent on a particular path."
So yes, the current text should be "the servers MUST in turn attempt to validate these paths using the same mechanisms, as specified in {{ Section 8.2 of RFC9000 }}."
No, this should not be a MUST. It is true that the path MUST be validated before it can be used. However, just because the serve receives a new packet on a path, that doesn't automatically mean that the server has to start path validation. It could e.g. also decide to simply withdraw that packet.
Actually, this is also not right. This is what RFC9000 says (section 9): So I guess we should maybe not use normative language here but refer to section 9 in RFC9000 instead...?
Just submitted PR for this issue.
This draft initially originates from a merging effort of previous Multipath proposals. While many points were found to be common between them, there remains one design point that still requires consensus: the number of packet number spaces that a Multipath QUIC connection should support (i.e., one for the whole connection vs. one per path). The current draft enables experimentation with both variants, but in the final version we will certainly need to choose between one of the versions.
The main issues mentioned so far: Efficiency. One number space per path should be more efficient, because implementations can directly reuse the loss-recovery logic specified for QUIC ACK size. Single space leads to lots of out of order delivery, which causes ACK sizes to grow Simplicity of code. Single space can be implemented without adding mane new code paths to uni-path QUIC Shorter header. Single space works well with NULL length CID The picoquic implementation shows that "efficiency" and "ack size" issues of single space implementations can be mitigated.


Question: multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <del> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </del> 7.1. <del> If the multipath option is negotiated to use one packet number space for all paths, the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control. When a packet is acknowledged, the state of the congestion control MUST be updated for the path where the acknowledged packet was originally sent.  The RTT is calculated based on the delay between the transmission of that packet and its first acknowledgement (see compute-rtt) and is used to update the RTT statistics for the sending path. Also loss detection MUST be adapted to allow for different RTTs on different paths.  For example, timer computations should take into account the RTT of the path on which a packet was sent.  Detections based on packet numbers shall compare a given packet number to the highest packet number received for that path. </del> <ins> If a zero-length connection ID is used, one packet number space for all paths.  That means the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. In this case, ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control, as explained in zero-length-cid-loss-and-congestion. Further, the receiver of packets with zero-length connection IDs should implement handling of acknowledgements as defined in sending- acknowledgements-and-handling-ranges. ECN handing is specified in ecn-handling, and mitigation of the RTT measurement is further explained in ack-delay-and-zero-length-cid- considerations. If a node does not want to implement this logic, it MAY instead limit its use of multiple paths as explained in restricted-sending-to-zero- length-cid-peer. </ins> 7.1.1. <ins> If zero-length CID and therefore also a single packet number space is used by the sender, the receiver MAY send ACK frames instead of ACK_MP frames to reduce overhead as the additional path ID field will anyway always carry the same value. </ins> If senders decide to send packets on paths with different transmission delays, some packets will very likely be received out of order.  This will cause the ACK frames to carry multiple ranges of

However, that required significant improvements in the code: to obtain good loss efficiency, picoquic remembers not just the path on which a packet was sent, but also the order of the packet on this path, i.e. a "virtual sequence number". The loss detection logic then operates on that virtual sequence number. to contain ACK size, picoquic implements a prioritization logic to select the most important ranges in an ACK, avoid acking the same range more than 3 or 4 times, and keep knowledge of already acknowledged ranges so range coalescing works. I think these improvements are good in general, and I will keep them in the implementations whether we go for single space or not. The virtual sequence number is for example useful if the CID changes for reasons not related to path changes in multiple number space variants. It is also useful in unipath variants to avoid interference between sequence numbers used in probes and the RACK logic. The ACK size improvements do reduce the size of ACKs in presence of out of order delivery, e.g., if the network is doing some kind of internal load balancing. On the other hand, the improvements are somewhat complex, would need to be described in separate drafts, and pretty much contradicts the "simplicity of code" argument. So we are left with the "Null length CID" issue. I see for cases: Client CID Priority ----------------------- long Used by many implementations NULL Preferred configuration of many big deployments long Rarely used, server load balancing does not work NULL Only mentioned in some P2P deployments The point here is that it is somewhat hard to deploy a large server with NULL CID and use server load balancing. This configuration is mostly favored by planned P2P deployments.
The big debate is for the configuration with NULL CID on client, long CID on server. The packets from server to client do not carry a CID, and only the last bytes of the sequence number. The client will need some logic to infer the full sequence number before decrypting the packet. The client could maybe use the incoming 5 tuple as part of the logic, but it is not obvious. It is much simpler to assume a direct map from destination CID to number space. That means, if a peer uses a NULL CID, all packets sent to that peer are in the same number space.


Question: multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <del> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </del> 7.1. <del> If the multipath option is negotiated to use one packet number space for all paths, the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control. When a packet is acknowledged, the state of the congestion control MUST be updated for the path where the acknowledged packet was originally sent.  The RTT is calculated based on the delay between the transmission of that packet and its first acknowledgement (see compute-rtt) and is used to update the RTT statistics for the sending path. Also loss detection MUST be adapted to allow for different RTTs on different paths.  For example, timer computations should take into account the RTT of the path on which a packet was sent.  Detections based on packet numbers shall compare a given packet number to the highest packet number received for that path. </del> <ins> If a zero-length connection ID is used, one packet number space for all paths.  That means the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. In this case, ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control, as explained in zero-length-cid-loss-and-congestion. Further, the receiver of packets with zero-length connection IDs should implement handling of acknowledgements as defined in sending- acknowledgements-and-handling-ranges. ECN handing is specified in ecn-handling, and mitigation of the RTT measurement is further explained in ack-delay-and-zero-length-cid- considerations. If a node does not want to implement this logic, it MAY instead limit its use of multiple paths as explained in restricted-sending-to-zero- length-cid-peer. </ins> 7.1.1. <ins> If zero-length CID and therefore also a single packet number space is used by the sender, the receiver MAY send ACK frames instead of ACK_MP frames to reduce overhead as the additional path ID field will anyway always carry the same value. </ins> If senders decide to send packets on paths with different transmission delays, some packets will very likely be received out of order.  This will cause the ACK frames to carry multiple ranges of

Revised table: Client CID What -- -- long Multiple number space NULL Multiple number spaces on client side (one per CID), single space on server side long Multiple number spaces on server side (one per CID), single space on client side NULL single number space on each side If a node advertises both NULL CID and multipath support, they SHOULD have logic to contain the size of ACK. If a node engages in multipath with a NULL CID peer, they SHOULD have special logic to make loss recovery work well.
I think the above points to a possible "unified solution": if multipath negotiated, tie number spaces to destination CID, use default (N=0) if NULL CID. use multipath ack; in multipath ACK, identify number space by sequence number of DCID in received packets. Default that number to zero if received with NULL CID. in ABANDON path, identify path either by DCID of received packets, SCID of sent packet, or "this path" (sending path) if CID is NULL.
I like the proposal of the "unified solution". I think the elegance lies in the fact that it allows us to automatically cover all four cases listed above. The previous dilemma for me was that on one hand we have some use cases where we need to support more than two paths and separate PN makes the job easier, but on the other hand, I think we should not ignore the NULL CID use cases as it is also important. Now, with this proposal, a big part of the problem is solved. The rest of challenge is to make sure single PN remains efficient in terms of ACK and loss recovery. On that part, we plan to do an A/B test and would love to share the results when we get them. There is one more problem as pointed in issue , when we want to take hardware offloads into account. In such a case, we may still need single PN for long server CID. However, if hardware supports nonce modification, this problem can be addressed with the proposed "unified solution".
If the APi does not support 96 bit sequence numbers, it should always be possible to create an encryption context per number space, using Key=current key and ID = current-ID + CID sequence number. Of course, that forces creation of multiple context, and that makes key rotation a bit harder to manage. But still, that should be possible.


Question: multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <del> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </del> 7.1. <del> If the multipath option is negotiated to use one packet number space for all paths, the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control. When a packet is acknowledged, the state of the congestion control MUST be updated for the path where the acknowledged packet was originally sent.  The RTT is calculated based on the delay between the transmission of that packet and its first acknowledgement (see compute-rtt) and is used to update the RTT statistics for the sending path. Also loss detection MUST be adapted to allow for different RTTs on different paths.  For example, timer computations should take into account the RTT of the path on which a packet was sent.  Detections based on packet numbers shall compare a given packet number to the highest packet number received for that path. </del> <ins> If a zero-length connection ID is used, one packet number space for all paths.  That means the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. In this case, ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control, as explained in zero-length-cid-loss-and-congestion. Further, the receiver of packets with zero-length connection IDs should implement handling of acknowledgements as defined in sending- acknowledgements-and-handling-ranges. ECN handing is specified in ecn-handling, and mitigation of the RTT measurement is further explained in ack-delay-and-zero-length-cid- considerations. If a node does not want to implement this logic, it MAY instead limit its use of multiple paths as explained in restricted-sending-to-zero- length-cid-peer. </ins> 7.1.1. <ins> If zero-length CID and therefore also a single packet number space is used by the sender, the receiver MAY send ACK frames instead of ACK_MP frames to reduce overhead as the additional path ID field will anyway always carry the same value. </ins> If senders decide to send packets on paths with different transmission delays, some packets will very likely be received out of order.  This will cause the ACK frames to carry multiple ranges of

Thanks for the summary NAME I think one point is missing in your list which is related to issue . Use of a single packet number space might not support ECN. Regarding the unified solution: I think what you actually say is that we would specify both solutions and everybody would need to implement both logics. At least regarding the "simplicity of code" argument, that would be the worst choice. If we can make the multiple packet number spaces solution work with one-sided CIDs, I'm tending toward such an approach. Use of multiple packet number spaces avoids ambiguity/needed "smartness" in ACK handling and packet scheduling which, as you say above, can make the implementation more complex and, moreover, wrong decisions may have large impact on efficient (both local processing and on-the-wire). I don't think we want to leave these things to each implementation individually.
The summary Christian made above about design comparison sounds indeed quite accurate. Besides ECN, my other concern about single packet number is that it require cleverness from the receiver side if you want to be performant in some scenarios. At the sender-side, you need to consider a path-relative packet number threshold instead of an absolute one to avoid spurious losses. Just a point I think we did not mentioned yet is that there can be some interactions between Multipath out-of-order number delivery and incoming packet number duplicate detection. This requires maintaining some state at the receiver side, as described by URL With single packet number space, the receiver should take extra care when updating the "minimum packet number below which all packets are immediately dropped". Otherwise, in presence of paths with very different latencies, the receiver might end up discarding packets from a (slower) path. I'm also preferring the multiple packet number spaces solution for the above reasons. I'm not against thinking for a "unified" solution (the proposal sounds interesting), but I wonder how much complexity this would add compared to requiring end hosts to use one-byte CIDs.
I think the issue is not really so much "single vs multiple number space" as "support for multipath and NULL CID". As noted by Mirja, there is an implementation cost there. My take is: If a node uses NULL CID and proposes support of multipath, then that node MUST implement code to deal with size of acknowledgements.


Question: multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <del> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </del> 7.1. <del> If the multipath option is negotiated to use one packet number space for all paths, the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control. When a packet is acknowledged, the state of the congestion control MUST be updated for the path where the acknowledged packet was originally sent.  The RTT is calculated based on the delay between the transmission of that packet and its first acknowledgement (see compute-rtt) and is used to update the RTT statistics for the sending path. Also loss detection MUST be adapted to allow for different RTTs on different paths.  For example, timer computations should take into account the RTT of the path on which a packet was sent.  Detections based on packet numbers shall compare a given packet number to the highest packet number received for that path. </del> <ins> If a zero-length connection ID is used, one packet number space for all paths.  That means the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. In this case, ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control, as explained in zero-length-cid-loss-and-congestion. Further, the receiver of packets with zero-length connection IDs should implement handling of acknowledgements as defined in sending- acknowledgements-and-handling-ranges. ECN handing is specified in ecn-handling, and mitigation of the RTT measurement is further explained in ack-delay-and-zero-length-cid- considerations. If a node does not want to implement this logic, it MAY instead limit its use of multiple paths as explained in restricted-sending-to-zero- length-cid-peer. </ins> 7.1.1. <ins> If zero-length CID and therefore also a single packet number space is used by the sender, the receiver MAY send ACK frames instead of ACK_MP frames to reduce overhead as the additional path ID field will anyway always carry the same value. </ins> If senders decide to send packets on paths with different transmission delays, some packets will very likely be received out of order.  This will cause the ACK frames to carry multiple ranges of

If a node sees a peer using NULL CID and supporting multipath, then that node MUST either use only one path at a time or implement code to deal with impact on loss detection due to out of order arrivals, and impact on congestion control including ambiguity of ECN signals. Then add sections on what it means to deal with the side of acknowledgements, out of order arrivals, and congestion control. I think this approach ends up with the correct compromises: It keeps the main case simple. Nodes that rely on multipath just use long CID and one number space per CID. Nodes that use long CID never have to implement the ACK length minimization algorithms. Nodes that see a peer using NULL CID have a variety of implementation choices, e.g., sending on just one path at a time, sending on mostly one path at a time in "make or break" transitions, or implementing sophisticated algorithms. If we do that, we can get rid of the single vs multiple space discussion, and end up with a single solution addressing all use cases.
Looking at all the discussions here and in other issues such as ECN, I think that we should try to write two different versions of section 7: one covering all the aspects of single packet number space (RTT estimation, retransmissions, ECN, handling of CIDs, ...) one covering all the aspects of multiple packet number space (RTT estimation, retransmissions, ECN, handling of CIDs, ...) There would be some overlap between these two sections and also some differences that would become clear as we specify them. At the end of this writing, we'll know whether it is possible to support/unify both or we need to recommend a single one. The other parts of the document are almost independent of that and can evolve in parallel with these two sections. However, I don't think that such a change would be possible by Monday
I don't think we should rush changes before we have agreed on the final vision.
It might be helpful to have these options as PRs (without merging) them, so people can understand all details of each approach.
I agree with NAME that supporting multi-path with null CIDs is a more fundamental issue than the efficiency comparison between single PN and separate PN, as it would ultimately impact the application scope of multipath QUIC. But indeed, we might want to implement proposed solution first and then decide if we want to adopt such a unified approach.


Question: multipath negotiation is ambiguous, they MUST be interpreted as acknowledging packets sent on path 0. <del> Endpoints negotiate the use of one packet number space for all paths or separate packet number spaces per path during the connection handshake nego.  While separate packet number spaces allow for more efficient ACK encoding, especially when paths have highly different latencies, this approach requires the use of a connection ID. Therefore use of a single number space can be beneficial when endpoints use zero-length connection ID for less overhead. </del> 7.1. <del> If the multipath option is negotiated to use one packet number space for all paths, the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control. When a packet is acknowledged, the state of the congestion control MUST be updated for the path where the acknowledged packet was originally sent.  The RTT is calculated based on the delay between the transmission of that packet and its first acknowledgement (see compute-rtt) and is used to update the RTT statistics for the sending path. Also loss detection MUST be adapted to allow for different RTTs on different paths.  For example, timer computations should take into account the RTT of the path on which a packet was sent.  Detections based on packet numbers shall compare a given packet number to the highest packet number received for that path. </del> <ins> If a zero-length connection ID is used, one packet number space for all paths.  That means the packet sequence numbers are allocated from the common number space, so that, for example, packet number N could be sent on one path and packet number N+1 on another. In this case, ACK frames report the numbers of packets that have been received so far, regardless of the path on which they have been received.  That means the senders needs to maintain an association between sent packet numbers and the path over which these packets were sent.  This is necessary to implement per path congestion control, as explained in zero-length-cid-loss-and-congestion. Further, the receiver of packets with zero-length connection IDs should implement handling of acknowledgements as defined in sending- acknowledgements-and-handling-ranges. ECN handing is specified in ecn-handling, and mitigation of the RTT measurement is further explained in ack-delay-and-zero-length-cid- considerations. If a node does not want to implement this logic, it MAY instead limit its use of multiple paths as explained in restricted-sending-to-zero- length-cid-peer. </ins> 7.1.1. <ins> If zero-length CID and therefore also a single packet number space is used by the sender, the receiver MAY send ACK frames instead of ACK_MP frames to reduce overhead as the additional path ID field will anyway always carry the same value. </ins> If senders decide to send packets on paths with different transmission delays, some packets will very likely be received out of order.  This will cause the ACK frames to carry multiple ranges of

Since NAME prodded me, we now have a PR for the "unified" proposal.
I totally agree with Christian that the issue is about "support for multipath and NULL CID", and the solution that Christian suggested looks really great! It both takes advantage of multiple spaces, and support NULL CID users without affect the efficiency of ACK arrangements. Besides, the solution is more convenient for implementations, because If both endpoints uses non-zero length cids, endpoints only need to support multiple spaces, and if one of the endpoints use NULL CID, it could use single pn space in one direction and could support NULL CID and multipath at the same time.
Very high level summary of IETF-113 discussion seems that there is interest and likely support for the unified solution (review minutes for further details).


Question: The endpoint accepting incoming QUIC connections. <ins> When used without qualification, the tuple of IP version, IP address, UDP protocol, and UDP port number that represents one end of a network path. </ins> An opaque identifier that is used to identify a QUIC connection at an endpoint.  Each endpoint sets a value for its peer to include in packets sent towards the endpoint.

Some definitional work for terms that were poorly used. Note that "host" in the remaining uses is strictly interpreted in the RFC 1122 sense, so I left those alone. I considered sorting the terms section differently, and might still alphabetize that, but this isn't problematic just yet.
The word "host" is undefined. If it is equal to "endpoint", it should be replaced. Otherwise, "host" should" be defined in Sec 1.2. The word "address" sometime means "IP address" and "a pair of IP address and port" in other cases. Sec 1.2 should define "address" as "a pair of IP address and port". And "IP" should be added in front of "address" which means "IP address".
fixed this.
Thank you!


Question: VPNs from residential users as they stopped commuting and switched to work-at-home. <ins> Again, it will be helpful for streaming operators to monitor traffic as described in measure-coll, watching for sudden changes in performance. </ins> 4. Streaming media latency refers to the "glass-to-glass" time duration,

Fixed a few typos, otherwise ready to merge.
Merging on Ali's review.
From NAME (URL) (In Section 3.6. Unpredictable Usage Profiles / Section 3.7. Extremely) Unpredictable Usage Profiles I am not fully convinced by the distinction between "unpredictable" and "extremely unpredictable". To me, these two sections could be merged and maybe also be shortened. More specifically, Section 3.7 lists a lot of statistics from the past. What is IMHO a bit missing in Section 3.7 are actual operational considerations. For instance, are there any lessons learnt for the future? Note that Section 3.6 has a reasonable conclusion at the end. Reviewer: Michael Scharf Review result: Ready with Issues This document has been reviewed as part of the transport area review team's ongoing effort to review key IETF documents. These comments were written primarily for the transport area directors, but are copied to the document's authors and WG to allow them to address any issues raised and also to the IETF discussion list for information. When done at the time of IETF Last Call, the authors should consider this review as part of the last-call comments they receive. Please always CC EMAIL if you reply to or forward this review. This informational document surveys network and transport protocol issues that affect quality of experience for streaming applications, in particular video. The overview covers many topics across different technologies. Readers may only be familiar with a subset of these topics and could therefore learn quite a bit from this kind of tutorial. Nonetheless, it remains somewhat unclear what the actual objective of some of the text is. For some topics, quite a bit of technical background is provided, but the discussion is not really comprehensive. In many sections, the document neither derives technical challenges, nor system/protocol requirements, nor practical usage guidance. It is just a kind of tutorial. Quite a bit of text also presents ongoing IETF work, and an RFC with this scope may thus get outdated soon. Section 6.2 deals with topics owned by TCPM. In similar past documents, I have asked for a TCPM presentation prior to an IETF last call in order to ensure that owners of running code are in the loop. I believe this strategy has worked well in the past.


Question: VPNs from residential users as they stopped commuting and switched to work-at-home. <ins> Again, it will be helpful for streaming operators to monitor traffic as described in measure-coll, watching for sudden changes in performance. </ins> 4. Streaming media latency refers to the "glass-to-glass" time duration,

Having said this, as TSV-ART reviewer I don't strongly disagree with a publication. All these issues may just be OK for an informational RFC. Some more specific comments: Section 3.2.1. Recognizing Changes from an Expected Baseline This section apparently assumes relatively static path properties, e.g., fixed network connectivity. It would be more useful to analyze the impact of Wifi and 4G/5G networks in one place. Some of this follows later in sections 3.2.1 and Yet, the split between these three sections is unclear. Having a discussion about today's Internet path characteristics and the dynamics at one place could be more useful. The text could also better distinguish between what matters to endpoints and considerations relevant only for middleboxes (such as passive monitors). Section 3.3 Path Requirements The statement "to find the bandwidth requirements for a router on the delivery path" may assume that the bottleneck on the path will be routers. Yet, on a path the bottlenecks could also be a link layer device (e.g., an Ethernet Switch, a Wifi access point, etc.). RFC 6077 (specifically Section 3.1.3) also explains that issue. A better wording may be "to find the bandwidth requirements for a delivery path". Section 3.6. Unpredictable Usage Profiles / Section 3.7. Extremely Unpredictable Usage Profiles I am not fully convinced by the distinction between "unpredictable" and "extremely unpredictable". To me, these two sections could be merged and maybe also be shortened. More specifically, Section 3.7 lists a lot of statistics from the past. What is IMHO a bit missing in Section 3.7 are actual operational considerations. For instance, are there any lessons learnt for the future? Note that Section 3.6 has a reasonable conclusion at the end. Section 4. Latency Considerations and Section 4.1. Ultra Low-Latency I am surprised by the definition "ultra low-latency (less than 1 second)", as well as some of the other numbers. For other real-time communication use cases, "ultra low-latency" would probably imply a latency requirement of the order of one millisecond.


Question: VPNs from residential users as they stopped commuting and switched to work-at-home. <ins> Again, it will be helpful for streaming operators to monitor traffic as described in measure-coll, watching for sudden changes in performance. </ins> 4. Streaming media latency refers to the "glass-to-glass" time duration,

For instance, isochronous traffic for motion control in industrial networks may require a latency of 1 ms, and such "ultra-low latency" requirements are discussed elsewhere, e.g., in the DetNet WG. The terminology in this section should be better explained to readers dealing with networked systems with much harder latency requirements. And a reference should be added for these definitions in the context of video streaming. Section 5.5.2 Head-of-Line Blocking If Head-of-Line Blocking is indeed a relevant operational problem, it would be useful to add a corresponding reference (e.g., with measurements). Section 5.5.3. Wide and Rapid Variation in Path Capacity The statement "As many end devices have moved to wireless connectivity for the final hop (Wi-Fi, 5G, or LTE), new problems in bandwidth detction have emerged from radio interference and signal strength effects." could be moved to earlier parts of the document. Quite a bit of new computers apparently only have Wifi connectivity and no Ethernet port, i.e., Wifi may just be their default. Also, wireless may not only be used on the last hop, for instance in meshed setups. This may make the problem even harder. Section 6. Evolution of Transport Protocols and Transport Protocol Behaviors I really wonder whether UDP vs. TCP vs. QUIC is actually the relevant distinction. What may actually matter are the transport services provided by the protocol stack (e.g., RFC 8095). I fully agree to a related comment in the INTDIR review. Section 6.1. UDP and Its Behavior UDP is also used for encrypted tunnels (OpenVPN, Wireguard, etc.). Does encrypted tunneling really have no operational impact on streaming apps other than circuit breakers? (I am thinking about 4K streaming over OpenVPN and the like.) Section 6.2. TCP and Its Behavior This text should IMHO be presented and be discussed in TCPM. Personally, I am not convinced that this document is a good place for discussing the long history of TCP congestion control. If the objective of the document is to provide a tutorial, IMHO it would be more useful to briefly explain the state-of-the-art in year 2022.


Question: VPNs from residential users as they stopped commuting and switched to work-at-home. <ins> Again, it will be helpful for streaming operators to monitor traffic as described in measure-coll, watching for sudden changes in performance. </ins> 4. Streaming media latency refers to the "glass-to-glass" time duration,

Some further specific comments on TCP congestion control: It is surprising that RFC 5681 is not referenced.793bis has normative statements regarding congestion control, and most stacks are compatible to what is written in 793bis. Several operating systems use CUBIC by default and support standard Reno. Another general good reference for TCP standards is RFC 7414.Does ECN, DCTCP, etc. really not matter at all, for instance, if a data center hosts video playout servers? Section 6.3. QUIC and Its Behavior As already noted, the discussion about head-of-line-blocking would really benefit from backing by a reference. Section 7. Streaming Encrypted Media As far as I know, streaming users sometimes use encrypted tunnels such as OpenVPN or WireGuard (or IPsec) to access video content. I may miss something, but it is unclear to me how that fits into the categories presented in Section


Question: 1. <del> QUIC RFC9000 has numerous extension points, including the version number that occupies the second through fifth octets of every long header (see RFC8999).  If experimental versions are rare, and QUIC version 1 constitutes the vast majority of QUIC traffic, there is the potential for middleboxes to ossify on the version octets always being 0x00000001. </del> <ins> QUIC QUIC has numerous extension points, including the version number that occupies the second through fifth octets of every long header (see RFC8999).  If experimental versions are rare, and QUIC version 1 constitutes the vast majority of QUIC traffic, there is the potential for middleboxes to ossify on the version octets always being 0x00000001. </ins> Furthermore, version 1 Initial packets are encrypted with keys derived from a universally known salt, which allow observers to

This uses kramdown tricks to make real links for the section references to RFC 9001 and uses a nicer label for those.


Question: 4.1. <del> (To the editor: check this repository URL after the draft is adopted. The working group may create its own repository) </del> <ins> This document is in the Github repository at: </ins> <del> This document is in the Github repository at https://github.com/GrumpyOldTroll/ietf-mops-drafts.  Readers are welcome to open issues and send pull requests for this document. </del> <ins> https://github.com/ietf-wg-mops/draft-ietf-mops-streaming-opcons Readers are welcome to open issues and send pull requests for this document. </ins> Substantial discussion of this document should take place on the MOPS working group mailing list (mops@ietf.org). <ins> Join: https://www.ietf.org/mailman/listinfo/mops Search: https://mailarchive.ietf.org/arch/browse/mops/ </ins> 4.2. Presentations: IETF 105 BOF: https://www.youtube.com/watch?v=4G3YBVmn9Eo&t=47m21s <del> IETF 106 Mops meeting: https://www.youtube.com/ </del> <ins> IETF 106 meeting: https://www.youtube.com/ </ins> watch?v=4_k340xT2jM&t=7m23s <del> List archive search for doc mentions: https://mailarchive.ietf.org/arch/browse/mops/?q=draft-ietf-mops- streaming-opcons </del> 5. This document requires no actions from IANA.

I actually used the URL in the draft to clone a copy, so I think this is good to go.


Question: 4.5.2. If the Record has no Unit, the Base Unit is used as the Unit.  Having <del> no Unit and no Base Unit is allowed. </del> <ins> no Unit and no Base Unit is allowed; any information that may be required about units applicable to the value then needs to be provided by the application context. </ins> 4.5.3.

As per Adam's comment, clarify that only the element() scheme of the XPointer Framework is supported (as that is the one that is required by RFC 7303).
LGTM


Question: defined in Section 6 of TLS13. A TLS alert is converted into a QUIC connection error.  The <del> AlertDescription value is added to 0x100 to produce a QUIC error code from the range reserved for CRYPTO_ERROR.  The resulting value is sent in a QUIC CONNECTION_CLOSE frame of type 0x1c. </del> <ins> AlertDescription value is added to 0x0100 to produce a QUIC error code from the range reserved for CRYPTO_ERROR.  The resulting value is sent in a QUIC CONNECTION_CLOSE frame of type 0x1c. </ins> QUIC is only able to convey an alert level of "fatal".  In TLS 1.3, the only existing uses for the "warning" level are to signal

One possible resolution to ; all literals are zero-padded to full bytes.
IANA made the following observation while creating the H3 registries: While the IANA copy of the registry uses two digits throughout, the QUIC draft is similarly inconsistent about whether literal hex values have any leading zeros. Is it worth updating our hex literals to use full bytes (, , etc.) or synchonizing on some other format? We probably shouldn't synchronize on trimming, since IANA has already skewed the other way.
NAME did fix this or do we need to inform IANA also?
We should probably shoot IANA an e-mail to update the registries with leading zeroes where they hadn't already assumed them.
I'll let you do that. The only registries that are affected are in HTTP/3 and QPACK.
Fixed both in the registry and the documents.
Yeah, this is easy to understand. The tables must have been annoying.


Question: private key.  The mechanism proposed in this document allows the delegation to be done off-line, with no per-transaction latency.  The figure below compares the message flows for these two mechanisms with <del> TLS 1.3 I-D.ietf-tls-tls13. </del> <ins> TLS 1.3 I-D.ietf-tls-tls13, where DC is delegated credentials. </ins> These two mechanisms can be complementary.  A server could use credentials for clients that support them, while using LURK to

Just some nits.
The public key name and type have been changed in , as NAME suggested on the mailing list.
NAME beat me to it!
oh looks like the travis build is failing for some reason
NAME I have no idea why pushing that worked ...
awesomesauce. it's probably some temporary failure while trying to fetch the new references or something


Question: The public key of the account key pair, encoded as a JSON Web Key object RFC7517. <del> "valid" or "deactivated" </del> <ins> The status of this registration.  Possible values are: "valid", "deactivated", and "revoked". "deactivated" should be used to indicate user initiated deactivation whereas "revoked" should be used to indicate administratively initiated deactivation. </ins> An array of URIs that the server can use to contact the client for issues related to this authorization.  For example, the server may

what's the difference between "deactivated" and "revoked"?
Deactivated is done by the user; revoked is done by an administrator.
OK, so we need to either say that or use clearer terms :) I talked with NAME offline, and I think he's going to propose something.


Question: The DTLSCiphertext structure omits the superfluous version number and type fields. <del> DTLS adds an explicit epoch and sequence number to the TLS record header.  This sequence number allows the recipient to correctly verify the DTLS MAC.  However, the number of bits used for the epoch and sequence number fields in the DTLSCiphertext structure have been reduced. </del> <ins> DTLS adds an epoch and sequence number to the TLS record header. This sequence number allows the recipient to correctly verify the DTLS MAC.  However, the number of bits used for the epoch and sequence number fields in the DTLSCiphertext structure have been reduced. </ins> The DTLSCiphertext structure has a variable length header.

The draft states in three places that DTLS use explicit sequence numbers, while this was true for DTLS 1.2, the sequence number in the DTLSCipherstructure is a mixture of implicit and explict. Maybe "Partly Explicit"?
Please provide me with a PR to add your name to acknowledgements.
Hi, I don’t know what PR is an abbreviation for but I guess it is not clear from my username who I am. I should probably update it, I did not intend to be anonymous :) John Mattsson, Ericsson, EMAIL


Question: clearer. Session Information ("i="), URI ("u="), Email Address ("e="), <del> Phone Number ("p="), Bandwidth ("b="), Repeat Times ("r="), and Time Zones ("z=") lines are not useful in this context and SHOULD NOT be included. </del> <ins> Phone Number ("p="), Repeat Times ("r="), and Time Zones ("z=") lines are not useful in this context and SHOULD NOT be included. </ins> Encryption Keys ("k=") lines do not provide sufficient security and MUST NOT be included.

Originally raised by Magnus: Section 5.2.1: o Session Information ("i="), URI ("u="), Email Address ("e="), Phone Number ("p="), Bandwidth ("b="), Repeat Times ("r="), and Time Zones ("z=") lines are not useful in this context and SHOULD NOT be included. Considering that b=CT is not useless, I am a bit surprised to see b= on this list. Yes, b=CT: is only useful is one like to provide a upper total bandwidth limitation. So, maybe some discussion of the possible choice?
This is just a mistake in 5.2.1 - we cover how to deal with it later in section 5. Prob need to remove the mention of b= here.


Question: who have not implemented WebRTC at all, WebRTC implementations need to be more careful. <del> Consider the case of a call center which accepts calls via RTCWeb. </del> <ins> Consider the case of a call center which accepts calls via WebRTC. </ins> An attacker proxies the call center's front-end and arranges for multiple clients to initiate calls to the call center.  Note that this requires user consent in many cases but because the data channel

I've found a few typos and fixed some sentences that didn't quite make sense. I don't think the meaning has been changed in any significant way.


Question: messages received. The 3-tuple (Cid, Sender ID, Partial IV) is called Transaction <del> Identifier (Tid), and SHALL be unique for each Base Key. The Tid is used as a unique challenge in the COSE object of the protected CoAP request.  The Tid is part of the Additional Authenticated Data (AAD, see sec-obj-cose) of the protected CoAP response message, which is how responses are bound to requests. </del> <ins> Identifier (Tid), and SHALL be unique for each Master Secret.  The Tid is used as a unique challenge in the COSE object of the protected CoAP request.  The Tid is part of the Additional Authenticated Data (AAD, see sec-obj-cose) of the protected CoAP response message, which is how responses are bound to requests. </ins> 3.2.

, and
This goes against Jim's wish to have things in bits rather than bytes... But HKDF uses octets and to combine the two parameters as wanted by Ludwig, they need the same unit.
Fixed in branch v-02
OSCOAP refers to HKDF [RFC5869], but it not clearly defined how the parameters map to HKDF-Extract and HKDF-Expand. E.g. IKM. Salt should be made optional.
In section 3.2.1. outlen is defined as "the key/IV size of the AEAD algorithm" and outputlength as "is the size of the AEAD key/IV in bytes encoded as an 8-bit unsigned integer" Isn't that essentially the same?
I just created a v-02 branch for the next update. I'll make an update as soon as possible trying to address this.
Fixed in brach v-02


Question: performance and functional characteristics; and it can communicate with different remote systems to optimize performance, robustness to failure, or some other metric.  Beyond these, if the API for the <del> system remains the same over time, new protocols and features could be added to the system's implementation without requiring changes in </del> <ins> system remains the same over time, new protocols and features can be added to the system's implementation without requiring changes in </ins> applications for adoption. <del> 3.1. </del> <ins> The normative requirements described here allow Transport Services APIs and Implementations to provide this functionlity without causing incompatibility or introducing security vulnerabilities.  The rest of this document describes the architecture non-normatively. </ins> <del> Functionality that is common across multiple transport protocols ought to be accessible through a unified set of API calls.  An application using a Transport Services API can implement logic for its basic use of transport networking (establishing the transport, and sending and receiving data) once, and expect that implementation to continue to function as the transports change. </del> <ins> 3.1. </ins> <del> As a baseline, any Transport Services API needs to allow access to the distilled minimal set of features offered by transport protocols </del> <ins> Any functionality that is common across multiple transport protocols SHOULD be made accessible through a unified set of Transport Services API calls.  As a baseline, any Transport Services API MUST allow access to the minimal set of features offered by transport protocols </ins> I-D.ietf-taps-minset. <ins> An application can specify constraints and preferences for the protocols, features, and network interfaces it will use via Properties.  A Transport Services API SHOULD offer Properties that are common to multiple transport protocols, in order to enable the system to appropriately select between protocols that offer equivalent features.  Similarly, a Transport Services API SHOULD offer Properties that are applicable to a variety of network layer interfaces and paths, in order to permit racing of different network paths without affecting the applications using the system.  Each Property is expected to have a default value. The default values for Properties SHOULD be selected to ensure correctness for the widest set of applications, while providing the widest set of options for selection.  For example, since both applications that require reliability and those that do not require reliability can function correctly when a protocol provides reliability, reliability ought to be enabled by default.  As another example, the default value for a Property regarding the selection of network interfaces ought to permit as many interfaces as possible. Applications using a Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, where the automated selection is constrained by the requirements and preferences expressed by the application. </ins> 3.2. There are applications that will need to control fine-grained details of transport protocols to optimize their behavior and ensure compatibility with remote systems.  A Transport Services system <del> therefore ought to also permit more specialized protocol features to be used.  The interface for these specialized options ought to be exposed differently from the common options to ensure flexibility. </del> <ins> therefore SHOULD permit more specialized protocol features to be used. </ins> A specialized feature could be required by an application only when using a specific protocol, and not when using others.  For example,

This proposal consolidates normative language into the section on design principles (now requirements), using the suggestions in .
Please review the latest updates!
Early art-art review of draft-ietf-taps-architecture My first observation and request is that the group rethink the separation of the architecture and interface documents. It is a warning-sign that the interface document is a normative reference for the architecture document. As the documents are currently constructed, a new reader cannot grasp the architecture without inferring some of it through reading the interface. Consider reformulating the architecture document as an overview - introduce principles, concepts, and terminology, but don't attempt to be normative. (There are signs that this may already be, or have been, a path the document was being pushed towards). If nothing else, rename it, and acknowledge that the actual architecture is specified using the interface definition. Note that the first sentence of the Overview in the interface document also highlights the current structural problem.
I don't entirely agree with this ART-ART comment: I think I might well agree with moving more of the description to this Arch document, but as discussed before, I also think it useful here to set the actual requirements for the architecture, and to define the architecture. It may need to be much clearer that this architecture consists of more the API.
For completeness, this is the first statement of the interface document review: This document reads fairly well. I have some structural concerns (see the early review of the architecture document). In particular I think this document contains the actual definition of the architecture and some of that is inferential.
Discussion at September 2021 interim: pass through this document with a view toward making it a high level introduction, not an architecture definition: "An Introduction to the Transport Services Architecture". remove normative language from this document (moving it to API, where necessary). move details not useful in a high level intro to API.
I disagree with this conclusion at first pass—we did a specific analysis and pass on this over a year ago, with PR . The normative language and the standards-track aspect was very intentional and something the WG had consensus on. The point is that the API is a specific incarnation of the TAPS architecture, but there are overarching architectural properties that go beyond a specific abstract API.
I completely agree with Tommy here, and recommend we fix this in . We should also of course check for any stray text or missing concepts.


Question: performance and functional characteristics; and it can communicate with different remote systems to optimize performance, robustness to failure, or some other metric.  Beyond these, if the API for the <del> system remains the same over time, new protocols and features could be added to the system's implementation without requiring changes in </del> <ins> system remains the same over time, new protocols and features can be added to the system's implementation without requiring changes in </ins> applications for adoption. <del> 3.1. </del> <ins> The normative requirements described here allow Transport Services APIs and Implementations to provide this functionlity without causing incompatibility or introducing security vulnerabilities.  The rest of this document describes the architecture non-normatively. </ins> <del> Functionality that is common across multiple transport protocols ought to be accessible through a unified set of API calls.  An application using a Transport Services API can implement logic for its basic use of transport networking (establishing the transport, and sending and receiving data) once, and expect that implementation to continue to function as the transports change. </del> <ins> 3.1. </ins> <del> As a baseline, any Transport Services API needs to allow access to the distilled minimal set of features offered by transport protocols </del> <ins> Any functionality that is common across multiple transport protocols SHOULD be made accessible through a unified set of Transport Services API calls.  As a baseline, any Transport Services API MUST allow access to the minimal set of features offered by transport protocols </ins> I-D.ietf-taps-minset. <ins> An application can specify constraints and preferences for the protocols, features, and network interfaces it will use via Properties.  A Transport Services API SHOULD offer Properties that are common to multiple transport protocols, in order to enable the system to appropriately select between protocols that offer equivalent features.  Similarly, a Transport Services API SHOULD offer Properties that are applicable to a variety of network layer interfaces and paths, in order to permit racing of different network paths without affecting the applications using the system.  Each Property is expected to have a default value. The default values for Properties SHOULD be selected to ensure correctness for the widest set of applications, while providing the widest set of options for selection.  For example, since both applications that require reliability and those that do not require reliability can function correctly when a protocol provides reliability, reliability ought to be enabled by default.  As another example, the default value for a Property regarding the selection of network interfaces ought to permit as many interfaces as possible. Applications using a Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, where the automated selection is constrained by the requirements and preferences expressed by the application. </ins> 3.2. There are applications that will need to control fine-grained details of transport protocols to optimize their behavior and ensure compatibility with remote systems.  A Transport Services system <del> therefore ought to also permit more specialized protocol features to be used.  The interface for these specialized options ought to be exposed differently from the common options to ensure flexibility. </del> <ins> therefore SHOULD permit more specialized protocol features to be used. </ins> A specialized feature could be required by an application only when using a specific protocol, and not when using others.  For example,

I think it would also be useful to somehow better explain that in figure 3 the API function is ONE part of the system.
See linked PR first please to see if this resolves this?
A PR was merged that addressed important text issues with separation of requirements and overview, and should more clearly set out that the requirements are for the system. The question of whether the overall TAPS systems requirements belong in this ID; the API; or a separate RFC needs some discussion based on the review, before this can be closed.
Many good text improvements in the PR NAME but I noticed two smaller updates in the PR that I think were incorrect. Fixed in
The review comment showed were many places where the wording of architecture can be improved. There are some high level principles that I think should remain in the architecture document. The decision in today's interim is to review the architecture to get this more into a shape that is valuable as a reference, and then check the updated draft.
I now suggest the issues are addressed and we close this issue without moving text between documents
I agree with that; closing.
Overall review: I think we need to keep the normative language for the key features that differentiate the API. I'd really like to use REQUIRED, RECOMMENDED, rather MUST and SHOULD - because these are not protocol operation requirements, but are implementation requirements to conform. This includes requirements based on Minset - where the WG dervied this API: This is after all why the WG worked on Minset as the basis of the API. I suggest we group all the requirements on architecture as short one or two sentence blocks (preferably as a bullet or definition list) collected into a single subsection, so they are easy to find and read and we say that these are the key architectural requirements. We insert this new requirements text either their own section, or in section 1.3 (although I am open to other places where we can call-out the architectural requirements). The following are key requirements, that define the TAPS architecture: It is RECOMMENDED that functionality that is common across multiple transport protocols is made accessible through a unified set of TAPS interface calls. As a baseline, a Transport Services system interface is REQUIRED to allow access to the minimal set of features offered by transport protocols {{?I-D.ietf-taps-minset}}.


Question: performance and functional characteristics; and it can communicate with different remote systems to optimize performance, robustness to failure, or some other metric.  Beyond these, if the API for the <del> system remains the same over time, new protocols and features could be added to the system's implementation without requiring changes in </del> <ins> system remains the same over time, new protocols and features can be added to the system's implementation without requiring changes in </ins> applications for adoption. <del> 3.1. </del> <ins> The normative requirements described here allow Transport Services APIs and Implementations to provide this functionlity without causing incompatibility or introducing security vulnerabilities.  The rest of this document describes the architecture non-normatively. </ins> <del> Functionality that is common across multiple transport protocols ought to be accessible through a unified set of API calls.  An application using a Transport Services API can implement logic for its basic use of transport networking (establishing the transport, and sending and receiving data) once, and expect that implementation to continue to function as the transports change. </del> <ins> 3.1. </ins> <del> As a baseline, any Transport Services API needs to allow access to the distilled minimal set of features offered by transport protocols </del> <ins> Any functionality that is common across multiple transport protocols SHOULD be made accessible through a unified set of Transport Services API calls.  As a baseline, any Transport Services API MUST allow access to the minimal set of features offered by transport protocols </ins> I-D.ietf-taps-minset. <ins> An application can specify constraints and preferences for the protocols, features, and network interfaces it will use via Properties.  A Transport Services API SHOULD offer Properties that are common to multiple transport protocols, in order to enable the system to appropriately select between protocols that offer equivalent features.  Similarly, a Transport Services API SHOULD offer Properties that are applicable to a variety of network layer interfaces and paths, in order to permit racing of different network paths without affecting the applications using the system.  Each Property is expected to have a default value. The default values for Properties SHOULD be selected to ensure correctness for the widest set of applications, while providing the widest set of options for selection.  For example, since both applications that require reliability and those that do not require reliability can function correctly when a protocol provides reliability, reliability ought to be enabled by default.  As another example, the default value for a Property regarding the selection of network interfaces ought to permit as many interfaces as possible. Applications using a Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, where the automated selection is constrained by the requirements and preferences expressed by the application. </ins> 3.2. There are applications that will need to control fine-grained details of transport protocols to optimize their behavior and ensure compatibility with remote systems.  A Transport Services system <del> therefore ought to also permit more specialized protocol features to be used.  The interface for these specialized options ought to be exposed differently from the common options to ensure flexibility. </del> <ins> therefore SHOULD permit more specialized protocol features to be used. </ins> A specialized feature could be required by an application only when using a specific protocol, and not when using others.  For example,

A Transport Services system automates the selection of protocols and features and network, based on a set of Properties supplied through the TAPS interface. It is RECOMMENDED that a Transport Services system offers Properties that are common to multiple transport protocols. Specifying common Properties enables a system to appropriately select between protocols that offer equivalent features. This design permits evolution of the transport protocols and functions without affecting the programs using the system. Specifying common Properties enables a Transport Services system to appropriately select an appropriate network interface. It is RECOMMENDED that a system offers Properties that are common to a variety network layer interfaces and paths. This design permits racing of different network paths without affecting the programs using the system. Applications using the Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, including the possibility to express requirements and preferences that constrain the choices that can be made by the system. If two different Protocol Stacks can be safely swapped, or raced in parallel (see Section 4.2.2), by the Transport Services system then they are considered to be "equivalent". Equivalent Protocol Stacks need to meet the requirements for racing specified in section XX. Applications using a Transport Services system MAY explicitly require or prevent the use of specific transport features (and transport protocols) and XXXinterfaces using the "REQUIRED" and "PROHIBIT" preferences. This allows an application to constrain the set of protocols and features that will be selected for the transport service. It is RECOMMENDED that the default usage by applications does not necessarily restrict this selection, so that the system can make informed choices (e.g., based on the availability of interfaces or set of transport protocols available for the specified remote endpoint). When a Transport Services system races between two different Protocol Stacks, both SHOULD use the same security protocols and options. However, a Transport Services system MAY race different security protocols, e.g., if the application explicitly specifies that it considers them equivalent. Transport Services systems MUST NOT automatically fall back from secure protocols to insecure protocols, or to weaker versions of secure protocols. A Transport Services system MAY allow applications to specify that fallback to a specific other version of a protocol is allowed. It is RECOMMENDED that in normal use, a Transport Services system is designed so that all selection decisions result in consistent choices for the same network conditions when they have the same set of preferences.


Question: performance and functional characteristics; and it can communicate with different remote systems to optimize performance, robustness to failure, or some other metric.  Beyond these, if the API for the <del> system remains the same over time, new protocols and features could be added to the system's implementation without requiring changes in </del> <ins> system remains the same over time, new protocols and features can be added to the system's implementation without requiring changes in </ins> applications for adoption. <del> 3.1. </del> <ins> The normative requirements described here allow Transport Services APIs and Implementations to provide this functionlity without causing incompatibility or introducing security vulnerabilities.  The rest of this document describes the architecture non-normatively. </ins> <del> Functionality that is common across multiple transport protocols ought to be accessible through a unified set of API calls.  An application using a Transport Services API can implement logic for its basic use of transport networking (establishing the transport, and sending and receiving data) once, and expect that implementation to continue to function as the transports change. </del> <ins> 3.1. </ins> <del> As a baseline, any Transport Services API needs to allow access to the distilled minimal set of features offered by transport protocols </del> <ins> Any functionality that is common across multiple transport protocols SHOULD be made accessible through a unified set of Transport Services API calls.  As a baseline, any Transport Services API MUST allow access to the minimal set of features offered by transport protocols </ins> I-D.ietf-taps-minset. <ins> An application can specify constraints and preferences for the protocols, features, and network interfaces it will use via Properties.  A Transport Services API SHOULD offer Properties that are common to multiple transport protocols, in order to enable the system to appropriately select between protocols that offer equivalent features.  Similarly, a Transport Services API SHOULD offer Properties that are applicable to a variety of network layer interfaces and paths, in order to permit racing of different network paths without affecting the applications using the system.  Each Property is expected to have a default value. The default values for Properties SHOULD be selected to ensure correctness for the widest set of applications, while providing the widest set of options for selection.  For example, since both applications that require reliability and those that do not require reliability can function correctly when a protocol provides reliability, reliability ought to be enabled by default.  As another example, the default value for a Property regarding the selection of network interfaces ought to permit as many interfaces as possible. Applications using a Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, where the automated selection is constrained by the requirements and preferences expressed by the application. </ins> 3.2. There are applications that will need to control fine-grained details of transport protocols to optimize their behavior and ensure compatibility with remote systems.  A Transport Services system <del> therefore ought to also permit more specialized protocol features to be used.  The interface for these specialized options ought to be exposed differently from the common options to ensure flexibility. </del> <ins> therefore SHOULD permit more specialized protocol features to be used. </ins> A specialized feature could be required by an application only when using a specific protocol, and not when using others.  For example,

This is intended to provide predictable outcomes to the application using the transport service. I would then suggest to use lower case within other parts of the document - except were pointed to from here, (we could refer to the requirements subsection, if we thought necessary).
for discussion at the interim, but IMO we could start the interim with a PR based on these suggestions and work from there.
I suspect that we need more protocol-specific properties to be able to talk to various kinds of native protocol peers (other than TCP and UDP: these should already work fine with what we have, I believe). A concrete example: we're hiding stream numbers. In NEAT, we needed to make them configurable though, because we wanted a NEAT client to be able to talk to a native SCTP server, where an application may expect certain type of data to arrive via a certain stream number. I suspect that we may need something similar for a QUIC peer. Should we address this? (I'd say yes) How .. is it a protocol-specific (per-Message) property? (I'd say yes... and perhaps a per-Connection property too, with text explaining that the per-Message one automatically becomes the Connection default unless otherwise specified) What other such things are we missing... what other expectations may e.g. a native QUIC or SCTP application have that we just cannot fulfil with our current system?


Question: 6.3.2.1. This sub-tree defines the L2VPN service type, according to the <del> several signalling options to exchange membership information between </del> <ins> several signaling options to exchange membership information between </ins> PEs of an L2VPN.  The following signaling options are supported: <del> Refers to the BGP control plane as described in RFC4761 and RFC6624. Refers to the BGP control plane as described in RFC7432 and RFC7209. </del> <ins> The service is a Multipoint VPLSs that use a BGP control plane as described in RFC4761 and RFC6624.  For this L2VPN service, the model allows configuring the Route Distinguisher, the Route Targets, the PWE encapsulation type and the MTU configurations to allow MTU mismatch. The service is a Multipoint VPLSs that use also a BGP control plane but also includes the additional features and related parameters RFC7432 and RFC7209.  The model allows the configuration of the EVPN service interface type, the local and remote VPWS Service Instance (VSI) RFC8214, the EVPN policies for handling MAC addresses and the Ethernet Segment Identifier (ESI) information. </ins> <del> Refers to LDP-signaled pseudowires RFC6074. </del> <ins> A Multipoint VPLSs that use a mesh of LDP-signaled Pseudowires RFC6074.  The model allows configuring the T-LDP PWE type, MTU mismatch and the list of AC and PW bindings. </ins> <del> Refers to L2TP-signaled pseudowires RFC6074. </del> <ins> The L2 VPN uses L2TP-signaled Pseudowires as described in RFC6074. The model allows configuring TBC. </ins> 6.3.2.2.

Adding a description to signaling options to improve readability. I have pending to update L2TP after the other issue is solved URL
Thanks, Victor. Does the text reflect the latest version of signaling/discovery split?
leaf l2tp-type { type identityref { base t-ldp-pwe-type; } description "T-LDP PWE type."; }
The type should be taken from this registry: URL
A new IANA-maintained module is created to fix this issue.


Question: 6.1.2. <ins> Note that this API has multiple ways to constrain and prioritize endpoint candidates based on the network interface: Specifying an interface on a RemoteEndpoint qualifies the scope of the remote endpoint, e.g., for link-local addresses. Specifying an interface on a LocalEndpoint explicitly binds all candidates derived from this endpoint to use the specified interface. Specifying an interface using the "interface" Selection Property (prop-interface) or indirectly via the "pvd" Selection Property (prop-pvd) influences the selection among the available candidates. While specifying an interface on an endpoint restricts the candidates available for connection establishment in the Pre-Establishment Phase, the Selection Properties prioritize and constrain the connection establishment. 6.1.3. </ins> An Endpoint can have an alternative definition when using different protocols.  For example, a server that supports both TLS/TCP and QUIC may be accessible on two different port numbers depending on which

Explain the different use-cases on specifying interfaces to
There's something obvious I'm not seeing: if I want to specify a multicast or link-local address for an endpoint, why not simply specify that address? What's the use of adding the interface into it? Some addresses will inherently limit interfaces; I thought interface preference was about preferring wifi to 4G, etc.
Like, in section 6.1.1:
Multicast groups can be scoped (see RFC7346), thus, may only be available on certain interfaces or even have different meanings based on the interface. So for Interface- and Link-Local scope, specifying the interface is strictly required. For others use cases (like coping with non-unique use of RFC1918 addresses) this may also be helpful.
Why does the API have both the ability to say LocalSpecifier.WithInterface and to set interface requirements on TransportProperties? Why not just keep the latter and remove the former?
There's history here: LocalSpecifier.WithInterface predates explicitly setting the interface name in interface requirements. There might be reasons to keep this: LocalSpecifier is more likely to show up in application UI than in transport properties. Any text proposed to keep both would need to have an unambiguous way to resolve conflicts between the two. This might be a source of hard-to-debug errors.
To discuss next interim


Question: different algorithms such as loss-based CUBIC RFC8312, delay-based COPA or BBR, or even something completely different. <del> We do have experience with deploying new congestion controllers without melting the Internet (CUBIC is one example), but the point mentioned in tcp-behavior about TCP being implemented in operating system kernels is also different with QUIC.  Although QUIC can be implemented in operating system kernels, one of the design goals when this work was chartered was "QUIC is expected to support rapid, distributed development and testing of features", and to meet this expectation, many implementers have chosen to implement QUIC in user space, outside the operating system kernel, and to even distribute QUIC libraries with their own applications. The decision to deploy a new version of QUIC is relatively uncontrolled, compared to other widely used transport protocols, and this can include new transport behaviors that appear without much notice except to the QUIC endpoints.  At IETF 105, Christian Huitema and Brian Trammell presented a talk on "Congestion Defense in Depth" CDiD, that explored potential concerns about new QUIC congestion controllers being broadly deployed without the testing and instrumentation that current major content providers routinely include.  The sense of the room at IETF 105 was that the current major content providers understood what is at stake when they deploy new congestion controllers, but this presentation, and the related discussion in TSVAREA minutes from IETF 105 (tsvarea-105, are still worth a look for new and rapidly growing content providers. </del> <ins> The Internet community does have experience with deploying new congestion controllers without melting the Internet.  As noted in RFC8312, both the CUBIC congestion controller and its predecessor BIC have significantly different behavior from Reno-style congestion controllers such as TCP NewReno RFC6582, but both CUBIC and BIC were added to the Linux kernel in order to allow experimentation and analysis, and both were then selected as the default TCP congestion controllers in Linux, and both were deployed globally. The point mentioned in tcp-behavior about TCP congestion controllers being implemented in operating system kernels is different with QUIC. Although QUIC can be implemented in operating system kernels, one of the design goals when this work was chartered was "QUIC is expected to support rapid, distributed development and testing of features", and to meet this expectation, many implementers have chosen to implement QUIC in user space, outside the operating system kernel, and to even distribute QUIC libraries with their own applications. It is worth noting that streaming operators using HTTP/3, carried over QUIC, can expect more frequent deployment of new congestion controller behavior than has been the case with HTTP/1 and HTTP/2, carried over TCP. </ins> It is worth considering that if TCP-based HTTP traffic and UDP-based HTTP/3 traffic are allowed to enter operator networks on roughly

From NAME Please consider using the same wording for the title as for 6.2 and 6.1 (nit) Who is "we" in " We do have experience" ? While I share some concerns about deploying a new transport in user space as opposed to more stable and controlled kernel space, I wonder whether this paragraph is relevant in this document. Does it bring any value ? If no, then please remove it.
We'll look at the date for CUBIC being enabled by default in the Linux kernel as "experience". We'll delete the paragraph as suggested.
was fixed elsewhere.
LGTM. I'm almost sad to lose the ref to "Congestion Defense in Depth", but I guess it makes good sense, that's not a consensus position.


Question: Endpoints which receive a Capsule with an unknown Capsule Type MUST silently drop that Capsule. <del> Receipt of a CAPSULE HTTP/3 Frame on a stream that is not a client- initiated bidirectional stream MUST be treated as a connection error of type H3_FRAME_UNEXPECTED. CAPSULE frames MUST NOT be sent on a stream before at least one HEADERS frame has been sent on that stream.  This removes the need to buffer capsules when the endpoint needs information from headers to determine how to react to the capsule.  If a CAPSULE frame is received on a stream before a HEADERS frame, the receiver MUST treat this as a connection error of type H3_FRAME_UNEXPECTED. </del> <ins> 4.4. </ins> <del> 4.1. </del> <ins> 4.4.1. </ins> The REGISTER_DATAGRAM_CONTEXT capsule (see iana-types for the value of the capsule type) allows an endpoint to inform its peer of the

Had offline conversation with Lucas, conclusion was that this is good enough to merge to allow interop - we'll have plenty of time to figure out details in subsequent issues and PRs.


Question: Transmit the Welcome message to the other new members <ins> Group IDs SHOULD be constructed in such a way that there's an overwhelmingly low probability of honest group creators generating the same group ID, even without assistance from the Delivery Service. For example, by making the group ID a freshly generated random value of size "KDF.Nh".  The Delivery Service MAY attempt to ensure that group IDs are globally unique by rejecting the creation of new groups with a previously used ID. </ins> The recipient of a Welcome message processes it as described in joining-via-welcome-message.  If application context informs the recipient that the Welcome should reflect the creation of a new group

Brendan to revise based on Rohan suggestion.
Interim 2022-05-26: Concerns around making the group_id totally random Malicious group creator can choose it malicious Federated systems get worse randomness Using AEAD.Nk depends on ciphersuite, could be awkward NAME will revise to say: Here are the requirements that group ID must satisfy If you don’t have an alternative plan, do it randomly
Group ids are signed to bind certain information to specific groups. But I can imagine a case where a user mis-uses the group id to be something like a "room title" which lets an adversary control the group id and introduces potential problems. If we can prove that it's fine for a group id to be reused, then should we get rid of group ids? If not (as I expect), then group ids need to be guaranteed to be unique [globally per signing key] and we need to specify that.
Discussed at 19 May interim. NAME to generate PR documenting fresh random value approach.
Virtual interim 2022-05-19: Agreement that some sort of uniqueness requirement is worthwhile Main question is what scope of uniqueness should be Globally? Within scope of DS? Fresh random value? Simple, but precludes application usage of the field Also could help avoid metadata leakage Client / DS should probably still check for collisions NAME to write up a PR on the fresh random value idea
Group IDs must be generated randomly by creator Clients and DS must check that group ids are unique as far as they know


Question: In pseudocode, this procedure is as follows: <del> decrypt an by reversing this process.  That is, they first parse </del> <ins> decrypt an by reversing this process.  That is, first parse </ins> "enc_response" into "response_nonce" and "ct".  They then follow the <del> same process to derive values for "aead_key" and "aead_nonce". </del> <ins> same process to derive values for "aead_key" and "aead_nonce", using their sending HPKE context, "sctxt", as the HPKE context, "context". </ins> The uses these values to decrypt "ct" using the Open function provided by the AEAD.  Decrypting might produce an error, as follows:

Why is it that media type registration is a journey of discovery each and every time? It's like the coastline paradox: the closer you get, the more detail resolves. Also, each requirement makes sense in isolation, but if you need an entire RFC to explain it, maybe something has gone wrong.
no strong feelings on this one.


Question: The inner JWS MUST have the same "url" header parameter as the outer JWS. <del> The inner JWS MAY omit the "nonce" header parameter.  The server MUST ignore any value provided for the "nonce" header parameter. </del> <ins> The inner JWS MUST omit the "nonce" header parameter. </ins> This transaction has signatures from both the old and new keys so that the server can verify that the holders of the two keys both

URL
Thanks NAME On Sat, Sep 15, 2018 at 08:51:55PM -0500, Benjamin Kaduk wrote: My apologies if I missed it when it went by, but did we ever hear more about this requirement from the proponents of externalAccountBinding? Thanks, Benjamin


Question: to cases where the Unencrypted Resolver and Designated Resolver have the same IP address. <ins> The constraints on validation of Designated Resolvers specified here apply specifically to the automatic discovery mechanisms defined in this document, which are referred to as Authenticated Discovery and Opportunistic Discovery.  Clients MAY use some other mechanism to validate and use Designated Resolvers discovered using the DNS SVCB record.  However, use of such an alternate mechanism needs to take into account the attack scenarios detailed here. </ins> 8. 8.1.

Three changes Re-ordered statements in the abstract to place the restriction on unencrypted bootstrapping next to the unencrypted statement (instead of after the encrypted bootstrapping statement, which makes the restriction seem like a conflict where it excludes bootstrapping from a name instead of an IP address) Re-ordered security considerations (placed the "clients can opt to do other things" statement at the end after the MUSTs/SHOULDs for this doc) s/client/clients for grammatical correctness


Question: actually allowed to evict other members; groups can enforce access control policies on top of these basic mechanism. <ins> 4.3. A group comprises a single linear sequence of epochs and groups are generally independent of one-another.  However, it can sometimes be useful to link epochs cryptographically, either within a group or across groups.  MLS derives a resumption pre-shared key (PSK) from each epoch to allow entropy extracted from one epoch to be injected into a future epoch.  This link guarantees that members entering the new epoch agree on a key if and only if were members of the group during the epoch from which the resumption key was extracted. MLS supports two ways to tie a new group to an existing group.  Re- initialization closes one group and creates a new group comprising the same members with different parameters.  Branching starts a new group with a subset of the original group's participants (with no effect on the original group).  In both cases, the new group is linked to the old group via a resumption PSK. Applications may also choose to use resumption PSKs to link epochs in other ways.  For example, the following figure shows a case where a resumption PSK from epoch "n" is injected into epoch "n+k".  This demonstrates that the members of the group at epoch "n+k" were also members at epoch "n", irrespective of any changes to these members' keys due to Updates or Commits. </ins> 5. The protocol uses "ratchet trees" for deriving shared secrets among a

Adds the overview text proposed in .
Thanks NAME !
Thanks for putting together the overview. The figures are great :-) Looks good to me with one proposition.