connection ID when the client initiates migration to the preferred address.  The active connection ID limit is an integer value specifying the maximum number of connection IDs from the peer that an endpoint is willing to store.  This value includes the connection ID received

This is option 4 from the discussion in . That is, making preferredaddress equivalent to NEWCONNECTION_ID with a sequence number of 1 in every way that matters.
Offshoot from review. Servers behind a load balancer, and the load balancer isn't CID-aware. Each server gives out an IP:port per client in the parameter. If the client moves to the SPA, then future migration is fine -- it will continue to talk to the same server using the preferred address. If the client doesn't move, those future migrations will break. There's currently no way to set to "only if you don't use my preferred address". Similarly, if the primary connection is to an anycast address, if the client fails to move to the preferred address, subsequent migration will likely break. Should we blanket recommend that clients not actively migrate if they ignore / are unsuccessful in using the SPA? Or should there be a way for the server to indicate that?
Good question. This brings up a related question for me about disableactivemigration. Given a server can decide not to give out any additional connection IDs, doesn't that give it a mechanism for disabling active migration? Ironically, given recent decisions on connection CIDs and SPA, if an SPA is supplied, then the client always has a second connection ID it can use on any path, not just the SPA. So my suggestion of not giving out extra CIDs fails for the original path, but not the SPA.
What we’ve said for this so far is that SPA is limited to offering a minimum of a single additional CID that needs to work in both places, the original server and the preferred server. As Ian says, the original server should not issue the client any more CIDs (if you have some that are deployment specific) until after they’ve successfully migrated to the new endpoint, at which point the preferred server can handle that issuance. And as he notes, you still have to deal with the fact that there are two CIDs that the client could use, from any incoming address. It seems as though the full way to solve this is to define potentially disjoint sets of TPs that different servers could set to match their particular requirements (around more than just migration). In that world, if you had an initial server that couldn’t handle migration and a preferred server that did handle migration, you could set different TPs to indicate that. It goes well beyond that though, any other TP might have a different values for the original server and then have those be different with the preferred server. But that does feel an awful lot like something that goes with full server migration. In the limited form that we have now, if you want to be able to push someone off to a different server, there are some limitations that you need to deal with, like being able to set up enough of a shared configuration for the two that the same TPs work with both. In practice, I think this means we end up with the following situation: The client either moves to the requested address or it doesn’t. If it does, then you’re in good shape, in theory you gave it TPs that matched what the server wanted and everybody is happy. If it doesn’t, then you’re unhappy, since in theory you had a reason you asked it to move in the first place. You might choose to close it down more aggressively than you would on the preferred server to save resources, or give it smaller limits for flow control, etc. When you’re in the “client didn’t move” state: If the client then tries to use the original 2nd CID that you gave, from the same IP address/port, then a non-CID aware load balancer should be fine. If the client then tries to use the original 2nd CID that you gave, from a different IP address/port, then a non-CID aware load balancer will not be happy. Of course, the problematic case is even narrower, since if the client is probing the new path they’ll see it as though there isn’t any connectivity, just like they will if there’s anything else on the new path that prevents their connection from working — this is always a risk when migrating, so they should be unsurprised and stick with their current path. If they’re forced to move for whatever reason, then it will be just as though they didn’t have connectivity over the new path (which they don’t) and whether or not you disable migration you’ll see the same outcome — the connection is closed. Interestingly, I’m not actually seeing a difference from the client’s perspective in terms of outcome whether you let them try with that CID and fail or if you told them to not try.
In very much shorter terms: isn’t this a situation that all clients already need to be able to handle for other reasons?
NAME I do not like the first option, because IMO banning migration on the original address is going to be a premature optimization based on what we have now, has negative effect on what we would have in the future. My assumption is that the majority of the server deployments would be CID-aware, especially in the long term, especially in terms of traffic volume. Note that the volume of traffic is more important than the number of server deployments, as migration is about (improving) end-user experience. IMO there are two ways forward: (a) State that disableactivemigration TP only applies to the original address. Per-definition, server-preferred address is an address that is capable of receiving packets from any address. There is no need to prohibit clients doing migration when connected to SPA. (b) Do nothing. Clients are RECOMMENDED to implement SPA, and this issue is about a small fraction of deployments that rely on non-CID-aware load balancers. We can ignore the problem.
IIRC, we had that conversation when deciding whether to add in the first place. The short version is that taking this approach prevents the client from rotating CIDs on the primary path; if we want the client to be able to rotate CIDs but not migrate, they need to be distinct signals. NAME in a way you're correct that clients will have to deal with the fallout regardless of what the server says. However, the actual use of is likely to trigger a little more than just "don't do that" -- a client might PING more actively to prevent NAT rebindings, for example. I think it's this hinting aspect that's actually more valuable, since as you note, clients need to be prepared for attempted migrations to fail even with servers that tacitly support them. I don't think this is necessarily true, though I agree that we're talking about smaller and smaller fractions of server space.
NAME While I agree that it might not necessarily be true at the moment, I might argue that it is going to be true. The basis of the resolution for and (both on consensus call) is that servers using preferred address would not be using the server's address for distinguishing between connections. We also know that a server cannot use client's address (unless there is out-of-band knowledge), because clients are not required to use the same 2-tuple when migrating to a different address (and also because there are NATs). The outcome is that the server has to rely on CID when using SPA. And therefore I might insist that servers using SPA would always be capable of handling further migration.
NAME So your suggestion is that disableactivemigration applies only to the original address and the SPA must always support active migration, because it has to use CID routing? I don't think that'd be a problem for our use cases, but I'm not sure about others.
NAME Exactly. I think that that would be the correct thing to do, if we are to make changes. I agree that existing clients might be applying after migrating to SPA, and that they do not want change. But they can remain as they are, because the change in spec does not lead to interoperability issues. To paraphrase, I think making the aforementioned change is strictly better than doing nothing, because it would be an improvement to some, while not hurting anybody.
A truth table might help here. ... --- no preferred address preferred address Right now, our definition of disableactivemigration is a simple "don't migrate". But the argument here says that the server that provides a preferred address is probably capable of handling migrations on the basis that it has to support one. We might insist on the source address not change, but the fact is that some NATs (those with address-dependent mappings) will change the source address. Changing to a definition where disableactivemigration is instead "don't move to a new address unless you use a preferred address" is a little more complicated, but doable. Is it worth the extra complexity this late in the process? Is that complexity justified when it would be trivially possible to define another extension that said "I support migration on my preferred address"?
I'd like to minimize changes, but I believe the status quo is that disableactivemigration applies to both the original address and SPA. As such, all suggestions above seem like a change. There is a mechanism to disable migration on the SPA, which is don't give out any more CIDs once they migrate. I don't expect any hosts to support migration on the original path and not the SPA, so I don't think that's a use case worth supporting(or encouraging).
My proposal is that we defer the ability to signal "I support migration, but only on my preferred address" to an extension.
NAME And leave the current behavior of disableactivemigration applies to both the original address and SPA? I'm a bit concerned that makes SPA somewhat useless. Is this issue a candidate for parking and we can come back to and see if we can live with the status quo?
Which, again, not only disables migration, it disables the ability to rotate CIDs. That's not something we want servers to be doing. NAME argument is convincing that it's improbable for something which supports SPA to not be able to uniquely identify connections (whether by CID or because the SPA endpoint is unique). Therefore, I'd support scoping disableactivemigration to the handshake address, and defer the ability to disable it on the SPA to an extension. Loosening this definition doesn't make any current clients break, so I suspect it's bearable despite the schedule.
If we're going to make a change, I support NAME and NAME direction of scoping disableactivemigration to the handshake address. And I agree that I don't expect this breaks many existing clients.
I personally find the proposed change confusing. I'm not sure I find the use-case of "my original address doesn't support migration but my preferred address does" compelling. Since this can be easily solved by an extension with no downside over it being in the core docs, I don't see a reason to change the spec at this point in the process.
As this has sat here a while, I'll reiterate my proposal again: we defer the ability to signal "I support migration, but only on my preferred address" to an extension. Which means that disableactivemigration applies to both the original address and SPA, so answer the question (sorry Ian, I missed that). I realize that you could change the meaning of disableactivemigration to apply to the address the server uses in the handshake, but there are interesting corner cases and I don't think we need to concern ourselves with those. For instance, if you connect over V4 and get disableactivemigration and serverpreferredaddress. If that SPA includes a V6 address you can't use, along with the same V4 address, what then? Does the server really support migration? I know that the current situation is unappealing, and I think that disableactivemigration is not good for the protocol. But it exists for very good reasons. Trying to limit the scope is admirable, but as we haven't seen a concrete proposal, I think we need to move on.
NAME If this example is concerning, I'd argue that is an issue that already exists, orthogonal to what we are discussing here. I am still not convinced that limiting the scope of disableactivemigration to just the original address is going to cause any new issue. That said, if nobody needs this change, I'm more than happy to move on with status-quo.
I've thought about this, and I don't believe we need this change, but I also think the change NAME suggested makes the current behavior better overall. A server that supports migration on the SPA(ie: unicast) and not on the handshake address(ie: anycast) will end up not disabling migration and hoping that most clients migrate to the SPA or don't support migration of any sort. That seems like a plausible hope, though I'm a bit leery of hope based protocol design.
Thinking about this more, a statement like: "Migration is never guaranteed to succeed. For servers that advertise a preferred address, it's expected migration is more likely to be successful when using the preferred address than the handshake address."
tl;dr: I'm arguing for "do nothing". This conversation has made it clear to me that an address, and not a server, is the thing that supports client migration. Which makes sense -- migration support at the server is entirely a question of routability, and it makes sense that it would be tied to the address in question and not the endpoint. So if we were to do anything, I think I would propose: scoping disableactivemigration to the handshake address, and adding two disableactivemigration flags to the preferredaddress TP, for the two addresses in the TP. That said, this is a clean enough extension. A new extension can define a new TP that provides this granular interpretation of the three server addresses. A server that wishes to support this can send both the SPA and this newSPA TPs, with the SPA TP (with the current scoping of disableactivemigration applying to all server addresses) providing cover for legacy clients. Also I expect that clients are likely to switch to using the SPA if provided. A server that has different routability properties for the two addresses can wait for some deployment and make decisions on whether to risk allowing migration based on data of what it sees as client behavior with SPA. I would argue for doing nothing now. Experience will teach us the right way forward. If necessary, this extension will be easy enough to do, and importantly, the current interpretation of disableactive_migration gives us the right kind of cover in the meantime.
I would rather not make the change in . It assumes that migration is more likely to succeed when attempted to the preferred address as opposed to the handshake address. While this assumption might hold in some server deployments, it is not a fact. I'd rather we didn't bake this assumption into the core spec. I'd prefer to see this feature as an extension.
NAME Would you mind elaborating why a server, accepting connections on a preferred address, might not be able to disambiguate connections using CIDs? As stated previously, when a client migrates to the preferred address, the client's address port tuple can change due to the existence of NATs. A server using preferred address need to take that into consideration. That means that a server has to disambiguate connections based on the CID, which in turn means that the server would be possible to support migration.
Hmmm thanks for that note NAME Looking at things under that light, I agree that the assumption I described above is already part of the protocol. Please disregard my comment above URL .
We don't prohibit the use of a zero-length connection ID in the preferredaddress transport parameter. So a server can use a connection ID during connection establishment, then remove privacy protections by moving to a unique address (see ). Of course, this is largely moot, as the privacy protection offered through the use of connection IDs toward a unique address is nil. In either case, we should clarify whether preferredaddress is exactly like NEWCONNECTIONID, or whether it has its own rules. Right now, it looks like it has its own rules and that is likely bad.
Zero-length connection IDs don't have a stateless reset token associated with them, but the preferred_address TP requires you to set a token. That's an awkward corner case.
That is just one of the strange things, yeah. We should clarify that.
So I see a number of ways forward here: Do nothing. Leave this ambiguous and potentially special. Make the fields in preferredaddress exactly equivalent to a NEWCONNECTIONID with a sequence number of 1. But allow the connection ID to be zero-length. A zero-length connection ID provides a new stateless reset token that is bound to the new address. (This is a weird option that would go against previous decisions to eliminate special handling for this connection ID, but I include it for the sake of completeness.) As 2, except that the new stateless reset token has connection-wide scope such that either of the provided tokens can be used. This is more consistent with established principles. Prohibit the use of zero-length connection IDs in preferredaddress. That is, you can't use preferredaddress without a connection ID. That is, preferredaddress effectively is a new server address plus a NEWCONNECTIONID frame. Something else. From my perspective, I rank these as 4 > 3 >> 1 > 2. But that is largely based on my bias against zero-length connection IDs (as much as it seems like that might be good for privacy, I think that privacy is better served by the use of connection IDs). If people intend to use preferred_address to migrate to a connection-specific address, then I'm OK with 3; we already have to cover the privacy hole that arises from that.
My preference goes to option 4 too. I do not think servers would use both 0-byte CID and preferredaddress TP at the same time. The only case a server using 0-byte CID can also use preferredaddress is: when the server is never accepting more than one connection at a time, or when the server knows that the client's address tuple does not change when the client migrates to a new address IMO that's a niche. And when these cases can be met, I do not see why the client cannot know the preferred address of the server beforehand.
I forgot to add: There is (maybe) a case for using a connection ID during setup and a zero-length connection ID after migrating to a preferred address. That isn't entirely crazy, but I personally would be comfortable losing that option. But I don't operate a server, so I want to give those who do a chance to let me know that this is preemptive.
NAME My personal take would be that people would not do that. When a client migrates to the preferred address, there's fair chance that the client's address tuple will change. That means that the server should be capable of receiving short header packets with 0-byte CID from any address, and determine that the packet belongs to the connection that is migrating to the preferred address. That's possible theoretically, but in practice ... I agree that a server might want to switch to zero-byte CID after migration to SPA completes. But to do that, we need a NEWCONNECTIONID that can carry a zero-byte CID, which is forbidden at the moment. I do not think we'd want to change the spec now to allow that.
That's an excellent point, and a good reason not to allow changing to zero-length connection IDs in combination with preferred_address. The nasty part of that is that in most cases that design would work because clients won't use the "new" connection ID until after the handshake is complete. Of course, it would fail spectacularly if a client did decide to switch earlier - and the client might attract the blame, where it is clearly the fault of the server.
I'll note that prohibiting a 0 length CID doesn't actually ensure the server's IP isn't unique, though it does remove an incentive to make it unique. I'd like to leave the option to switch to a zero-length CID later in the connection, and I think I'd be fine with 4 if we allowed that. An easy example is a large upload. I mostly don't care about the CID, but if I'm uploading a 10GB file, maybe saving some bytes matters. Also, if you're constantly filling the pipe, NAT rebinds are extremely unlikely. Another example is an environment like a datacenter where there are no NATs and routes are very stable, unless a peer intentionally changes it, so the 5-tuple is sufficient. In that case, the ideal operating condition is to typically use a 0 byte CID and switch to a non-0 length anytime a peer changes IP or port. Changing port has some clear benefits in some cases, so it's in no way a threoretical use case. NAME may have thoughts on this range of use case as well.
In our implementation, we plan to do all routing based purely on the CID. IMO, the couple of byte overhead in a 1500 MTU (at least; in datacenter) packet isn't going to make much difference, so it's just simpler to have a common CID routing scheme. That being said, I personally am not against allowing for others to do something different. I've never understood why we make any restrictions on changing between 0-length and non-0-length CIDs at all. IMO, the complexity is that it can change length at all, not that it's zero or not.
NAME we already prohibit changing from non-zero-length to zero-length using NEWCONNECTIONID. I don't want to relitigate that. I want to concentrate on this specific issue.
If we're not going to re-litigate that, then I'd suggest we continue allowing a 0-length CID for the preferred address case.
NAME While I would not argue strongly against continue allowing that, I would appreciate it if you could clarify why allowing a zero-byte CID would be useful. As pointed out in URL, when you provide a zero-byte CID in SPA, you need to be prepared for receiving a short header packet using zero-length CID from any IP address, and be capable of associating that to an existing connection. This is because the client (or the network) might change the client's address tuple when it sends packets to the preferred address. I would anticipate that you wouldn't want to do that in your file upload case (or in any deployment that might accept multiple connections concurrently).
NAME If I know 5-tuples are stable, then using a zero-length CID is always fine, just like it's typically fine in the server to client direction. But in that case, hopefully I can avoid needing to use SPA, making the decision on this issue moot. After thinking about this more, we're probably better off with option 4 for the sake of consistency and if we want to allow 0-length CIDs in more cases, I can write an extension for how that would work, since I don't think it's only a matter of relaxing some constraints.
Thanks all. I think that we reached the same conclusion in the end, via different paths. The proposed fix here is firmly design chairs. A bugfix only, but still worth careful review.
Apologies for getting to this late, but I'm not sure I agree that we should disallow this. Here's a use-case that I think makes sense: the client wants to upload a very large file to the server over HTTP/3 and it has a small MTU, making it particularly sensitive to per-packet overheads. Let's also assume that we are in the future and the server supports IPv6 and has its own /64 (shocking!). In this scenario, a good way to reduce per-packet overhead is to operate the server on a given address, and then switch each client to their own unique server IP using preferred address, and then use zero-length CIDs. I've always found this use-case a good use of preferred address, and I would rather we didn't ban it without a good reason (and making the design cleaner doesn't necessarily count as a good reason at this stage in the process).
That is an example we discussed. As Kazuho noted, that doesn't work as we have already agreed that any connection ID (in this case, a zero-length one) can be used prior to handshake completion. That's why we ended up on 4. As Ian noted, this isn't ideal, but this is an area that is ripe for extensions that might allow this sort of thing.
I don't understand what you meant by the following: Can you clarify? I was under the impression that the client switches to the preferred address after the handshake is done.
Yes, you can't migrate (act on the address information) before the handshake is confirmed. However, the connection ID is not special; it can be treated just like any other connection ID you learn, so you can act on the connection ID information before you confirm the handshake. So that means that a server that opts to include a zero-length connection ID in the preferred_address might receive a packet with the zero-length connection ID at the old address. That might not work out.
What's the point of including a connection ID (and stateless reset token) in the transport parameter if that CID isn't tied to the preferred address in any way? At that point might as well remove that CID entirely and rely on NEWCONNECTIONID frames. My understanding was that the reason for there being a CID in the TP was specifically to make that CID special. Would it make sense to just say that this CID MUST NOT be used when sending to the handshake server IP?
the reason for having a connection ID in the transport parameter is only so that the client has a connection ID to use for the new path
NAME I agree with your intuition, but I think we discussed this in Zurich and ended up concluding the preferredaddress CIDs weren't special. This was in issue and also came up in (which I realize hasn't gone out for a consensus call yet, that PR seems stalled?). In the upload example, the server needs to know that the client is going to send a lot of data, which it might know via SNI, or it might not. I tend to think the more annoying restriction is that you can't change from a non-0 length CID to a 0-length one and vice versa. By supplying a 0-length CID later in the connection along with other CIDs, the client could choose whether it wanted to use a 0-length CID at any time. Thinking out loud, an easier solution may be a TP like 'acceptemptyconnectionid' that indicates at anytime, the peer can switch to a 0-length CID if it wants. It could either think the 5-tuple is stable or its happy re-establishing the path with a non 0-length CID after a NAT rebind. In this world, the Connection ID is more of a path re-establishment token.
Thanks for the details NAME and NAME I get a sense I might be trying to re-litigate past consensus without providing new information, so I'll back down. I do think that the ghost of hypothetical linkability has caused us to make some very odd design choices, but this isn't new to this issue/PR. In the end, this is the kind of feature that can be enabled by extensions so I can live with us further restricting the base spec for consistency.

connection ID when the client initiates migration to the preferred address. <ins> The Connection ID and Stateless Reset Token fields of a preferred address are identical in syntax and semantics to the corresponding fields of a NEW_CONNECTION_ID frame (frame-new-connection-id).  A server that chooses a zero-length connection ID MUST NOT provide a preferred address.  Similarly, a server MUST NOT include a zero- length connection ID in this transport parameter.  A client MUST treat violation of these requirements as a connection error of type TRANSPORT_PARAMETER_ERROR. </ins> The active connection ID limit is an integer value specifying the maximum number of connection IDs from the peer that an endpoint is willing to store.  This value includes the connection ID received


approach 2^(-L/2), that is, the birthday bound.  For the algorithms described in this document, that probability is one in 2^64. <del> In some cases, inputs shorter than the full size required by the packet protection algorithm might be used. </del> To prevent an attacker from modifying packet headers, the header is transitively authenticated using packet protection; the entire packet header is part of the authenticated additional data.  Protected

Now that we require a minimum packet size, this is no longer true.
Is this no longer true for the algorithms defined in the spec, or no longer true for any packet protection algorithms for any AEAD? It might be worth moving the note to the end of 5.4.1 instead.

approach 2^(-L/2), that is, the birthday bound.  For the algorithms described in this document, that probability is one in 2^64.  To prevent an attacker from modifying packet headers, the header is transitively authenticated using packet protection; the entire packet header is part of the authenticated additional data.  Protected


A client that is unable to retry requests loses all requests that are in flight when the server closes the connection.  An endpoint MAY <del> send multiple GOAWAY frames indicating different identifiers, but MUST NOT increase the identifier value they send, since clients might already have retried unprocessed requests on another connection. Receiving a GOAWAY containing a larger identifier than previously received MUST be treated as a connection error of type H3_ID_ERROR. </del>  An endpoint that is attempting to gracefully shut down a connection can send a GOAWAY frame with a value set to the maximum possible

The endpoint sends frames, which carry -- not send -- identifier values.
The way it's intended to parse, "An endpoint ... MUST NOT increase the value they send," is probably incorrect in number or gender. But yes, changing the verb to reflect that the subject is the frames instead of the endpoint is also an option.
OK. I have no preference how this sentence is fixed -- as long as it's fixed.
This looks good. While we are at this paragraph, why say "client?" GOAWAY can be sent by either side now.
It can, but a server isn't going to retry push streams on a new connection; it's going to decide de novo whether to push there. Arguably, nothing breaks if the client increases the Push ID on the server after the fact.
BTW, thanks for catching this. I appreciate how many of these ambiguities you're winkling out.
NAME NAME OK, lgtm, then.
Maybe, given the ambiguity, the right answer is to get rid of "they" in the first place?

A client that is unable to retry requests loses all requests that are in flight when the server closes the connection.  An endpoint MAY  <ins> send multiple GOAWAY frames indicating different identifiers, but the identifier in each frame MUST NOT be greater than the identifier in any previous frame, since clients might already have retried unprocessed requests on another connection.  Receiving a GOAWAY containing a larger identifier than previously received MUST be treated as a connection error of type H3_ID_ERROR. </ins> An endpoint that is attempting to gracefully shut down a connection can send a GOAWAY frame with a value set to the maximum possible


receipt by sending one or more ACK frames containing the packet number of the received packet.  13.2. Endpoints acknowledge all packets they receive and process.  However,

This PR adds explicit context around the choice of ACK frequency in the draft. It also takes in Bob Briscoe's suggestion in , of calling it a guidance in light of the best information we have. Also does some editorial cleanup of text blocks that were not in the right sections.
Thanks NAME NAME -- comments incorporated.
As currently specified, the default QUIC ACK policy models TCP’s ACK policy, but consumes significantly more return path capacity. This can impact resource usage of the return channel, can constrain forward throughput, and can impact the performance of other sessions sharing an uplink (draft-fairhurst-quic-ack-scaling). This was not discussed in Zurich - where discussion moved instead to the separate topic of draft-iyengar-quic-delayed-ack. In many current network segments (e.g., cable networks, mobile cellular, WiFi and other radio tech), the effects of asymetry are presently mitigated by network devices RFC3449 that rely upon observing TCP headers. This results in ossification problems, but does practically have beneit. It reduces the TCP ACK rate over the constrained link, it reduces consumption of "radio" transmission opportunities, and reduces traffic volume from ACKs. These mitigations are not possible, nor desirable for QUIC. However, the default performance of QUIC is therefore much worse than for TCP. In QUIC, the default ACK Ratio has to set by the endpoints implementing the protocol because the sender and receiver have no information about the "cost" of sending an ACK using a link along an asymmetric path. These endpoints can also choose a non-default value to enable a new CC, scale to higher rates, etc (e.g. draft-iyengar-quic-delayed-ack). We have shown there can be significant benefit from an ACK Ratio of 1:10 and this is expected to alos have benefit for higher levels of asymmetry. The proposed change from 1:2 to 1:10 is predicated on QUIC's design decisions: starting a connection with an Initial Window of 10 is acceptable, QUIC uses PTO as the loss detection method (a predicate to issue ), and QUIC has some form of pacing at the sender, at least to mitigate IW bursts.
Even though we've avoided straying this far from the TCP spec, I think allowing this would be good for the internet. It should probably be documented for TCP, too. How does ack stretching of this nature work for BBR bandwidth estimation? It's harder to use the ack pacing, no?
I don't believe QUIC ACKs are significantly larger. In fact, I believe the ACK frame itself is typically smaller than TCP ACKs, but the savings there are used on QUIC's 16-byte auth hash and likely a non-zero-length CID. Do you have a size analysis written up somewhere? I think this is the core issue. It's not that QUIC ACKs are really consuming significantly more capacity in terms of bytes on the wire, it's that middleboxes can't decide to thin the ACKs, because they don't know which packets are ACK-only packets. Did you implement the optimization already described in the transport draft?: "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." NAME did and I believe found it hugely reduced the number of ACKs, sometimes too much. As I've said before, I would really prefer not specify an approach in QUIC transport that is a heuristic like does. In particular, on lower bandwidth networks, 10 packets may be the entire congestion window and is too large a value. On higher bandwidth networks, every 20 packets has shown to be slightly better than 10, at least when using BBR. I wrote to add some editorial text about why every 2 packets is a SHOULD, not a MUST. Chrome did not default enable the ACK strategy recommends until BBR was launched, because there was a performance regression with Cubic. I'd rather not assume everyone is using BBR or a BBR-like congestion controller. NAME BBR does fine with stretch ACKs. Cubic and Reno suffer in some cases, however.
QUIC ACKs are larger! The smallest IPv4 QUIC ACK packet is 51 bytes. The ACK frame is 4 bytes, but is sent using a QUIC packet. For IPv4 this is: 20 (IP) + 8 (UDP) +3 (1+1+1 QUIC header ... assuming a 1 B CID) + 4 (minimum ACK frame size)+ 16 (crypto). Other types of frames may also be sent in the same packet, increasing this size. That size difference matters. The point is that QUIC ACKs are consuming more capacity in terms of bytes on the wire, and often also important: in more transmission opportunities at the physical layer. That makes QUIC perform more poorly than it needs to across some paths. The text "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." Is permissive to do something different - alas the effect of ordering of the processing depends on rate, timing, etc - The present text is not a recommendation to do something appropriate, which I think it needs to. If this were to describe that a receiver SHOULD reduce the ACK to data packet ratio to 1:10, then that would help. On low bandwidth links, then I agree that waiting for 10 packets would not be suitable, but then wouldn't the rule 4 ACKs/RTT still cover this case? (I have no problems believing every an ACK 20 packets can be slightly better than 10 in some cases, but then we have chosen IW at 10 and that relates to bustiness and responsiveness of the basic spec, so I think 1:10 makes more sense as the recommendation. If the CC and path information suggest 1:20 - perhaps as the rate increases, then a sender can "tune" as per the transport parameter proposal. That I STILL assert that this is a different issue from the expected default.)
Would it help to give a recommendation like, if the congestion window is large, the ACK radio should be reduced? Not sure if we can easily give a threshold for the window but to align with the 4 ACKs per RTT rule we could say 40 or to be more conservation maybe 100?
If the receiver knew the cwnd... we could add that. The current proposal we made is here (along with some of the arguments about why QUIC and TCP are slightly different): draft-fairhurst-quic-ack-scaling
So the min TCP ack is 40 bytes vs QUIC at 50(0 byte CIDs are common, so I'll assume that), 25% larger in the minimum case. However, with a few SACK blocks, TCP ACKs are likely to be larger than QUIC ACKs, since SACK consumes 10 bytes for the first block and 8 for each additional block. By the time the second block is added, there's a good chance QUIC ACKs are smaller and it's almost certainly true with 3 SACK blocks(which I think is a typical number). So I'll go back to my previous statement that the real issue is that the network can't thin QUIC ACKs effectively.
May be "yes", may be "no" on a larger size after loss - since QUIC ACK ranges are often larger - but that depends on loss pattern, etc. In our case, we saw more bytes. But "yes" on me saying there is a need for QUIC to make good default decisions about the ACK rate, because QUIC becomes responsible for transport working across heterogeneous paths and we need to get good defaults rather than rely on "ACK-Thining" and other network modification.
Anybody has a gut feeling how very low latency congestion control fares with sparse ACKs?. I have experienced that DCTCP is more picky with ACK aggregation than e.g Reno. I suspect that ACK-thinning can have the give the issue as well. A too sparse ACK spacing can also give problems. Another aspect is how e.g. TCP Prague with paced chirps work with sparse ACKs ?. To me the RTT/4 rule makes sense but I feel that the best alternative depends a lot on both the congestion control and the source material. Large file transfers are more trivial than sources that are occasionally idle.
I think this isn't the correct question. This is about the default policy. The default policy is about what a receiver does by "default" - the current text uses a value of ACK every other packet, 1:2. TCP specified this when transmission rates were low, and TCP depended on individual ACKs to clock data (no pacing, IW=2), detect loss (no PTO), increase cwnd (per ACK packet), etc. In response, TCP causes ACK congestion and strange interactions with radio access. Performance over WiFi, DOCSIS etc suffers unless the network tries to address this. I don't see why QUIC, which doesn't rely on any of these, needs to set a DEFAULT of 1:2. This is literally a few lines of code difference, and we could use 1:10. These CC's you mention may have different needs and that needs to be handled by telling the remote that your CC has these specific needs - which I think is what Jana's extension is about.
I am kind of the donkey in the middle... ACK thinning in networks should be discouraged. Standardization of ACK thinning mechanisms was proposed in 3GPP RAN2, this request was turned down mainly with reference to that it can harm transport protocol performance and does not work safely with QUIC. Nothing stops UE (terminal and modem) vendors from implement proprietary ACK-thinning but I certainly hope that 5G UE vendors don't attempt to thin QUIC-ACKs as this is a slippery slope. High ACK rates is a 5G problem mainly for high band NR (=5G at e.g 28GHz). It is mainly a coverage issue as UEs have a limited output power due to practical and regulatory requirements. At the same time one need to address that end users don't only download movies today and even VoD streaming implies uplink traffic in the form of chunk requests. And then we have an increasing amount of people doing uplink streaming or upload an infinite amount of pictures of food plates and cats :-) Operators and 5G vendors have to deal with this and solve the problem with deployment of 5G access in various frequency bands to get the best downlink and uplink performance. Reduced ACK rates is of course welcome but I don't currently see a problem that the initial ACK ratio is 1:2 , to increase to 1:? as the congestion window grows, bounded by e.g an RTT/4 rule. But here I need to admit that I at least I have not studied this in detail. Ergo.. I am not convinced that it is necessary to hardwire the QUIC spec to 1:10 , but I also cannot say that it should not be done.
This is a small enough change. However, there is still a bar for those, and the current level of discussion makes me believe we should close this with no action?
Just so that it's easy to find, there is data and a continued discussion at URL
Folks who care about a change here need to prioritize progressing this issue. We're close to being able to last-call the specs, and would like this resolved in time.
We’ve completed a round of updated analysis of the proposed change to the QUIC spec, using quicly (changing to a 1:10 policy), and chromium (which currently does 1:10 by default). Our hope is that this change to the spec, to actually make standard end-to-end QUIC work better than TCP over asymmetric paths! This is a short slide deck about the performance of the proposed change: URL There have been a number of other change made by the QUIC ID editors relating to ACK processing, so we wanted to be sure these results were up to date. (These results do not consider additional ACKs after re-ordering - reflecting the latest text removing that.) We also really hope this will motivate a change the practice of forcing of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths. People may also be interested in work for L4S to evaluate the impact of ACK Thining for TCP-Prague. Please do provide feedback, or any questions on this - as Lars notes, we think the time is running out for considering new changes, even relatively simple ones like this. Gorry P.S. First email: URL
I believe that it is beneficial to use a 1:10 policy, I understand that this means that a 1:2 policy that still be used in the initial phase (flow start) or does the timer mechanisms kick in to avoid that ACKs are transmitted too sparsely in this case ?.
Yes, our proposal counted ACK-eliciting packets at the receiver (because the receiver does not in general know the congestion state, so you need a heuristic), and we finally chose the same method in currect QUIC Chromium - i.e. for the first 100 packets we do not use ACK 1:10, changing only after 100 packets.
OK, it sounds like a good idea to use 1:10 after 100 packets or so. Just one potentially stupid question, Is there any special treatment necessary after an RTO ?
[This seems to be a forwarded comment from NAME Hi, Gorry Thank you for sharing your experiments. I'm delighted to see results showing that use of ack10 gives us comparable throughput to ack2. Recently, I have been looking into how ack-frequency affects the computational efficiency of the server [1]. Using the (almost) same setup, I have today run a benchmark comparing three ACK strategies: ack2, ack10, ack 1/8cwnd. ack2: 334Mbps ack10: 414Mbps ack 1/8cwnd: 487Mbps As can be seen, ack10 is better than ack2, but lags behind ack1/8cwnd. That's because there are still frequent ACKs. They take considerable amount of CPU cycles. If we want to see QUIC being as lightweight as TLS over TCP, it is critically important to reduce ACK frequency as much as possible. This is because the cost of processing ACK in QUIC is much more expensive than TCP, due QUIC being a userspace transport. Based on this, if we are to change what we have in the base drafts, I'd much prefer adopting what we have in the proposed delayed ack extension [2], rather than just changing the default ack policy from ack2 to ack10. By adopting what we have in the delayed ack extension, we can expect better performance, and the endpoints would have the freedom to change the behavior when the default ack policy is deemed insufficient (regardless of the default being ack2 or ack10). [1] URL [2] URL 2020428() 20:37 Gorry Fairhurst : issue. We're close to being able to last-call the specs, and would like this resolved in time. QUIC spec, using quicly (changing to a 1:10 policy), and chromium (which currently does 1:10 by default). Our hope is that this change to the spec, to actually make standard end-to-end QUIC work better than TCP over asymmetric paths! relating to ACK processing, so we wanted to be sure these results were up to date. (These results do not consider additional ACKs after re-ordering - reflecting the latest text removing that.) of QUIC fallback to TCP (with PEP/Thining) over highly asymmetric paths. People may also be interested in work for L4S to evaluate the impact of ACK Thining for TCP-Prague. think the time is running out for considering new changes, even relatively simple ones like this. URL
These results/article on computational efficiency are really interesting. Please see below. This is useful, we see here there's a clear benefit here in using a larger ACK Ratio also in terms of server load. We can see already the benefit at 1:10 and GSO helps as expected. We think that this is also really true. QUIC ACKs are also larger than TCP ACKs (see URL). I don't think the two approaches conflict. The difference between the two approaches is that the small change to the base specification to use an ACK Ratio 1:10 ensures the default return path is at least as good as TCP for common traffic (even with common TCP ACK Thinning). Without this change, the end user could experience significant differences in performance between TCP and between different QUIC versions over asymmetric paths. Whereas to me, the transport parameter extension looks like an exciting way to enable many more things, but there are also many ways this could usefully evolve based on CC, offload, pacing etec and I think this will take much more to work-out what is needed, safe and best ... then to reach consensus. We still also need to specify a default ACK Ratio in this case as well. Best wishes, Gorry
I'm greatly in favour of draft-iyengar-quic-delayed-ack-00 (BTW, where would you prefer me to give review comments about that?) I am much less keen on any default delayed ACK policy, whether 1:2 as now in quic-transport, or the 2-10/100 numbers proposed in draft-fairhurst-quic-ack-scaling-02 My concerns would melt away if the default policy was explicitly /implementation-dependent/ and the draft instead gave /guidance/ to implementers. The draft could still give the same 2-10/100 policy, but as /guidance/ for the current public Internet. The draft should also state the conditions under which this guidance was determined (e.g. Internet paths with BDP between x and y, or whatever). What's the difference? Stating a default policy implies this is a matter of interoperability. It's not. If we change to giving guidance, I'm sure most implementers will just use the guidance. However, implementers will be free to evolve their default as they gain experience. This reduces the possibility that other behaviours will evolve around any default like 2-10/100. For instance, network monitoring using it as a signature. Or congestion controls optimizing around it. In turn, this will reduce the chance of ossification around the first numbers we thought of. Senders can always use the ACK frequency frame to ask a receiver to change from its default. Then receivers will gradually learn what senders prefer them to do, and change their default accordingly. IMO, a continually improving default is important for short flows, so senders can avoid requesting a different ACK frequency right from the start.
Need to admit that I have not had too much time to follow the QUIC work lately. I too believe that draft-iyengar-quic-delayed-ack-00 is a good long term solution that allows endpoint to set a desired ACK frequency that is not more than necessary. As experiments show there are good incentives to keep the ACK frequency as low as possible in terms of client/server load and asymmetric links limitations. An ACK-FREQUENCY frame will allow to find the proper balance between the requirements given by the congestion control, the traffic type and the above given limitations. I am not too concerned if it takes an extra development/standards cycle to have that in place in the standard and in running code as long as the QUIC community shows other SDOs that the challenge with QUIC ACKs is being addressed.
NAME NAME could you please trim the quoted text when you reply by email?
(I trimmed the comments, but forwarding inline replies to email messages doesn't work well.)
Given that this is an area where there is still obviously experimentation and refinement, NAME comment seems on point; baking a specific algorithm as requirements into the specifications doesn't seem like a good idea (although putting constraints on endpoints might make sense). Can we focus on general guidance rather than documenting a specific algorithm? What would that PR look like?
+1 to NAME suggestion. If we want to make changes for v1, they need to happen now, and they must be in a form that can let us get consensus.
To be clear, when I said that having a default policy is not a matter of interoperability, that's only strictly true if QUIC supports something like draft-iyengar-quic-delayed-ack for the sender to ask the receiver to use a certain policy. Bob
I've proposed more context and text around our current guidance in the draft, see .
I believe that the text around asymmetric links look fine, not really necessary to write up much more. What can be good to know that for the 5G case with limited UL (uplink = modem->base station transmission) , the pain that a flow with an unnecessary high ACK rate causes is on the flow itself or other flows sharing the same access.
There are some different points of pain here, the default and the need to adapt to the endpoint: An ACK Ratio (AR 1:1): is the finest granuality of feedback. A QUIC AR of 1:1 would consume return path capacity proportional to about 6% of the forward traffic (assuming no loss, etc). This is more than TCP (about 3%). An ACK Ratio (AR 1:2): For a return path bottleneck, QUIC with an AR of 1:2 will still have more impact on sharing flows than TCP would with the same AR 1:2 (because QUIC ACKs are larger, and because networks do not currently thin QUIC ACKs). So QUIC needs to choose the default - is it 1:2, 1:4, 1:8, 1:10? A QUIC AR 1:10 results in an ACK traffic volume of about 1% of the forward traffic. To me, that's a small proportion, which is why I argued that 1:10 is a good recommendation. The default is important when: the endpoints have no knowledge of the asymetry or the impact they have on a link's layer 2 loading, or the endpoints need to reduce the collateral damage to other IP flows sharing the return path capacity. Negotiation: The AR can be driven by the application or endpoint - e.g., a higher rate connection may well have a benefit from an AR of 1:10, 1:20, 1:100, etc - but such ARs are seldom chosen to reduce the impact on sharing a return bottleneck, more likely they are chosen to benefit the performance of an application or endpoint. That consideration is different, and likely depends on the application or ednpoint - which to me is where the flexibility offered by a transport parameter is so important. draft-iyengar-quic-delayed-ack will still need guidance on the "default". I argued that the default policy impacts the congestion behaviour when there is appreciable asymmetry this causes at layer 2 or the collateral damage (congestion impact on sharing flows). Sure, the choice of 1% is just a number I said was reasonable, but I've seldom encountered L2 technologies where less than 1% of forward traffic would be significantly better.
NAME : I understand your position. While I agree that ack thinning happens with TCP, it is not what I expect to find on a common network path. And as far as I know, the endpoints still implement the RFC 5681 recommendation of acking every other packet. What you are arguing is that with ack thinners in the network, TCP's behavior on those network paths is different. However, I do not believe we understand the performance effects of these middleboxes, that is, how they might reduce connection throughput when the asymmetry is not a problem. Importantly, I don't think we should be specifying as default what some middleboxes might be doing without fully understanding the consequences of doing this. Please bear in mind that Chrome's ack policy of 1:10 works with BBR, not with Cubic or Reno. I do not believe there is a magic number at the moment, and as NAME noted, even 1:10 is not a small enough ratio under some conditions. Given this, and given that we know 1:2 is used by TCP endpoints, it makes the most sense to specify that as the suggested guidance. The new text proposed in describes the tradeoff involved, and the rationale for using 1:2. I am strongly against doing anything substantially different than what we have in the draft now without overwhelming information supporting a different position.
Agree with NAME 1:2 seems a reasonable default, and there seems to be considerable interest in draft-iyengar-quic-delayed-ack, which has a better proposal for stacks that care.
I think the new ACK text is improving the specification, but I'd like to be sure we have thought about this decision. Let me try to carefully respond on-list here, to see where we agree/disagree: I disagree, I think we shouldn't be perpetuating a myth that a sender receives a TCP ACK for every other packet. Even in the 90's implementations TCP stacks often generated stretch-ACKs for 3 segments. Since then, limited byte counting was introduced and significantly improved the sender's response to stretch ACKs, and that has been good for everyone. Senders still rely on byte-counting in TCP to handle Stretch-ACKs, and this need is not decreasing: new network cards reduce per-packet receive processing using Large Receive Offload (LRO) or Generic Receiver Offload (GRO). I am saying many TCP senders actually do today see stretch-ACKs. There are papers that have shown significant presence of stretch ACKs, e.g., (H. Ding and M. Rabinovich. TCP stretch acknowledgements and timestamps: findings and implications for passive RTT measurement. ACM SIGCOMM CCR, 45(3):20–27, 2015.). A TCP Stretch-ACK of 3 or 4 was common in their datasets (about 10% of cases). In some networks, the proportion of Stretch-ACKs will be much much higher, since ACK Thining is now widely deployed in WiFi drivers as well as cable, satellite and other access technologies - often reducing ACK size/rate by a factor of two. I am intrigued and would like to know more, in case I missed something? Typical algorithms I am aware of track a queue at the "bottleneck" and then make a decision based on upon the contents of the queue. If there is no asymmtery, there isn't normally a queue and the algorithm won't hunt to discard an "ACK". But, maybe, there is some truth in being wary: Encrypted flows can also build return path queues (as does QUIC traffic). Without an ability to interpret the transport data (which nobody, including me, wants for QUIC), a bottleneck router is forced to either use another rule to drop packets from the queue (such as the smallest packets) or to allow a queue to grow, limiting forward direction throughput. I would say neither of these outcomes are great because the router is trying to fix the transport's asymetry. However, if QUIC causes this problem, I suggest some method will proliferate as QUIC traffic increases. I didn't find something in QUIC spec. that was a major concern. It looked to me like QUIC had learned from TCP how to handle stretch-ACKs. After that, we changed quicly to use 1:10 with Reno, and things worked well. Didn't NAME also use Reno? Sure, the deisgn of TCP recognised that AR 1:1 generated too much traffic, AR 1:2 for TCP resulted in an Forward:Return traffic ratio of ~1.5%, and QUIC increases this ~3%. However, as network link speeds increased, this proved too low for many network links and so, TCP ACK thinning came into being. This often reduces the TCP AR to around 1:4 ( and will do this without bias, whatever the outcome is.
NAME Just to clarify, in our experiment, we reduced the ACK frequency from the default (2) to after leaving slow start, then updated it every 4 PTO. I think that would work fine on a high-bandwidth, stable network. But I am not sure if it would work as well as ack2 on all network conditions. Anyways, I'd explain our experiment as an optimization of ack 2, as it was reduction of ack rate after leaving slow start. I think that that was a sensible choice, because IIUC having frequent acks during slow start is essential. Assuming that we'd prefer using ack 2 during slow start, the question is if we need to hard-code in the base draft how to optimize after leaving slow start. While I agree that continuing to use ack 2 after slow start is concerning for certain networks, I am reluctant to spending time on looking for such a fixed set of values across diverse networks (including much slower links or lossy ones). Rather, my preference goes to shipping the base-draft as-is, then moving on to finalize the ack-frequency frame. I think we can assume most if not all of the popular clients to support the ack-frequency frame. Then, it becomes less of a question what the default in the base-draft is.
Thanks kazuho, 1/8 or 1/4 cwnd seems reasonable, as does using Ack 2 for the first part of slow start ... that closed to what we proposed. And yes there will anyway be cases where any default does not work. If QUIC decides to retain Ack 2, can we at least capture that Ack 2 with QUIC doesn't have the same return path performance as Ack 2 with TCP - so that people know that less frequent Acts are something to think about from the start? I'm doing a talk in PANRG this week, and have decided to talk-through this topic (I was already going to talk about things related to return paths). That'll give people a view of some data - and a little thought about the implications this has for QUIC, and whether there is an incentive for network device to thin QUIC to restore performance.
NAME : I'd be in support of saying that ack thinning happens for TCP and cite those studies, and that in the absence of that, there might be throughput consequences for QUIC. I think we've said exactly that in other words in , but if we want to say more there, that's fine with me. That would make this an editorial issue, but I'm entirely happy to do more words there. I am against changing the recommendation here for another arbitrary recommendation. I think the right approach is to allow endpoints to specify this during a connection, which is where the extension is helpful. As NAME notes, our hope is that the extension gets widely deployed. If you want to make the argument for the extension to become part of the base draft, that's a different conversation, but that's not the one we're having here.
Attempting to summarise the current status of this issue as a chair: there appears to have been a fair amount of discussion about making a design change, with representatives both in favour and against. It appears there is no omnipotent default policy, choosing an optimal one depends on many factors. The editors have landed an editorial change that attempts to better describe the rationale behind the QUICS's current design and highlights some of the factors. There may be further editorial improvements but they no not need to block the design issue. The proposal therefore is to close this issue with no design change. We had prepared some slides for the Vancouver IETD to discuss QUIC's ACKs, and how to make QUIC work better than TCP in common asymmetric paths. This is a short slide deck that summarises what we have learned about ACKs in QUIC: URL It compares the ACK traffic of TCP with that of QUIC from the point of view of a path where QUIC's return traffic is somehow constrained or could impact other user traffic. We think that ACKs are a concern for various shared capacity return link (e.g., radio-based methods in WiFi or mobile cellular were ACKs consume spectrum/radio resource or battery; shared capacity satellite broadband return link or the uplink of a cable system). For TCP, such systems commonly involve a PEP or an ACK Filtering mechanism (which clearly isn't wanted or even possible for QUIC). The baseline QUIC performance (as per current spec) suffers over these paths. So our first question is: Do other people think it is important that the default specified for QUIC works well with asymmetric paths ? Gorry

receipt by sending one or more ACK frames containing the packet number of the received packet. <ins> An endpoint SHOULD treat receipt of an acknowledgment for a packet it did not send as a connection error of type PROTOCOL_VIOLATION, if it is able to detect the condition. </ins> 13.2. Endpoints acknowledge all packets they receive and process.  However,


receiver's max_ack_delay value in determining timeouts for timer- based retransmission, as detailed in Section 5.2.1 of QUIC-RECOVERY. <del> An ACK frame SHOULD be generated for at least every second ack- eliciting packet.  This recommendation is in keeping with standard practice for TCP RFC5681.  A receiver could decide to send an ACK frame less frequently if it has information about how frequently the sender's congestion controller needs feedback, or if the receiver is CPU or bandwidth constrained. </del>  In order to assist loss detection at the sender, an endpoint SHOULD send an ACK frame immediately on receiving an ack-eliciting packet

receiver's max_ack_delay value in determining timeouts for timer- based retransmission, as detailed in Section 5.2.1 of QUIC-RECOVERY.  <ins> Since packets containing only ACK frames are not congestion controlled, an endpoint MUST NOT send more than one such packet in response to receiving an ack-eliciting packet. An endpoint MUST NOT send a non-ack-eliciting packet in response to a non-ack-eliciting packet, even if there are packet gaps which precede the received packet.  This avoids an infinite feedback loop of acknowledgements, which could prevent the connection from ever becoming idle.  Non-ack-eliciting packets are eventually acknowledged when the endpoint sends an ACK frame in response to other events. </ins> In order to assist loss detection at the sender, an endpoint SHOULD send an ACK frame immediately on receiving an ack-eliciting packet


codepoint in the IP header SHOULD be acknowledged immediately, to reduce the peer's response time to congestion events. <del> As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response.  In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets. Packets containing PADDING frames are considered to be in flight for congestion control purposes QUIC-RECOVERY.  Sending only PADDING frames might cause the sender to become limited by the congestion controller with no acknowledgments forthcoming from the receiver. Therefore, a sender SHOULD ensure that other frames are sent in addition to PADDING frames to elicit acknowledgments from the receiver. </del>  An endpoint that is only sending ACK frames will not receive acknowledgments from its peer unless those acknowledgements are

codepoint in the IP header SHOULD be acknowledged immediately, to reduce the peer's response time to congestion events.  <ins> The algorithms in QUIC-RECOVERY are expected to be resilient to receivers that do not follow guidance offered above.  However, an implementer should only deviate from these requirements after careful consideration of the performance implications of doing so. </ins> An endpoint that is only sending ACK frames will not receive acknowledgments from its peer unless those acknowledgements are


be acknowledged, an endpoint MAY wait until an ack-eliciting packet has been received to bundle an ACK frame with outgoing frames. <del> The algorithms in QUIC-RECOVERY are resilient to receivers that do not follow guidance offered above.  However, an implementor should only deviate from these requirements after careful consideration of the performance implications of doing so. Packets containing only ACK frames are not congestion controlled, so there are limits on how frequently they can be sent.  An endpoint MUST NOT send more than one ACK-frame-only packet in response to receiving an ack-eliciting packet.  An endpoint MUST NOT send a non- ack-eliciting packet in response to a non-ack-eliciting packet, even if there are packet gaps which precede the received packet.  Limiting ACK frames avoids an infinite feedback loop of acknowledgements, which could prevent the connection from ever becoming idle.  However, the endpoint acknowledges non-ACK-eliciting packets when it sends an ACK frame. </del>  <del> An endpoint SHOULD treat receipt of an acknowledgment for a packet it did not send as a connection error of type PROTOCOL_VIOLATION, if it is able to detect the condition. </del>  <del> 13.2.2. </del>  When an ACK frame is sent, one or more ranges of acknowledged packets are included.  Including older packets reduces the chance of spurious

be acknowledged, an endpoint MAY wait until an ack-eliciting packet has been received to bundle an ACK frame with outgoing frames.  <ins> 13.2.2. </ins>  <ins> A receiver determines how frequently to send acknowledgements in response to ack-eliciting packets.  This determination involves a tradeoff. </ins>  <ins> Endpoints rely on timely acknowledgment to detect loss; see Section 5 of QUIC-RECOVERY.  Window-based congestion controllers, such as the one in Section 6 of QUIC-RECOVERY, rely on acknowledgments to manage their congestion window.  In both cases, delaying acknowledgments can adversely affect performance. On the other hand, reducing the frequency of packets that carry only acknowledgements reduces packet transmission and processing cost at both endpoints.  It can also improve connection throughput on severely asymmetric links; see Section 3 of RFC3449. A receiver SHOULD send an ACK frame after receiving at least two ack- eliciting packets.  This recommendation is general in nature and consistent with recommendations for TCP endpoint behavior RFC5681. Knowledge of network conditions, knowledge of the peer's congestion controller, or further research and experimentation might suggest alternative acknowledgment strategies with better performance characteristics. A receiver MAY process multiple available packets before determining whether to send an ACK frame in response. 13.2.3. </ins> When an ACK frame is sent, one or more ranges of acknowledged packets are included.  Including older packets reduces the chance of spurious


acknowledgments to be lost.  A sender cannot expect to receive an acknowledgment for every packet that the receiver processes. <del> 13.2.3. </del>  When a packet containing an ACK frame is sent, the largest acknowledged in that frame may be saved.  When a packet containing an

acknowledgments to be lost.  A sender cannot expect to receive an acknowledgment for every packet that the receiver processes.  <ins> 13.2.4. </ins> When a packet containing an ACK frame is sent, the largest acknowledged in that frame may be saved.  When a packet containing an


algorithm could cause spurious retransmits, but the sender will continue making forward progress. <del> 13.2.4. </del>  A receiver limits the number of ACK Ranges (ack-ranges) it remembers and sends in ACK frames, both to limit the size of ACK frames and to

algorithm could cause spurious retransmits, but the sender will continue making forward progress.  <ins> 13.2.5. </ins> A receiver limits the number of ACK Ranges (ack-ranges) it remembers and sends in ACK frames, both to limit the size of ACK frames and to


that would otherwise be non-ack-eliciting, to avoid an infinite feedback loop of acknowledgements. <del> 13.2.5. </del>  An endpoint measures the delays intentionally introduced between the time the packet with the largest packet number is received and the

that would otherwise be non-ack-eliciting, to avoid an infinite feedback loop of acknowledgements.  <ins> 13.2.6. </ins> An endpoint measures the delays intentionally introduced between the time the packet with the largest packet number is received and the


NOT include delays that it does not control when populating the Ack Delay field in an ACK frame. <del> 13.2.6. </del>  ACK frames MUST only be carried in a packet that has the same packet number space as the packet being ACKed; see packet-protected.  For

NOT include delays that it does not control when populating the Ack Delay field in an ACK frame.  <ins> 13.2.7. </ins> ACK frames MUST only be carried in a packet that has the same packet number space as the packet being ACKed; see packet-protected.  For


Note that the same limitation applies to other data sent by the server protected by the 1-RTT keys.  13.3. QUIC packets that are determined to be lost are not retransmitted

Note that the same limitation applies to other data sent by the server protected by the 1-RTT keys. <ins> 13.2.8. Packets containing PADDING frames are considered to be in flight for congestion control purposes QUIC-RECOVERY.  Packets containing only PADDING frames therefore consume congestion window but do not generate acknowledgments that will open the congestion window.  To avoid a deadlock, a sender SHOULD ensure that other frames are sent periodically in addition to PADDING frames to elicit acknowledgments from the receiver. </ins> 13.3. QUIC packets that are determined to be lost are not retransmitted


opening of either the send or receive side causes the stream to open in both directions. <del> An endpoint MUST open streams of the same type in increasing order of stream ID. </del> These states are largely informative.  This document uses stream states to describe rules for when and how different types of frames can be sent and the reactions that are expected when

The text in 2.1 is more complete, so we don't need this.
When I read 3.0 I was a bit confused, because when the discussion of streams takes place in section 2.1, the requirement on stream ID ordering is already mentioned, but the reminder of what the types are is fresh in the mind. Perhaps this should be revised.

opening of either the send or receive side causes the stream to open in both directions.  These states are largely informative.  This document uses stream states to describe rules for when and how different types of frames can be sent and the reactions that are expected when


a PATH_CHALLENGE frame is not adequate validation, since the acknowledgment can be spoofed by a malicious peer. <del> Note that receipt on a different local address does not result in path validation failure, as it might be a result of a forwarded packet (see off-path-forward) or misrouting.  It is possible that a valid PATH_RESPONSE might be received in the future. </del> 8.6. Path validation only fails when the endpoint attempting to validate

In early versions of path validation, there was a requirement that PATH_CHALLENGE be sent on the same path. This is, for various reasons, either unnecessary or harmful, so we no longer have that requirement. But there was some residual text that assumed that property. This removes a paragraph citing symmetry directly, and rewords text that assumes return reachability - a property that QUIC no longer provides mechanisms for.
In , we removed the requirement that PATHRESPONSE be sent/received on the same path that PATHCHALLENGE was sent. However, there is still text in the document that relies on this assumption. In 8.5, it explains that receipt of the PATHRESPONSE on another interface doesn't cause validation to fail, because "It is possible that a valid PATHRESPONSE might be received in the future." But receipt of the PATHRESPONSE, over any interface is valid, and causes the validation to succeed. In 9.2, it suggests using path validation to demonstrate return reachability, but since PATHRESPONSE need not be sent on the same path, path validation doesn't demonstrate that.
Note that sending the response on the old path can lead to failures in "break before make" scenarios. Shall we mention that?

a PATH_CHALLENGE frame is not adequate validation, since the acknowledgment can be spoofed by a malicious peer.  8.6. Path validation only fails when the endpoint attempting to validate


Receiving acknowledgments for data sent on the new path serves as proof of the peer's reachability from the new address.  Note that since acknowledgments may be received on any path, return <del> reachability on the new path is not established.  To establish return </del>  reachability on the new path, an endpoint MAY concurrently initiate <del> path validation migrate-validate on the new path or it MAY choose to wait for the peer to send the next non-probing frame to its new address. </del>  9.3.

Receiving acknowledgments for data sent on the new path serves as proof of the peer's reachability from the new address.  Note that since acknowledgments may be received on any path, return  <ins> reachability on the new path is not established.  No method is provided to establish return reachability, as endpoints independently determine reachability on each direction of a path.  To establish </ins> reachability on the new path, an endpoint MAY concurrently initiate  <ins> path validation migrate-validate on the new path.  An endpoint MAY defer path validation until after a peer sends the next non-probing frame to its new address. </ins> 9.3.


4.1.4. <del> As keys for new encryption levels become available, TLS provides QUIC with those keys.  Separately, as keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption level are available.  These events are not asynchronous; they always occur immediately after TLS is provided with new handshake bytes, or after TLS produces handshake bytes. </del>  TLS provides QUIC with three items as a new encryption level becomes available:

These two statements are - to the extent that I can best determine - redundant. One is better.
The first two sentences of QUIC-TLS, Section 4.1.4 clearly intend to say different things, but I can't figure out the distinction. Is there a way we could reword them to make them clearer?
I think that this is not separately at all. Let's reduce that to a single statement.

4.1.4.  <ins> As keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption level are available.  These events are not asynchronous; they always occur immediately after TLS is provided with new handshake bytes, or after TLS produces handshake bytes. </ins> TLS provides QUIC with three items as a new encryption level becomes available:


CONNECTION_CLOSE in response to any UDP datagram that is received. However, an endpoint without the packet protection keys cannot identify and discard invalid packets.  To avoid creating an unwitting <del> amplification attack, such endpoints MUST reduce the frequency with which it sends packets containing a CONNECTION_CLOSE frame.  To minimize the state that an endpoint maintains for a closing </del>  connection, endpoints MAY send the exact same packet. Allowing retransmission of a closing packet is an exception to the

We weren't very concrete in saying how endpoints generate packets with CONNECTIONCLOSE, particularly those that throw away keys. We have a rather vague requirement. In short, follow the same rules we established for the handshake. Wording this as an aggregate number allows for stochastic reactions and larger CONNECTIONCLOSE frames. This way, if you get a 25-byte packet and respond with a 200-byte packet, you can do that, but you have to respond to 3 in 8 or fewer in that way. Note that this limit only applies if the endpoint throws away decryption keys. Endpoints with keys aren't blind amplifiers.
closing period ({{draining}}) and send a packet containing a CONNECTIONCLOSE in response to any UDP datagram that is received. However, an endpoint without the packet protection keys cannot identify and discard invalid packets. To avoid creating an unwitting amplification attack, such endpoints MUST reduce the frequency with which it sends packets containing a CONNECTIONCLOSE frame. To This seems like a pretty vague requirement for a 2119 MUST. Like, I could only send it 9999/10000 packets and that would be compliant.
I think that it would be good to follow the 3x rule here also, but that's a non-editorial change.
Do folks agree the PR resolves this?
I think the PR resolves this and is a sensible choice.
Yup, the PR resolves this.
OK, labelling as such.

CONNECTION_CLOSE in response to any UDP datagram that is received. However, an endpoint without the packet protection keys cannot identify and discard invalid packets.  To avoid creating an unwitting  <ins> amplification attack, such endpoints MUST limit the cumulative size of packets containing a CONNECTION_CLOSE frame to 3 times the cumulative size of the packets that cause those packets to be sent. To minimize the state that an endpoint maintains for a closing </ins> connection, endpoints MAY send the exact same packet. Allowing retransmission of a closing packet is an exception to the


likely the receiver will be able to process all the packets in a single pass.  A packet with a short header does not include a length, so it can only be the last packet included in a UDP datagram.  An <del> endpoint SHOULD NOT coalesce multiple packets at the same encryption level. </del>  Receivers MAY route based on the information in the first packet contained in a UDP datagram.  Senders MUST NOT coalesce QUIC packets

This uses positive framing for the recommendation to use frames rather than packets. And it makes that recommendation directly, rather than implying it.
Here are three quasi-nits about S 12.2. Handshake, 1-RTT; see Section 4.1.4 of {{QUIC-TLS}}) makes it more likely the receiver will be able to process all the packets in a single pass. A packet with a short header does not include a length, so it can only be the last packet included in a UDP datagram. An endpoint SHOULD NOT coalesce multiple packets at the same encryption level. Is the implication here that you should instead pack all the frames into one packet? UDP datagram. Senders MUST NOT coalesce QUIC packets for different connections into a single UDP datagram. Receivers SHOULD ignore any subsequent packets with a different Destination Connection ID than the first packet in the datagram. Is there a valid reason to have different SCIDs? ({{packet-version}}), and packets with a short header ({{short-header}}) do not contain a Length field and so cannot be followed by other packets in the same UDP datagram. Note also that there is no situation where a Retry or Version Negotiation packet is coalesced with another packet. Should this be a MUST NOT?
Yes. See . I don't see why. It's just a fact, not a conformance requirement.
For (1) I think it would be helpful to say that.
LGTM, one suggestion.

likely the receiver will be able to process all the packets in a single pass.  A packet with a short header does not include a length, so it can only be the last packet included in a UDP datagram.  An  <ins> endpoint SHOULD include multiple frames in a single packet if they are to be sent at the same encryption level, instead of coalescing multiple packets at the same encryption level. </ins> Receivers MAY route based on the information in the first packet contained in a UDP datagram.  Senders MUST NOT coalesce QUIC packets


the PTO expiry, subject to address validation limits; see Section 8.1 of QUIC-TRANSPORT. <del> Peers can also use coalesced packets to ensure that each datagram elicits at least one acknowledgement.  For example, clients can </del>  coalesce an Initial packet containing PING and PADDING frames with a 0-RTT data packet and a server can coalesce an Initial packet containing a PING frame with one or more packets in its first flight.

This should resolve some parts of .
NAME If you wish, I will split this PR into small ones.
I have dropped the commit of "in flight".
I also changed into .
NAME The commit for "towards" has been dropped.
Thanks. BTW, that thing is an another odd English quirk.
Unfortunately, this PR bundles a bunch of unrelated changes. Some of them I like, some of them I don't.Thanks for the PR, NAME !

the PTO expiry, subject to address validation limits; see Section 8.1 of QUIC-TRANSPORT.  <ins> Endpoints can also use coalesced packets to ensure that each datagram elicits at least one acknowledgement.  For example, a client can </ins> coalesce an Initial packet containing PING and PADDING frames with a 0-RTT data packet and a server can coalesce an Initial packet containing a PING frame with one or more packets in its first flight.


outdated frame, such as a MAX_DATA frame carrying a smaller maximum data than one found in an older packet.  Upon detecting losses, a sender MUST take appropriate congestion control action.  The details of loss detection and congestion control are described in QUIC-RECOVERY.

Added changes to recovery as well.
By design, TCP is capable of handling late-arriving acks. If a TCP endpoint receives an ack for a byte range after declaring loss of that byte range, retransmission of that byte range is cancelled. In contrast, QUIC uses monotonically increasing packet numbers. That is definitely an improvement, but does have a downside. If loss detection is too sensitive, and if endpoints drop information regarding what had been sent in a packet at the moment that packet is declared lost, then it would not recognize the late acks. I think that we should suggest (even if not recommend) QUIC endpoints to recognize late-arriving acks, so that our loss detection can be as aggressive and as effective as TCP, if not better. (Please let me know if such text already exists. I am opening the issue because I could not find one)
I have to agree. We had to implement late acknowledgment by marking "lost" packets and retaining them. I believe that this was very helpful in reducing the amount of redundant information we send. Failing to retain packets that were marked as lost was also the cause of a bug. As the lost packet was removed from our outstanding calculation (a different value to the "in-flight" counter we use for congestion control), we believed that there were no packets outstanding and we didn't try to send anything when the next PTO fired. As the loss detection timer is shorter than the PTO timer, we believed that there was nothing to send often (and we don't check all sources of frames, which would have told us otherwise).
Agreed.
Yeah, we should have text on this if we don't already.
The resolution in introduces a new recommendation for senders, so this ought to be considered a design issue if we're happy with that resolution.
URL we have this text:
NAME Thank you for pointing that out. I missed that. I tend to think that we should adopt . As stated in my original comment, handling of late acks is the prerequisite of many of the normative requirements that exist in the recovery draft. Therefore, it is better to have a normative requirement for handling of late acks.
I'm not opposing adding new normative text. If new text is added, I think that the existing text should be aligned to it. In the PR, it recommends 3PTO, but the recovery draft A.1 says 1RTT.
FWIW, our implementation, as of a little while ago, tracks for the current PTO. If everything is working nicely, that limits our commitment. As we are sending probes, the time lengthens so that we send a probe before discarding state. If it takes two probes to get an ACK, we won't recognize that, but we will always catch an ACK for the last set of probes. And the state commitment is minimal unless we're forced to probe.
I'll note that the PR is against transport, but this issue is tagged as recovery.
Thanks for pointing out this inconsistency, NAME I would argue that this should be a PTO, since that allows for including the RTT variance in the waiting period. Using a PTO includes the maximum ack delay as well -- which may not be necessary -- but it's convenient to use the PTO period.
I wanted to clarify my comment on the PR about believing that 1 RTT is a preferable recommendation to 3*PTO. We definitely don't want to discard this information BEFORE the packet is lost or acknowledged. But after it's been declared lost, there's no need to wait longer than the reordering window to receive an acknowledgement, because an immediate ACK will be sent when the out of order packet arrives. I tend to think an RTT is a plenty long reordering window in practice. If nothing is inflight before and nothing sent after the packet is lost AND the late immediate ACK for the reordered packet is lost, then it could take longer than the reordering window to receive an ACK, but that seems like an edge case that's not very interesting.
NAME : I've changed it to 1 PTO. We've commonly used PTO as a time period for everything else in the transport, and so it's convenient to use PTO here too. Ultimately, this is the amount of time you're holding on to packets marked as lost, so the extra time waiting for a PTO (over an RTT) really shouldn't make any measurable difference. If you still think we should say RTT, I'm fine with that.
I'm fine with PTO, and I prefer PTO above RTT. There could be reasons where there are some packet reordering due to multiple paths being used (e.g., ECMP, clustered endpoint). In such case, the variance of latency becomes more like a constant independent of RTT. I think it is a good idea to recommend a value (i.e. PTO) that can cover such constant even if RTT is very small. PTO has that would cover that constant.
I'm not convinced I understand the rationale completely, but I'm ok with 1 PTO. I believe we should update the recovery draft in the same PR.
So, design?
We have text about this in recovery, but it's there was no normative text and it's in the appendix, so I think this is design.
Let's not thrash too much on this one.

outdated frame, such as a MAX_DATA frame carrying a smaller maximum data than one found in an older packet. <ins> A sender SHOULD avoid retransmitting information from packets once they are acknowledged.  This includes packets that are acknowledged after being declared lost, which can happen in the presence of network reordering.  Doing so requires senders to retain information about packets after they are declared lost.  A sender can discard this information after a period of time elapses that adequately allows for reordering, such as a PTO (Section 6.2 of QUIC-RECOVERY), or on other events, such as reaching a memory limit. </ins> Upon detecting losses, a sender MUST take appropriate congestion control action.  The details of loss detection and congestion control are described in QUIC-RECOVERY.


be acknowledged, an endpoint MAY wait until an ack-eliciting packet has been received to include an ACK frame with outgoing frames.  13.2.2. A receiver determines how frequently to send acknowledgements in

It's no longer normative and it's in a single section. I only moved text so far, and not as far as NAME suggested. I can go farther here or in a follow-up PR.
NAME I would like to merge this. WDYT?
Can we merge this soon. I'm worried about it developing conflicts.
Yep, NAME lost the chance to fix the PR. We are - of course - still open for editorial improvements in the form of new pull requests though.
calls them "exemplary" but 13.2.4 has MUST in it.
For reference, this refers to and in the editor's copy. I'm in favour of striking the exemplary note and saying that while the recovery draft is advisory, this document depends on implementations following something like what is recommended.
The specific MUSTs aren't really about the ACK algorithm per se, but about state endpoints will have to maintain for other reasons that are typically folded into your loss recovery code. If an implementation is unable to do duplicate detection for a packet number range, it MUST fail toward rejecting all packets, not toward accepting potential duplicates. An implementation must track the greatest packet number ever received, for use in recovering future packet numbers. In fact, the first requirement is discussed in 12.3 and could probably be dropped here. The second isn't actually called out (MUST) in 17.1, but could easily be moved there.
I think I'm inclined towards reorganizing the text and putting all the 'exemplary' parts into one section(ie: "Limiting ACK Ranges") and moving the normative text either into "Managing ACK Ranges" or elsewhere in the draft.
I thought I could clean this up fairly quickly, but I think a fair number of paragraphs need to move, including some of NAME suggestions. I'm happy to try to improve this in a few days if no one else has beaten me to it.
Anyone want to help NAME out and take this on?
Sorry about dropping this one. I'm fine with it, and happy to make editorials one later.Thanks Ian. When you lay it out like this, it seems so obvious. I checked that we don't have citations for {{ack-tracking}}, so this works nicely. If you care about the "MAY", then we can fix that in another PR.

be acknowledged, an endpoint MAY wait until an ack-eliciting packet has been received to include an ACK frame with outgoing frames. <ins> A receiver MUST NOT send an ack-eliciting frame in all packets that would otherwise be non-ack-eliciting, to avoid an infinite feedback loop of acknowledgements. </ins> 13.2.2. A receiver determines how frequently to send acknowledgements in


single QUIC packet.  If it does not, then older ranges (those with the smallest packet numbers) are omitted. <del> ack-tracking and ack-limiting describe an exemplary approach for determining what packets to acknowledge in each ACK frame.  Though the goal of these algorithms is to generate an acknowledgment for every packet that is processed, it is still possible for acknowledgments to be lost.  A sender cannot expect to receive an acknowledgment for every packet that the receiver processes. 13.2.4. When a packet containing an ACK frame is sent, the largest acknowledged in that frame may be saved.  When a packet containing an ACK frame is acknowledged, the receiver can stop acknowledging packets less than or equal to the largest acknowledged in the sent ACK frame. In cases without ACK frame loss, this algorithm allows for a minimum of 1 RTT of reordering.  In cases with ACK frame loss and reordering, this approach does not guarantee that every acknowledgement is seen by the sender before it is no longer included in the ACK frame. Packets could be received out of order and all subsequent ACK frames containing them could be lost.  In this case, the loss recovery algorithm could cause spurious retransmissions, but the sender will continue making forward progress. 13.2.5. </del> A receiver limits the number of ACK Ranges (ack-ranges) it remembers and sends in ACK frames, both to limit the size of ACK frames and to avoid resource exhaustion.  After receiving acknowledgments for an ACK frame, the receiver SHOULD stop tracking those acknowledged ACK <del> Ranges. </del>  It is possible that retaining many ACK Ranges could cause an ACK frame to become too large.  A receiver can discard unacknowledged ACK

single QUIC packet.  If it does not, then older ranges (those with the smallest packet numbers) are omitted.  A receiver limits the number of ACK Ranges (ack-ranges) it remembers and sends in ACK frames, both to limit the size of ACK frames and to avoid resource exhaustion.  After receiving acknowledgments for an ACK frame, the receiver SHOULD stop tracking those acknowledged ACK  <ins> Ranges.  Senders can expect acknowledgements for most packets, but QUIC does not guarantee receipt of an acknowledgment for every packet that the receiver processes. </ins> It is possible that retaining many ACK Ranges could cause an ACK frame to become too large.  A receiver can discard unacknowledged ACK


value than what was included in a previous ACK frame could cause ECN to be unnecessarily disabled; see ecn-validation.  A receiver that sends only non-ack-eliciting packets, such as ACK frames, might not receive an acknowledgement for a long period of time.  This could cause the receiver to maintain state for a large

value than what was included in a previous ACK frame could cause ECN to be unnecessarily disabled; see ecn-validation. <ins> ack-tracking describes an exemplary approach for determining what packets to acknowledge in each ACK frame.  Though the goal of this algorithm is to generate an acknowledgment for every packet that is processed, it is still possible for acknowledgments to be lost. 13.2.4. When a packet containing an ACK frame is sent, the largest acknowledged in that frame may be saved.  When a packet containing an ACK frame is acknowledged, the receiver can stop acknowledging packets less than or equal to the largest acknowledged in the sent ACK frame. </ins> A receiver that sends only non-ack-eliciting packets, such as ACK frames, might not receive an acknowledgement for a long period of time.  This could cause the receiver to maintain state for a large


send a PING or other small ack-eliciting frame occasionally, such as once per round trip, to elicit an ACK from the peer. <del> A receiver MUST NOT send an ack-eliciting frame in all packets that would otherwise be non-ack-eliciting, to avoid an infinite feedback loop of acknowledgements. </del>  <del> 13.2.6. </del>  An endpoint measures the delays intentionally introduced between the time the packet with the largest packet number is received and the

send a PING or other small ack-eliciting frame occasionally, such as once per round trip, to elicit an ACK from the peer.  <ins> In cases without ACK frame loss, this algorithm allows for a minimum of 1 RTT of reordering.  In cases with ACK frame loss and reordering, this approach does not guarantee that every acknowledgement is seen by the sender before it is no longer included in the ACK frame. Packets could be received out of order and all subsequent ACK frames containing them could be lost.  In this case, the loss recovery algorithm could cause spurious retransmissions, but the sender will continue making forward progress. </ins>  <ins> 13.2.5. </ins> An endpoint measures the delays intentionally introduced between the time the packet with the largest packet number is received and the


packets, the ACK Delay field in acknowledgements for those packet types SHOULD be set to 0. <del> 13.2.7. </del>  ACK frames MUST only be carried in a packet that has the same packet number space as the packet being acknowledged; see packet-protected.

packets, the ACK Delay field in acknowledgements for those packet types SHOULD be set to 0.  <ins> 13.2.6. </ins> ACK frames MUST only be carried in a packet that has the same packet number space as the packet being acknowledged; see packet-protected.


Note that the same limitation applies to other data sent by the server protected by the 1-RTT keys. <del> 13.2.8. </del>  Packets containing PADDING frames are considered to be in flight for congestion control purposes QUIC-RECOVERY.  Packets containing only

Note that the same limitation applies to other data sent by the server protected by the 1-RTT keys.  <ins> 13.2.7. </ins> Packets containing PADDING frames are considered to be in flight for congestion control purposes QUIC-RECOVERY.  Packets containing only