Delimiter-separated formats, such as HTTP/1.1. Common Message Framers can be provided by the Transport Services <del> implementation, but an implemention ought to allow custom Message </del>  Framers to be defined by the application or some other piece of software.  This section describes one possible interface for defining Message Framers as an example.

Just some points about the new text on framers in the API draft: In 10.2 it is specified in which order multiple framers run on incoming and outgoing messages, it is however never specified what the order looks like on Start and Stop events. In 10.6, "MessageFramer.DeliverAndAdvanceReceiveCursor" appears to be missing the "Data" argument that is present in "MessageFramer.Deliver". Also in 10.6, in the second paragraph it says: When it is probably meant to say While the framer implementation is able to notify the Connection that it has finished handling a Start event (by calling MessageFramer.MakeConnectionReady), there is no such way for Stop events.
As notes for the responses to these points: If there are multiple framers, the "lowest" one (the one closest to the transport) would receive start first, and the next would get the start event when the lower one is ready. Similar for stop. Good calls on the missing data argument and the typo re application vs framer. I was thinking to use the failConnection call to complete a stop, potentially with no error?

Delimiter-separated formats, such as HTTP/1.1. Common Message Framers can be provided by the Transport Services  <ins> implementation, but an implementation ought to allow custom Message </ins> Framers to be defined by the application or some other piece of software.  This section describes one possible interface for defining Message Framers as an example.


implementation to communicate control data to the remote endpoint that can be used to parse Messages.  At any time if the implementation encounters a fatal error, it can also cause the Connection to fail and provide an error.  Before an implementation marks a Message Framer as ready, it can also dynamically add a protocol or framer above it in the stack.  This allows protocols like STARTTLS, that need to add TLS conditionally,

implementation to communicate control data to the remote endpoint that can be used to parse Messages. <ins> Similarly, when a Message Framer generates a "Stop" event, the framer implementation has the opportunity to write some final data or clear up its local state before the "Closed" event is delivered to the Application.  The framer implementation can indicate that it has finished with this. </ins> At any time if the implementation encounters a fatal error, it can also cause the Connection to fail and provide an error. <ins> Should the framer implementation deem the candidate selected during racing unsuitable it can signal this by failing the Connection prior to marking it as ready.  If there are no other candidates available, the Connection will fail.  Otherwise, the Connection will select a different candidate and the Message Framer will generate a new "Start" event. </ins> Before an implementation marks a Message Framer as ready, it can also dynamically add a protocol or framer above it in the stack.  This allows protocols like STARTTLS, that need to add TLS conditionally,


Upon receiving this event, a framer implementation is responsible for performing any necessary transformations and sending the resulting <del> data to the next protocol.  Implementations SHOULD ensure that there is a way to pass the original data through without copying to improve </del>  performance. To provide an example, a simple protocol that adds a length as a

Upon receiving this event, a framer implementation is responsible for performing any necessary transformations and sending the resulting  <ins> data back to the Message Framer, which will in turn send it to the next protocol.  Implementations SHOULD ensure that there is a way to pass the original data through without copying to improve </ins> performance. To provide an example, a simple protocol that adds a length as a


discard a header once the content has been parsed). To deliver a Message to the application, the framer implementation <del> can either directly deliever data that it has allocated, or deliver a </del>  range of data directly from the underlying transport and <del> simulatenously advance the receive cursor. </del>  Note that "MessageFramer.DeliverAndAdvanceReceiveCursor" allows the framer implementation to earmark bytes as part of a Message even

discard a header once the content has been parsed). To deliver a Message to the application, the framer implementation  <ins> can either directly deliver data that it has allocated, or deliver a </ins> range of data directly from the underlying transport and  <ins> simultaneously advance the receive cursor. </ins> Note that "MessageFramer.DeliverAndAdvanceReceiveCursor" allows the framer implementation to earmark bytes as part of a Message even


A specialized feature could be required by an application only when using a specific protocol, and not when using others.  For example, <del> if an application is using UDP, it could require control over the checksum or fragmentation behavior for UDP; if it used a protocol to frame its data over a byte stream like TCP, it would not need these options.  In such cases, the API ought to expose the features in such a way that they take effect when a particular protocol is selected, but do not imply that only that protocol could be used.  For example, if the API allows an application to specify a preference to use a partial checksum, communication would not fail when a protocol such as TCP is selected, which uses a checksum covering the entire payload. </del>  Other specialized features, however, could be strictly required by an application and thus constrain the set of protocols that can be used.

Addresses
The architecture draft uses UDP checksums as an example of a protocol-specific property in Section 3.2 and 4.1.2. I find this a bit confusing because in the Interface draft, checksum coverage is a Generic Connection Property (Section 11.1.2). Should we maybe use a different example of a protocol-specific property, such as the TCP User Timeout we have in the Interface draft?
Sure, we can switch the example around
Thanks, LGTM!Thanks – works

A specialized feature could be required by an application only when using a specific protocol, and not when using others.  For example,  <ins> if an application is using TCP, it could require control over the User Timeout Option for TCP; these options would not take effect for other transport protocols.  In such cases, the API ought to expose the features in such a way that they take effect when a particular protocol is selected, but do not imply that only that protocol could be used.  For example, if the API allows an application to specify a preference to use the User Timeout Option, communication would not fail when a protocol such as QUIC is selected. </ins> Other specialized features, however, could be strictly required by an application and thus constrain the set of protocols that can be used.


Connection Properties: The Connection Properties are used to configure protocol-specific options and control per-connection behavior of the Transport Services system; for example, a <del> protocol-specific Connection Property can express that if UDP is used, the implementation ought to use checksums.  Note that the presence of such a property does not require that a specific protocol will be used.  In general, these properties do not explicitly determine the selection of paths or protocols, but can be used in this way by an implementation during connection </del>  establishment.  Connection Properties are specified on a Preconnection prior to Connection establishment, and can be modified on the Connection later.  Changes made to Connection

Connection Properties: The Connection Properties are used to configure protocol-specific options and control per-connection behavior of the Transport Services system; for example, a  <ins> protocol-specific Connection Property can express that if TCP is used, the implementation ought to use the User Timeout Option. Note that the presence of such a property does not require that a specific protocol will be used.  In general, these properties do not explicitly determine the selection of paths or protocols, but can be used in this way by an implementation during connection </ins> establishment.  Connection Properties are specified on a Preconnection prior to Connection establishment, and can be modified on the Connection later.  Changes made to Connection


will never show up when queuing the value of a preference - the effective preference must be returned instead. <del> Internally, the transport system will first exclude all protocols and paths that match a Prohibit, then exclude all protocols and paths that do not match a Require, then sort candidates according to </del>  Preferred properties, and then use Avoided properties as a <del> tiebreaker.  Selection Properties that select paths take preference over those that select protocols.  For example, if an application indicates a preference for a specific path by specifying an interface, but also a preference for a protocol not available on this path, the transport system will try the path first, ignoring the protocol preference. </del>  Selection and Connection Properties, as well as defaults for Message Properties, can be added to a Preconnection to configure the

Took a stab at normative language for the rules to process Selection Properties, as discussed in . As far as I remember, we always said that outcomes have to be deterministic/consistent, so I made everything that matters a MUST. I guess our example illustrates that even given the same Selection Properties, the outcome might change depending on the situation, e.g., your path might still be Wifi but it might or might not support your preferred protocol. So, does this really have to be all MUST, or is there a SHOULD in there? Discuss. :) Also, if anyone can name a good reason why the outcome has to be deterministic, please suggest text - maybe something like "avoid surprises for the application", but this didn't really sound right, so I left it out for now.
In section 5.2. in the API doc: "Internally, the transport system will first exclude all protocols and paths that match a Prohibit, then exclude all protocols and paths that do not match a Require, then sort candidates according to Preferred properties, and then use Avoided properties as a tiebreaker. Selection Properties that select paths take preference over those that select protocols. For example, if an application indicates a preference for a specific path by specifying an interface, but also a preference for a protocol not available on this path, the transport system will try the path first, ignoring the preference." This sounds rather like an example algorithm that should maybe go into the implementation doc instead? Or is there any part of this that we need to specify normatively in order to ensure that all systems implement the same?
This goes back to issue on "conflicting policies". At the time, we decided that there must be a deterministic outcome, therefore, text was added to both the API and the Implementation document, see . If we keep it that way, perhaps the text here should point out that the goal is to resolve conflicting policies, so we achieve a deterministic outcome.
Ah okay, thanks for the background info; missed that. I think we need to use normative language here then. Maybe also a separate subsection makes this more prominent.
This reads to me like the rules for processing polices rather than an example algorithm. I think it is part of the API... Normative language would be good and a stronger opening that explains why the rules exist.
I don't think a separate subsection would make this a better read - this fits fine as it stands, as an intro to the transport property specification IMO. Regarding normative language, that's a separate discussion. So, can we close this issue?
Agree that we don't need a separate subsection. I can try to tweak the beginning of this part a bit to explain why the rules exist as NAME suggested. Regarding normative language - we definitely have it in the API draft, right? So why not have the discussion on it here as well? if we're serious about deterministic outcomes, I guess the whole thing needs to be a MUST, right?
Sorry, true, that normative language debate was about the arch draft, and we definitely have it in the API draft. So yes, let's discuss it here. I also think a MUST would make sense.
So the order of filtering out require/prohibit is not important (outcome is the same) - making the order a MUST does not make sense. Also, first sorting by preference and then filtering has the same outcome as doing it the other way around. the only thing we need to say is that filtering MUST happen and sorting SHOULD happen.
ok - I have to admit I didn't think this through, I just agree that we want a deterministic outcome from this.
Why did we require a deterministic behavior in first place? Does this prevent TAPS implementations from running experiments like Apple did with TCP SACK and ECN? Does this prevent probabilistic path probing?
This was discussed in the TAPS BoF? ... I thought it was about developers being assured that X will happen if they choose Y and not finding the ground shifting as they use the interface.
Interim: Prefers and avoids allow the implementation to decide the ordering and weighting; require and prohibit are the only hard requirements. Applications should be able to cope with any possibility that isn't prohibited. Likely should add implementation text to say how preferences weight racing.
During the interim discussion, we found that was too strict and limiting, so we opted for another approach that moves the selection algorithm to the Implementation draft. I'm preparing another PR implementing these changes.
The text that ended up in the implementation draft is not fully consistent with the racing text, opening a new issue for this and closing this issue.
LGTM, merging this. thanks, Theresa!This reads good to me. Regarding the request for text, I don't have any ideas... but I think this could be fine as it stands? Needs more eyes though.

will never show up when queuing the value of a preference - the effective preference must be returned instead.  <ins> The implementation MUST ensure a consistent outcome given the same Selection Properties.  Internally, it MUST exclude all protocols and paths that match a Prohibit and exclude all protocols and paths that do not match a Require.  It MUST then sort candidates according to </ins> Preferred properties, and then use Avoided properties as a  <ins> tiebreaker. Note that the protocols and paths which are available on a specific system may vary, such that application preferences may conflict with each other.  For example, if an application indicates a preference for a specific path by specifying an interface, but also a preference for a protocol, a situation might occur in which the preferred protocol is not available on the preferred path.  In such cases, implementations MUST ensure a deterministic outcome by prioritizing Selection Properties that select paths over those that select protocols.  Therefore, the transport system MUST try the path first, ignoring the protocol preference if the protocol does not work on the path. </ins> Selection and Connection Properties, as well as defaults for Message Properties, can be added to a Preconnection to configure the


configuration details for transport security protocols, as discussed in security-parameters.  It does not recommend use (or disuse) of specific algorithms or protocols.  Any API-compatible transport <del> security protocol should work in a TAPS system. </del>

This ended up in more text that I thought. I'm not certain if we need all of it. Further I'm unsure about the concrete wording in many cases, so any comments are more than welcome!
This is supposed to address issue
I do mentioned name resolution, however, leakage though DNS might rather be a DNS problem and not necessarily an issue that needs to be considered in taps. Sharing cache state is a good point. I vaguely remember that we already address that somewhere else (could also be in the implementation draft as this is rather an implementation issue). But maybe we can add a pointer...?
Leakage though DNS may need to be addressed by TAPS, as with the current sockets API, the application has complete control about when DNS resolution is done (and, at least as long as things like DoH/DoT are predominantly implemented by nonsystem resolver libraries, how). TAPS as envisioned gives the system more control over resolution.
(IIRC the architecture does explicitly address and mention the shared cache issue; adding a line to refer to that in privacy considerations might be nice but isn't IMO necessary.)
Gorry, I addressed all your nits/comments.
I think it is a good start, but I miss two aspects: Information leakage through DNS, which can also make addresses on different paths linkable Information leakage within an application though shared caches, e.g., within a browser. I agree this is a good starting point; note that I took the liberty to directly fix some nits (such as capitalizing TAPS) and typos; these never change the intended meaning.

configuration details for transport security protocols, as discussed in security-parameters.  It does not recommend use (or disuse) of specific algorithms or protocols.  Any API-compatible transport  <ins> security protocol should work in a TAPS system.  Security consideration for these protocols should be discussed in the respective specifications. The desribed API is used to exchange information between an application and the transport system.  While it is not necessarily expected that both systems are implemented by the same authority, it is expected that the transport system implementation is either provided as a library that is selected by the application from a trusted party, or that it is part of the operating system that the application also relies on for other tasks. In either case, the TAPS API is an internal interface that is used to change information locally between two systems.  However, as the transport system is responsible for network communication, it is in the position to potentially share any information provided by the application with the network or another communication peer.  Most of the information provided over the TAPS API are useful to configure and select protocols and paths and are not necessarily privacy sensitive.  Still, there is some information that could be privacy sensitve because this might reveal usage characteristics and habits of the user of an application. Of course any communication over a network reveals usage characteristics, as all packets as well as their timing and size are part of the network-visible wire image RFC8546.  However, the selection of a protocol and its configuration also impacts which information is visible, potentially in clear text, and which other enties can access it.  In most cases information that is provided for protocol and path selection should not directly translate to information that is can be observed by network devices on the path. But there might be specific configuration information that are intended for path exposure, such as e.g. a DiffServ codepoint setting, that is either povided directly by the application or indirectly configured over a traffic profile. Further, applications should be aware that communication attempts can lead to more than one connection establishment.  This is for example the case when the transport system also excecutes name resolution; or when support mechanisms such as TURN or ICE are used to establish connectivity; or if protocols or paths are raised; or if a path fails and fallback or re-establishment is supported in the transport system. These communication activities are not different from what is used today, however, the goal of a TAPS transport system is to support such mechanisms as a generic service within the transport layer. This enables applications to more dynamically benefit from innovations and new protocols in the transport system but at the same time may reduce transparency of the underlying communication actions to the application itself.  The TAPS API is designed such that protocol and path selection can be limited to a small and controlled set if required by the application for functional or security purposes.  Further, TAPS implementations should provide an interface to poll information about which protocol and path is currently in use as well as provide logging about the communication events of each connection. </ins>


ConnectionError<> occurs when a Connection transitions to Closed state due to an error in all other circumstances.  The interface provides the following guarantees about the ordering of operations:

Would be nice if someone has time to do some ascii art :-)
Wouldn't that be fig. 4 in the architecture draft? Except that, looking at this now, I see inconsistencies! E.g., I don't see "RendezvousDone" in the figure. Also, instead of ConnectionReceived, it says "Connection Received" - rather, this should probably marked as an event by writing it as "ConnectionReceived<>". The figure is also missing InitiateWithSend() ... probably it would be good to make this figure both complete and consistent with section 13, "Connection State and Ordering of Operations and Events", of the API draft? I don't think very much is missing there.
I think it would be nice to have a real state diagram here, probably with states like "closed", "open" and "listen" and the events that create a transition between these state.
I'm not sure that Figure 4 in the Arch diagram serves the purpose that Mirja's trying to address with the state diagram in API. I also don't think we need FIgure 4 to align with the specific naming and CamelCasingConventions in API.
Thanks! I guess you could add a loop for sent<> and received<> in established state but maybe that also too much.Works for me

ConnectionError<> occurs when a Connection transitions to Closed state due to an error in all other circumstances. <ins> The following diagram shows the possible states of a Connection and the events that occur upon a transition from one state to another. </ins> The interface provides the following guarantees about the ordering of operations:


protocols and paths without requiring major changes to the application. <del> design explains the design principles behind the Transport </del>  Services API.  These principles are intended to make sure that transport protocols can continue to be enhanced and evolve without <del> requiring too many changes by application developers. </del>  concepts presents the Transport Services architecture diagram and defines the concepts that are used by both the API and

This proposal consolidates normative language into the section on design principles (now requirements), using the suggestions in .
Please review the latest updates!
Early art-art review of draft-ietf-taps-architecture My first observation and request is that the group rethink the separation of the architecture and interface documents. It is a warning-sign that the interface document is a normative reference for the architecture document. As the documents are currently constructed, a new reader cannot grasp the architecture without inferring some of it through reading the interface. Consider reformulating the architecture document as an overview - introduce principles, concepts, and terminology, but don't attempt to be normative. (There are signs that this may already be, or have been, a path the document was being pushed towards). If nothing else, rename it, and acknowledge that the actual architecture is specified using the interface definition. Note that the first sentence of the Overview in the interface document also highlights the current structural problem.
I don't entirely agree with this ART-ART comment: I think I might well agree with moving more of the description to this Arch document, but as discussed before, I also think it useful here to set the actual requirements for the architecture, and to define the architecture. It may need to be much clearer that this architecture consists of more the API.
For completeness, this is the first statement of the interface document review: This document reads fairly well. I have some structural concerns (see the early review of the architecture document). In particular I think this document contains the actual definition of the architecture and some of that is inferential.
Discussion at September 2021 interim: pass through this document with a view toward making it a high level introduction, not an architecture definition: "An Introduction to the Transport Services Architecture". remove normative language from this document (moving it to API, where necessary). move details not useful in a high level intro to API.
I disagree with this conclusion at first pass—we did a specific analysis and pass on this over a year ago, with PR . The normative language and the standards-track aspect was very intentional and something the WG had consensus on. The point is that the API is a specific incarnation of the TAPS architecture, but there are overarching architectural properties that go beyond a specific abstract API.
I completely agree with Tommy here, and recommend we fix this in . We should also of course check for any stray text or missing concepts. I think it would also be useful to somehow better explain that in figure 3 the API function is ONE part of the system.
See linked PR first please to see if this resolves this?
A PR was merged that addressed important text issues with separation of requirements and overview, and should more clearly set out that the requirements are for the system. The question of whether the overall TAPS systems requirements belong in this ID; the API; or a separate RFC needs some discussion based on the review, before this can be closed.
Many good text improvements in the PR NAME but I noticed two smaller updates in the PR that I think were incorrect. Fixed in
The review comment showed were many places where the wording of architecture can be improved. There are some high level principles that I think should remain in the architecture document. The decision in today's interim is to review the architecture to get this more into a shape that is valuable as a reference, and then check the updated draft.
I now suggest the issues are addressed and we close this issue without moving text between documents
I agree with that; closing.
Overall review: I think we need to keep the normative language for the key features that differentiate the API. I'd really like to use REQUIRED, RECOMMENDED, rather MUST and SHOULD - because these are not protocol operation requirements, but are implementation requirements to conform. This includes requirements based on Minset - where the WG dervied this API: This is after all why the WG worked on Minset as the basis of the API. I suggest we group all the requirements on architecture as short one or two sentence blocks (preferably as a bullet or definition list) collected into a single subsection, so they are easy to find and read and we say that these are the key architectural requirements. We insert this new requirements text either their own section, or in section 1.3 (although I am open to other places where we can call-out the architectural requirements). The following are key requirements, that define the TAPS architecture: It is RECOMMENDED that functionality that is common across multiple transport protocols is made accessible through a unified set of TAPS interface calls. As a baseline, a Transport Services system interface is REQUIRED to allow access to the minimal set of features offered by transport protocols {{?I-D.ietf-taps-minset}}. A Transport Services system automates the selection of protocols and features and network, based on a set of Properties supplied through the TAPS interface. It is RECOMMENDED that a Transport Services system offers Properties that are common to multiple transport protocols. Specifying common Properties enables a system to appropriately select between protocols that offer equivalent features. This design permits evolution of the transport protocols and functions without affecting the programs using the system. Specifying common Properties enables a Transport Services system to appropriately select an appropriate network interface. It is RECOMMENDED that a system offers Properties that are common to a variety network layer interfaces and paths. This design permits racing of different network paths without affecting the programs using the system. Applications using the Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, including the possibility to express requirements and preferences that constrain the choices that can be made by the system. If two different Protocol Stacks can be safely swapped, or raced in parallel (see Section 4.2.2), by the Transport Services system then they are considered to be "equivalent". Equivalent Protocol Stacks need to meet the requirements for racing specified in section XX. Applications using a Transport Services system MAY explicitly require or prevent the use of specific transport features (and transport protocols) and XXXinterfaces using the "REQUIRED" and "PROHIBIT" preferences. This allows an application to constrain the set of protocols and features that will be selected for the transport service. It is RECOMMENDED that the default usage by applications does not necessarily restrict this selection, so that the system can make informed choices (e.g., based on the availability of interfaces or set of transport protocols available for the specified remote endpoint). When a Transport Services system races between two different Protocol Stacks, both SHOULD use the same security protocols and options. However, a Transport Services system MAY race different security protocols, e.g., if the application explicitly specifies that it considers them equivalent. Transport Services systems MUST NOT automatically fall back from secure protocols to insecure protocols, or to weaker versions of secure protocols. A Transport Services system MAY allow applications to specify that fallback to a specific other version of a protocol is allowed. It is RECOMMENDED that in normal use, a Transport Services system is designed so that all selection decisions result in consistent choices for the same network conditions when they have the same set of preferences. This is intended to provide predictable outcomes to the application using the transport service. I would then suggest to use lower case within other parts of the document - except were pointed to from here, (we could refer to the requirements subsection, if we thought necessary).
for discussion at the interim, but IMO we could start the interim with a PR based on these suggestions and work from there.
I suspect that we need more protocol-specific properties to be able to talk to various kinds of native protocol peers (other than TCP and UDP: these should already work fine with what we have, I believe). A concrete example: we're hiding stream numbers. In NEAT, we needed to make them configurable though, because we wanted a NEAT client to be able to talk to a native SCTP server, where an application may expect certain type of data to arrive via a certain stream number. I suspect that we may need something similar for a QUIC peer. Should we address this? (I'd say yes) How .. is it a protocol-specific (per-Message) property? (I'd say yes... and perhaps a per-Connection property too, with text explaining that the per-Message one automatically becomes the Connection default unless otherwise specified) What other such things are we missing... what other expectations may e.g. a native QUIC or SCTP application have that we just cannot fulfil with our current system?

protocols and paths without requiring major changes to the application.  <ins> requirements explains the fundamental requirements for a Transport </ins> Services API.  These principles are intended to make sure that transport protocols can continue to be enhanced and evolve without  <ins> requiring significant changes by application developers. </ins> concepts presents the Transport Services architecture diagram and defines the concepts that are used by both the API and


performance and functional characteristics; and it can communicate with different remote systems to optimize performance, robustness to failure, or some other metric.  Beyond these, if the API for the <del> system remains the same over time, new protocols and features could be added to the system's implementation without requiring changes in </del>  applications for adoption. <del> 3.1. </del>  <del> Functionality that is common across multiple transport protocols ought to be accessible through a unified set of API calls.  An application using a Transport Services API can implement logic for its basic use of transport networking (establishing the transport, and sending and receiving data) once, and expect that implementation to continue to function as the transports change. </del>  <del> As a baseline, any Transport Services API needs to allow access to the distilled minimal set of features offered by transport protocols </del>  I-D.ietf-taps-minset.  3.2. There are applications that will need to control fine-grained details of transport protocols to optimize their behavior and ensure compatibility with remote systems.  A Transport Services system <del> therefore ought to also permit more specialized protocol features to be used.  The interface for these specialized options ought to be exposed differently from the common options to ensure flexibility. </del>  A specialized feature could be required by an application only when using a specific protocol, and not when using others.  For example,

performance and functional characteristics; and it can communicate with different remote systems to optimize performance, robustness to failure, or some other metric.  Beyond these, if the API for the  <ins> system remains the same over time, new protocols and features can be added to the system's implementation without requiring changes in </ins> applications for adoption.  <ins> The normative requirements described here allow Transport Services APIs and Implementations to provide this functionlity without causing incompatibility or introducing security vulnerabilities.  The rest of this document describes the architecture non-normatively. </ins>  <ins> 3.1. </ins>  <ins> Any functionality that is common across multiple transport protocols SHOULD be made accessible through a unified set of Transport Services API calls.  As a baseline, any Transport Services API MUST allow access to the minimal set of features offered by transport protocols </ins> I-D.ietf-taps-minset. <ins> An application can specify constraints and preferences for the protocols, features, and network interfaces it will use via Properties.  A Transport Services API SHOULD offer Properties that are common to multiple transport protocols, in order to enable the system to appropriately select between protocols that offer equivalent features.  Similarly, a Transport Services API SHOULD offer Properties that are applicable to a variety of network layer interfaces and paths, in order to permit racing of different network paths without affecting the applications using the system.  Each Property is expected to have a default value. The default values for Properties SHOULD be selected to ensure correctness for the widest set of applications, while providing the widest set of options for selection.  For example, since both applications that require reliability and those that do not require reliability can function correctly when a protocol provides reliability, reliability ought to be enabled by default.  As another example, the default value for a Property regarding the selection of network interfaces ought to permit as many interfaces as possible. Applications using a Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, where the automated selection is constrained by the requirements and preferences expressed by the application. </ins> 3.2. There are applications that will need to control fine-grained details of transport protocols to optimize their behavior and ensure compatibility with remote systems.  A Transport Services system  <ins> therefore SHOULD permit more specialized protocol features to be used. </ins> A specialized feature could be required by an application only when using a specific protocol, and not when using others.  For example,


preference to use the User Timeout Option, communication would not fail when a protocol such as QUIC is selected. <del> Other specialized features, however, could be strictly required by an </del>  application and thus constrain the set of protocols that can be used. For example, if an application requires support for automatic handover or failover for a connection, only protocol stacks that

preference to use the User Timeout Option, communication would not fail when a protocol such as QUIC is selected.  <ins> Other specialized features, however, can be strictly required by an </ins> application and thus constrain the set of protocols that can be used. For example, if an application requires support for automatic handover or failover for a connection, only protocol stacks that


3.3. <del> The Transport Services API is envisioned as the abstract model for a family of APIs that share a common way to expose transport features and encourage flexibility.  The abstract API definition I-D.ietf- taps-interface describes this interface and how it can be exposed to application developers. Implementations that provide the Transport Services API I-D.ietf- taps-impl will vary due to system-specific support and the needs of the deployment scenario.  It is expected that all implementations of Transport Services will offer the entire mandatory API.  All implementations are expected to offer an API that is sufficient to use the distilled minimal set of features offered by transport protocols I-D.ietf-taps-minset, including API support for TCP and UDP transport.  However, some features provided by this API will not be functional in certain implementations.  For example, it is possible that some very constrained devices might not have a full TCP implementation beneath the API. To preserve flexibility and compatibility with future protocols, top- level features in the Transport Services API ought to avoid referencing particular transport protocols.  The mappings of these API features to specific implementations of each feature is explained in the I-D.ietf-taps-impl along with the implications of the feature on existing protocols.  It is expected that I-D.ietf-taps-interface will be updated and supplemented as new protocols and protocol features are developed. </del>  It is important to note that neither the Transport Services API I- D.ietf-taps-interface nor the Implementation document I-D.ietf-taps- impl define new protocols or protocol capabilities that affect what is communicated across the network.  Use of a Transport Services <del> system does not require that a peer on the other side of a connection </del>  uses the same API or implementation.  A Transport Services system <del> acting as a connection initiator can communicate with any existing system that implements the transport protocol(s) selected by the Transport Services system.  Similarly, a Transport Services system acting as a listener can receive connections for any protocol that is supported by the system from existing initiators that implement the protocol, independent of whether the initiator uses Transport Services as well or not. </del>  4.

3.3.  <ins> A Transport Services implementation can select Protocol Stacks based on the Properties communicated by the application.  If two different Protocol Stacks can be safely swapped, or raced in parallel (see racing), then they are considered to be "equivalent".  Equivalent Protocol Stacks are defined as stacks that can provide the same transport properties and interface expectations as requested by the application. The following two examples show non-equivalent Protocol Stacks: If the application requires preservation of message boundaries, a Protocol Stack that runs UDP as the top-level interface to the application is not equivalent to a Protocol Stack that runs TCP as the top-level interface.  A UDP stack would allow an application to read out message boundaries based on datagrams sent from the remote system, whereas TCP does not preserve message boundaries on its own, but needs a framing protocol on top to determine message boundaries. If the application specifies that it requires reliable transmission of data, then a Protocol Stack using UDP without any reliability layer on top would not be allowed to replace a Protocol Stack using TCP. The following example shows equivalent Protocol Stacks: If the application does not require reliable transmission of data, then a Protocol Stack that adds reliability could be regarded as an equivalent Protocol Stack as long as providing this would not conflict with any other application-requested properties. To ensure that security protocols are not incorrectly swapped, Transport Services systems MUST only select Protocol Stacks that meet application requirements (I-D.ietf-taps-transport-security).  Systems SHOULD only race Protocol Stacks where the transport security protocols within the stacks are identical.  Transport Services systems MUST NOT automatically fall back from secure protocols to insecure protocols, or to weaker versions of secure protocols.  A Transport Services system MAY allow applications to explicitly specify that fallback to a specific other version of a protocol is allowed, e.g., to allow fallback to TLS 1.2 if TLS 1.3 is not available. 3.4. </ins> It is important to note that neither the Transport Services API I- D.ietf-taps-interface nor the Implementation document I-D.ietf-taps- impl define new protocols or protocol capabilities that affect what is communicated across the network.  Use of a Transport Services  <ins> system MUST NOT require that a peer on the other side of a connection </ins> uses the same API or implementation.  A Transport Services system  <ins> acting as a connection initiator is able to communicate with any existing system that implements the transport protocol(s) and all the required properties selected by the Transport Services system. Similarly, a Transport Services system acting as a listener can receive connections for any protocol that is supported by the system from existing initiators that implement the protocol, independent of whether the initiator uses a Transport Services system or not. In normal use, a Transport Services system SHOULD result in consistent protocol and interface selection decisions for the same network conditions given the same set of Properties.  This is intended to provide predictable outcomes to the application using the API. </ins> 4.


4.2.3. <del> The Transport Services architecture defines a mechanism that allows applications to easily make use of various network paths and Protocol Stacks without requiring major changes in application logic.  In some cases, changing which Protocol Stacks or network paths are used will require updating the preferences expressed by the application that uses the Transport Services system.  For example, an application can enable the use of a multipath or multistreaming transport protocol by modifying the properties in its Pre-Connection configuration.  In some cases, however, the Transport Services system will be able to automatically change Protocol Stacks without an update to the application, either by selecting a new stack entirely, or by racing multiple candidate Protocol Stacks during connection establishment. This functionality in the API can be a powerful driver of new protocol adoption, but needs to be constrained carefully to avoid unexpected behavior that can lead to functional or security problems. If two different Protocol Stacks can be safely swapped, or raced in parallel (see racing), then they are considered to be "equivalent". Equivalent Protocol Stacks need to meet the following criteria: Both stacks MUST offer the interface requested by the application for connection establishment and data transmission.  For example, if an application requires preservation of message boundaries, a Protocol Stack that runs UDP as the top-level interface to the application is not equivalent to a Protocol Stack that runs TCP as the top-level interface.  A UDP stack would allow an application to read out message boundaries based on datagrams sent from the remote system, whereas TCP does not preserve message boundaries on its own, but needs a framing protocol on top to determine message boundaries. Both stacks MUST offer the transport services that are requested by the application.  For example, if an application specifies that it requires reliable transmission of data, then a Protocol Stack using UDP without any reliability layer on top would not be allowed to replace a Protocol Stack using TCP.  However, if the application does not require reliability, then a Protocol Stack that adds reliability could be regarded as an equivalent Protocol Stack as long as providing this would not conflict with any other application-requested properties. Both stacks MUST offer security protocols and parameters as requested by the application I-D.ietf-taps-transport-security. Security features and properties, such as cryptographic algorithms, peer authentication, and identity privacy vary across security protocols, and across versions of security protocols. Protocol equivalence ought not to be assumed for different protocols or protocol versions, even if they offer similar application configuration options.  To ensure that security protocols are not incorrectly swapped, Transport Services systems SHOULD only automatically generate equivalent Protocol Stacks when the transport security protocols within the stacks are identical. Specifically, a Transport Services system would consider protocols identical only if they are of the same type and version.  For example, the same version of TLS running over two different transport Protocol Stacks are considered equivalent, whereas TLS 1.2 and TLS 1.3 RFC8446 are not considered equivalent.  However, Transport Services systems MAY allow applications to indicate that they consider two different transport protocols equivalent, e.g., to allow fallback to TLS 1.2 if TLS 1.3 is not available. 4.2.4. </del> By default, stored properties of the implementation, such as cached protocol state, cached path state, and heuristics, may be shared (e.g. across multiple connections in an application).  This provides

4.2.3.  By default, stored properties of the implementation, such as cached protocol state, cached path state, and heuristics, may be shared (e.g. across multiple connections in an application).  This provides


D.ietf-taps-transport-security. As described above in equivalence, if a Transport Services system <del> races between two different Protocol Stacks, both SHOULD use the same security protocols and options.  However, a Transport Services system MAY race different security protocols, e.g., if the application explicitly specifies that it considers them equivalent. </del>  Applications need to ensure that they use security APIs appropriately.  In cases where applications use an interface to

D.ietf-taps-transport-security. As described above in equivalence, if a Transport Services system  <ins> races between two different Protocol Stacks, both need to use the same security protocols and options.  However, a Transport Services system can race different security protocols, e.g., if the application explicitly specifies that it considers them equivalent. </ins> Applications need to ensure that they use security APIs appropriately.  In cases where applications use an interface to


QUIC, and applications ought not to use private keys intended for server authentication as keys for client authentication. <del> Moreover, Transport Services systems MUST NOT automatically fall back </del>  from secure protocols to insecure protocols, or to weaker versions of <del> secure protocols.  For example, if an application requests a specific version of TLS, but the desired version of TLS is not available, its connection will fail.  Applications are thus responsible for implementing security protocol fallback or version fallback by creating multiple Transport Services Connections, if so desired. Alternatively, a Transport Services system MAY allow applications to specify that fallback to a specific other version of a protocol is allowed. </del>

QUIC, and applications ought not to use private keys intended for server authentication as keys for client authentication.  <ins> Moreover, Transport Services systems must not automatically fall back </ins> from secure protocols to insecure protocols, or to weaker versions of  <ins> secure protocols (see equivalence).  For example, if an application requests a specific version of TLS, but the desired version of TLS is not available, its connection will fail.  Applications are thus responsible for implementing security protocol fallback or version fallback by creating multiple Transport Services Connections, if so desired.  Alternatively, a Transport Services system MAY allow applications to specify that fallback to a specific other version of a protocol is allowed. </ins>


Transport Properties MUST always be specified while security parameters are OPTIONAL. <del> Message Framers (see framing), if required, should be added to the </del>  Preconnection during pre-establishment. 5.1.

In the API doc sec 5 says. "Message Framers (see Section 10), if required, should be added to the Preconnection during pre-establishment." Should this be SHOULD or even MUST? If SHOULD is correct I think we need to say slightly more about when it makes sense to add a framer later.
I think it should be "can be added..."; not normative, just descriptive of fact.
Can we start with the "If' perhaps... "If messages frames are to be used, they MUST be added to the Preconnection during pre-establishment."?

Transport Properties MUST always be specified while security parameters are OPTIONAL.  <ins> If Message Framers are used (see framing), they MUST be added to the </ins> Preconnection during pre-establishment. 5.1.


4.2.1. <del> Immediate, or simultaneous, racing should be avoided by implementations.  This approach can consume network resources and establish state that will not be used. </del>  4.2.2.

I actually really like: "Simultaneous" and "Staggered" as descriptive terms that avoid the question of "delaying".
Opening this to follow up on a comment by NAME on that is now closed. "Simultaneous" and "Staggered" would be better names than "Immediate" and "Delayed". Just from these terms, I would normally think that "Immediate" means to start racing right away, whereas "Delayed" means that nothing would happen for some time. So these two terms are slightly misleading in my opinion.)
I agree this would be an improvement.
In Section 4.2.1., I didn't really understand what "immediate" means from the text. I think it just needs one sentence.
lgtm (side note - nothing blocking, this is just an idea: to me, "Simultaneous" and "Staggered" would be better names than "Immediate" and "Delayed". Just from these terms, I would normally think that "Immediate" means to start racing right away, whereas "Delayed" means that nothing would happen for some time. So these two terms are slightly misleading in my opinion.)Looks good, thanks! I will close this and open a separate issue for the naming.

4.2.1.  <ins> Immediate racing is when multiple alternate branches are started without waiting for any one branch to make progress before starting the next alternative.  This means the attempts are effectively simultaneous.  Immediate racing should be avoided by implementations, since it consumes extra network resources and establishes state that might not be used. </ins> 4.2.2.


The Rendezvous() Action causes the Preconnection to listen on the Local Endpoint for an incoming Connection from the Remote Endpoint, <del> while simultaneously trying to establish a Connection from the Local Endpoint to the Remote Endpoint.  This corresponds to a TCP simultaneous open, for example. </del>  The Rendezvous() Action returns a Connection object.  Once Rendezvous() has been called, any changes to the Preconnection MUST

NAME 7df2d9c should address this
To we need multiple local and remote candidates. Should this be represented as either: 1) taking two lists of objects; or 2) as two objects each with multiple addresses, ports, etc. I suggest option 1, since this makes it easier to resolve a to multiple preconnections, for NAT binding discovery. Relates to but with also different types of endpoint.
I like option 1 because it contains complexity.
See this email thread: URL Copy + pasting a relevant part from an email from NAME To me at least, it wasn't clear from the text that rendezvous() would kick off ICE-style probing. It's kind of obvious (when else would it happen?), but I think it would be good for the draft to spell this out.
modulo the do-we-want-to-bind-tightly-to-ICE question... Dear all, I know way too little about rendezvous mechanisms, so I may get this completely wrong - please bear with me! It may be that I describe an issue that’s a non-issue. But, if I do understand this right, maybe that’s something which needs fixing? Ok, here it goes… Our current Rendezvous text in the API draft says the following: "The Rendezvous() Action causes the Preconnection to listen on the Local Endpoint for an incoming Connection from the Remote Endpoint, while simultaneously trying to establish a Connection from the Local Endpoint to the Remote Endpoint. This corresponds to a TCP simultaneous open, for example.” Now, if an application chooses unreliable data transmission, it may get SCTP with partial reliability, or it may (in the future) get QUIC with datagrams, or whatever… and then, the protocol may be able to do what’s needed (something similar to a simultaneous open). However, in case of UDP: Listening is quite okay, “simultaneously trying to establish a Connection” really means nothing - no packet will be sent. I believe that applications doing this over UDP will implement their own “SYN”-type signal for this - but then it’s up to the application, which, over TAPS, would need to: create a passive connection, send a Message from that same connection, i.e. using the same local address and port (I believe?) - which is a matter of configuring the local endpoint, and doable if the underlying transport system can pull it off. I think there’s nothing in our API preventing an application from doing this. If all of the above is correct, the problem that I see is that an application may ask for unreliable transmission, and then have one case in which calling “Rendezvous” is the right thing to do, and another case in which it is not - so it would need to be able to distinguish between these two cases: is it a protocol with Rendezvous-capability, or do I have to do it myself. Maybe, in case of UDP, the call to Rendezvous should just fail, and our text should say that Rendezvous failing would inform the application that it would have to implement it by itself, by issuing Listen and then sending a Message to the peer. Thoughts? Am I completely off track, or is this something that needs a text fix? Cheers, Michael

The Rendezvous() Action causes the Preconnection to listen on the Local Endpoint for an incoming Connection from the Remote Endpoint,  <ins> while also simultaneously trying to establish a Connection from the Local Endpoint to the Remote Endpoint. If there are multiple Local Endpoints or Remote Endpoints configured, then initiating a rendezvous action will systematically probe the reachability of those endpoints following an approach such as that used in Interactive Connectivity Establishment (ICE) RFC5245. If the endpoints are suspected to be behind a NAT, Rendezvous() can be initiated using Local and Remote Endpoints that support a method of discovering NAT bindings such as Session Traversal Utilities for NAT (STUN) RFC8489 or Traversal Using Relays around NAT (TURN) RFC5766.  In this case, the Local Endpoint will resolve to a mixture of local and server reflexive addresses.  The Resolve() action on the Preconnection can be used to discover these bindings: The Resolve() call returns a list of Preconnection Objects, that represent the concrete addresses, local and server reflexive, on which a Rendezvous() for the Preconnection will listen for incoming Connections.  These resolved Preconnections will share all other Properties with the Preconnection from which they are derived, though some Properties may be made more-specific by the resolution process. An application that uses Rendezvous() to establish a peer-to-peer connection in the presence of NATs will configure the Preconnection object with a Local Endpoint that supports NAT binding discovery.  It will then Resolve() on that endpoint, and pass the resulting list of candidate local addresses to the peer via a signalling protocol, for example as part of an ICE RFC5245 exchange within SIP RFC3261 or WebRTC RFC7478.  The peer will, via the same signalling channel, return the remote endpoint candidates.  These remote endpoint candidates are then configured on the Preconnection, allowing the Rendezvous() Action to be initiated. </ins> The Rendezvous() Action returns a Connection object.  Once Rendezvous() has been called, any changes to the Preconnection MUST


other, and become part of a Connection Group.  Calling Clone on any of these Connections adds another Connection to the Connection Group, and so on.  "Entangled" Connections share all Connection Properties <del> except "Priority (Connection)", see conn-priority.  Like all other Properties, Priority is copied to the new Connection when calling Clone(), but it is not entangled: Changing Priority on one Connection does not change it on the other Connections in the same Connection Group. </del>  It is also possible to check which Connections belong to the same Connection Group.  Calling GroupedConnections() on a specific

(well, the priority framework is still confusing, this PR makes it clear that that is intentional, and why)
Martin (I think) said that he'd check this offline after the Sep 11 interim.
At 8.1.2 you say lower values have higher priorities for connPrio. But later, at msgPrio, higher values have higher priority, and in the examples where you speak of the interaction of connPrio and msgPrio, you imply that higher means higher for both properties. Please check for consistency and clarify.
Lower values having higher priorities for connPrio is a remnant that we must have missed when we flipped this, e.g. PR which I don't know... too many iterations have mangled the text.
Aha, now I see what happened: following some more PRs, also , in PR , NAME re-introduced some text saying that sends on Connections with lower-priority values will be prioritized over sends on Connections with a higher-priority value. This reverses the logic for some part of the text - I suspect that NAME (very understandably!) forgot what happened and this created this mix. We have been editing these documents for way too long!
I'm not sure the priority framework really fits together. Sec 6.4 says we 'should' do something that sounds like WFQ, but 7.1.5 allows the application to decide what happens. Meanwhile message priority suggests a strict priority-based scheduler. Meanwhile, in sec 8.1.3.2 "Note that this property is not a per-message override of the connection Priority - see Section 7.1.3. Both Priority properties may interact, but can be used independently and be realized by different mechanisms." At the risk of overindexing on a partly-baked draft in httpbis, how would I implement strict priority of HTTP/3 streams over a taps API that implements QUIC? Would I need to request strict priority in both connection priority and message priority? Or I guess connection priority would be sufficient?
We seem comfortable with the mechanisms in the API, but we should have more text connecting the various sections explaining priority.
My comment concerned "7.1.3. Priority (Connection)", my thought was this just influences the sender-side to control queuing within a connection group. If so, if it worth calling this property "Group Priority" and explain this is sender-side?
I find the priority framework perhaps overspecified at the connection level and underspecified at the message level, with the interaction between the two left explicitly unstated. I would almost rather see the priority type and meaning left completely implementation-specific; otherwise, my suspicion is that someone will write an application using numeric priorities, declare victory when it works fine on one platform, and not understand that the behavior is not universal. In general, I think if TAPS is going to produce a consistent, easily-understandable, and portable API, it needs to stick to easily-understood abstractions. Message-level priority is sort of like the TCP URG mechanism (though better specified and more useful, for sure), but I'm thinking urgent messages sent explicitly out-of-band (e.g., new Connection) with implementation-specific QoS will produce more consistent, predictable, and understandable outcomes in the pessimal case.
Ok for me (and thanks for doing this!)Some nits, but I believe this addresses my concerns in the issue.

other, and become part of a Connection Group.  Calling Clone on any of these Connections adds another Connection to the Connection Group, and so on.  "Entangled" Connections share all Connection Properties  <ins> except "Connection Priority", see conn-priority.  Like all other Properties, Connection Priority is copied to the new Connection when calling Clone(), but it is not entangled: Changing Connection Priority on one Connection does not change it on the other Connections in the same Connection Group. </ins> It is also possible to check which Connections belong to the same Connection Group.  Calling GroupedConnections() on a specific


Attempts to clone a Connection can result in a CloneError: <del> The Connection Property "Priority" operates on entangled Connections as in msg-priority: when allocating available network capacity among Connections in a Connection Group, sends on Connections with higher Priority values will be prioritized over sends on Connections with lower Priority values.  A transport system implementation should, if possible, assign each Connection the capacity share (M-N) x C / M, where N is the Connection's Priority value, M is the maximum Priority value used by all Connections in the group and C is the total available capacity.  However, this Priority setting is purely advisory, and provides no guarantees about the way capacity is shared.  Each implementation is free to implement a way to share capacity that it sees fit. </del>  7.

Attempts to clone a Connection can result in a CloneError:  <ins> The Connection Priority Connection Property operates on entangled Connections using the same approach as in msg-priority: when allocating available network capacity among Connections in a Connection Group, sends on Connections with lower Priority values will be prioritized over sends on Connections with higher Priority values.  Capacity will be shared among these Connections according to the Connection Group Transmission Scheduler property (conn- scheduler).  See priority-in-taps for more. </ins> 7.


Group.  As noted in groups, this property is not entangled when Connections are cloned, i.e., changing the Priority on one Connection in a Connection Group does not change it on the other Connections in <del> the same Connection Group. </del>  7.1.4.

Group.  As noted in groups, this property is not entangled when Connections are cloned, i.e., changing the Priority on one Connection in a Connection Group does not change it on the other Connections in  <ins> the same Connection Group.  See priority-in-taps. </ins> 7.1.4.


prioritization. Note that this property is not a per-message override of the <del> connection Priority - see conn-priority.  Both Priority properties may interact, but can be used independently and be realized by different mechanisms. </del>  8.1.3.3.

prioritization. Note that this property is not a per-message override of the  <ins> connection Priority - see conn-priority.  The Priority properties may interact, but can be used independently and be realized by different mechanisms; see priority-in-taps. </ins> 8.1.3.3.


will not result in a SendError separate from the InitiateError signaling the failure of Connection establishment.  8.3. Once a Connection is established, it can be used for receiving data

will not result in a SendError separate from the InitiateError signaling the failure of Connection establishment. <ins> 8.2.7. The Transport Services interface provides two properties to allow a sender to signal the relative priority of data transmission: the Priority Message Property msg-priority, and the Connection Priority Connection Property conn-priority.  These properties are designed to allow the expression and implementation of a wide variety of approaches to transmission priority in the transport and application layer, including those which do not appear on the wire (affecting only sender-side transmission scheduling) as well as those that do (e.g.  I-D.ietf-httpbis-priority. A Transport Services system gives no guarantees about how its expression of relative priorities will be realized; for example, if a transport stack that only provides a single in-order reliable stream is selected, prioritization information can only be ignored. However, the Transport Services system will seek to ensure that performance of relatively-prioritized connections and messages is not worse with respect to those connections and messages than an equivalent configuration in which all prioritization properties are left at their defaults. The Transport Services interface does order Connection Priority over the Priority Message Property.  In the absense of other externalities (e.g., transport-layer flow control), a priority 1 Message on a priority 0 Connection will be sent before a priority 0 Message on a priority 1 Connection in the same group. </ins> 8.3. Once a Connection is established, it can be used for receiving data


to an inbound SYN with a SYN-ACK. Calling "Clone" on a TCP Connection creates a new Connection with <del> equivalent parameters.  The two Connections are otherwise independent. </del>  SEND.TCP.  TCP does not on its own preserve Message boundaries. Calling "Send" on a TCP connection lays out the bytes on the TCP

This provides more detail on how to implement Clone with TCP, and clarifies that SCTP's "Initiate" should not yield a stream of an already existing SCTP Association in case "activeReadBeforeSend" is Preferred or Required.
Currently the "Clone" part of, e.g., TCP (section 10.1) only says: "Calling Clone on a TCP Connection creates a new Connection with equivalent parameters. The two Connections are otherwise independent." This text should reflect that equivalent behavior must be maintained upon later Connection configuration calls. And, there should be more text in general on cloning (see issue ).
This text must refer to a new "do I want a Connection that supports initiate followed by listen?" property. This property will be added to address - so I'll address this issue when the PR addressing has landed.
Added a question for NAME

to an inbound SYN with a SYN-ACK. Calling "Clone" on a TCP Connection creates a new Connection with  <ins> equivalent parameters.  These Connections, and Connections generated via later calls to "Clone" on one of them, form a Connection Group.  To realize "entanglement" for these Connections, with the exception of "Connection Priority", changing a Connection Property on one of them must affect the Connection Properties of the others too.  No guarantees of honoring the Connection Property "Connection Priority" are given, and thus it is safe for an implementation of a transport system to ignore this property.  When it is reasonable to assume that Connections traverse the same path (e.g., when they share the same encapsulation), support for it can also experimentally be implemented using a congestion control coupling mechanism (see for example TCP-COUPLING or RFC3124). </ins> SEND.TCP.  TCP does not on its own preserve Message boundaries. Calling "Send" on a TCP connection lays out the bytes on the TCP


If this is the only Connection object that is assigned to the SCTP association or stream mapping has not been negotiated, <del> CONNECT.SCTP is called.  Else, a new stream is used: if there are enough streams available, "Initiate" is just a local operation that assigns a new stream number to the Connection object.  The number of streams is negotiated as a parameter of the prior CONNECT.SCTP call, and it represents a trade-off between local resource usage and the number of Connection objects that can be mapped without requiring a reconfiguration signal.  When running out of streams, ADD_STREAM.SCTP must be called. </del>  If this is the only Connection object that is assigned to the SCTP association or stream mapping has not been negotiated,

If this is the only Connection object that is assigned to the SCTP association or stream mapping has not been negotiated,  <ins> CONNECT.SCTP is called.  Else, unless the Selection Property "activeReadBeforeSend" is Preferred or Required, a new stream is used: if there are enough streams available, "Initiate" is just a local operation that assigns a new stream number to the Connection object.  The number of streams is negotiated as a parameter of the prior CONNECT.SCTP call, and it represents a trade-off between local resource usage and the number of Connection objects that can be mapped without requiring a reconfiguration signal.  When running out of streams, ADD_STREAM.SCTP must be called. </ins> If this is the only Connection object that is assigned to the SCTP association or stream mapping has not been negotiated,


be sent in multiple parts. Protocols that provide the framing (such as length-value protocols, <del> or protocols that use delimiters) provide data boundaries that may be longer than the typical datagram.  Each Message for framing protocols </del>  corresponds to a single frame, which may be sent either as a complete <del> Message, or in multiple parts. </del>  5.1.

Now I feel like I'm shaving the yak. This is more than what we talked about in the interim, but it felt like the following sentence required additional clarity. Let me know if I misunderstood the intended meaning, in which case I will rephrase.

be sent in multiple parts. Protocols that provide the framing (such as length-value protocols,  <ins> or protocols that use delimiters) may support Message sizes that do not fit within a single datagram.  Each Message for framing protocols </ins> corresponds to a single frame, which may be sent either as a complete  <ins> Message in the underlying protocol, or in multiple parts. </ins> 5.1.


5.4. <del> Entangled Connections can be created using the Clone Action: Calling Clone on a Connection yields a group of Connections: the parent Connection on which Clone was called, and a resulting cloned Connection.  The new Connection is actively openend, and it will send a Ready Event or an EstablishmentError Event.  The Connections within a group are "entangled" with each other, and become part of a Connection Group.  Calling Clone on any of these Connections adds another Connection to the Connection Group, and so on.  "Entangled" Connections share all Connection Properties except "Connection Priority" (see conn-priority) .  Like all other Properties, Connection Priority is copied to the new Connection when calling Clone(), but it is not entangled: Changing Connection Priority on one Connection does not change it on the other Connections in the same Connection Group. The stack of Message Framers associated with a Connection are also copied to the cloned Connection when calling Clone.  In other words, a cloned Connection has the same stack of Message Framers as the Connection from which they are Cloned, but these Framers may internally maintain per-Connection state. </del>  It is also possible to check which Connections belong to the same Connection Group.  Calling GroupedConnections() on a specific

This changes the use of the term "entanglement" as discussed in , and moves some text around in this section for better clarity and less repetition. Also, addressing , it introduces a framer as an optional parameter to the Clone call.
NAME I incorporated all of your suggestions, thanks!
This is not fully clear from section 5.4., however, if connections in a group are each QUIC streams, I think it should be possible to use different framers on each stream. So it might be okay that a framers is copied when clone is called but changing a framer on one connection should not change the framer on the other connection in a group...?
Also an related editorial comment (that could be addressed in the same PR eventually): this section on groups only at the very end has a paragraph starting with "Note that calling Clone() can result in on-the-wire signaling..." which kind of explains that groups are supposed to be used for multistream protocol but can also be mapped to separate connection if the protocol doesn't not support multiplexing of stream on one connection. I think this information should be added at the very beginning of this section.
Actually, now looking at the framer section, I guess you would rather like to already set the new framer before you clone a connection...?
I agree with everything you say here; about your last suggestion, maybe a framer should be an optional argument of the clone call?
I don’t actually understand the difference between a connection in a Connection Group and an “entangled connection” ? Both terms are used in API, but in different ways (possibly by different authors) - do we really really need the word “entangled”?? if we do we need, can we define and explain this.
I think that having the word is good as it indicates a deeper relationship than mere "grouping". Just based on terms, I'd (correctly) imagine that I can define priorities between members of a group, and that they somehow belong together... as in: Close with a common call, etc. ... but I don't think that I would expect, just from this term, that changing a property on one of them affects all the others. Hence the phrasing: "The connections within a group are 'entangled' with each other". However, I agree that in most other places, we should talk about "groups" instead. I can try to fix this.
Sounds good - we should add that explanation upfront in the definitions then also?
[Yes, I think so.] EDIT: no, now I don't think so anymore. (see )
I have addressed this in PR . Note that the text interchangeably talked about both Connections and Connection Properties as being "entangled". I have now written this such that the term "entanglement" applies to Connection Properties only, whereas Connections in a group are, well, members of the same Connection Group.
Looks good to me.

5.4.  <ins> Connection Groups can be created using the Clone Action: Calling Clone on a Connection yields a Connection Group containing two Connections: the parent Connection on which Clone was called, and a resulting cloned Connection.  The new Connection is actively openend, and it will send a Ready Event or an EstablishmentError Event.  Calling Clone on any of these Connections adds another Connection to the Connection Group.  Connections in a Connection Group share all Connection Properties except "Connection Priority" (see conn-priority), and these Connection Properties are entangled: Changing one of the Connection Properties on one Connection in the Connection Group automatically changes the Connection Property for all others.  For example, changing "Timeout for aborting Connection" (see conn-timeout) on one Connection in a Connection Group will automatically make the same change to this Connection Property for all other Connections in the Connection Group.  Like all other Properties, "Connection Priority" is copied to the new Connection when calling Clone(), but in this case, a later change to the "Connection Priority" on one Connection does not change it on the other Connections in the same Connection Group. Message Properties are also not entangled.  For example, changing "Lifetime" (see msg-lifetime) of a Message will only affect a single Message on a single Connection. A new Connection created by Clone can have a Message Framer assigned via the optional "framer" parameter of the Clone Action.  If this parameter is not supplied, the stack of Message Framers associated with a Connection is copied to the cloned Connection when calling Clone.  Then, a cloned Connection has the same stack of Message Framers as the Connection from which they are Cloned, but these Framers may internally maintain per-Connection state. </ins> It is also possible to check which Connections belong to the same Connection Group.  Calling GroupedConnections() on a specific


is just a new stream of an already active multi-streaming protocol instance. <del> Changing one of the Connection Properties on one Connection in the group changes it for all others.  Message Properties, however, are not entangled.  For example, changing "Timeout for aborting Connection" (see conn-timeout) on one Connection in a group will automatically change this Connection Property for all Connections in the group in the same way.  However, changing "Lifetime" (see msg- lifetime) of a Message will only affect a single Message on a single Connection, entangled or not. </del> If the underlying protocol supports multi-streaming, it is natural to <del> use this functionality to implement Clone.  In that case, entangled Connections are multiplexed together, giving them similar treatment not only inside endpoints, but also across the end-to-end Internet path. </del>  Note that calling Clone() can result in on-the-wire signaling, e.g., <del> to open a new connection, depending on the underlying Protocol Stack. When Clone() leads to multiple connections being opened instead of multi-streaming, the Transport Services system will ensure consistency of Connection Properties by uniformly applying them to all underlying connections in a group.  Even in such a case, there are possibilities for a Transport Services system to implement prioritization within a Connection Group TCP-COUPLING RFC8699. </del>  Attempts to clone a Connection can result in a CloneError: <del> The Connection Priority Connection Property operates on entangled Connections using the same approach as in msg-priority: when allocating available network capacity among Connections in a </del>  Connection Group, sends on Connections with lower Priority values will be prioritized over sends on Connections with higher Priority values.  Capacity will be shared among these Connections according to

is just a new stream of an already active multi-streaming protocol instance.  If the underlying protocol supports multi-streaming, it is natural to  <ins> use this functionality to implement Clone.  In that case, Connections in a Connection Group are multiplexed together, giving them similar treatment not only inside endpoints, but also across the end-to-end Internet path. </ins> Note that calling Clone() can result in on-the-wire signaling, e.g.,  <ins> to open a new transport connection, depending on the underlying Protocol Stack.  When Clone() leads to the opening of multiple such connections, the Transport Services system will ensure consistency of Connection Properties by uniformly applying them to all underlying connections in a group.  Even in such a case, there are possibilities for a Transport Services system to implement prioritization within a Connection Group TCP-COUPLING RFC8699. </ins> Attempts to clone a Connection can result in a CloneError:  <ins> The "Connection Priority" Connection Property operates on Connections in a Connection Group using the same approach as in msg-priority: when allocating available network capacity among Connections in a </ins> Connection Group, sends on Connections with lower Priority values will be prioritized over sends on Connections with higher Priority values.  Capacity will be shared among these Connections according to


When set to true, this property will initiate new Connections using as little cached information (such as session tickets or cookies) as <del> possible from previous connections that are not entangled with it. Any state generated by this Connection will only be shared with entangled connections.  Cloned Connections will use saved state from within the Connection Group.  This is used for separating Connection Contexts as specified in I-D.ietf-taps-arch. </del>  Note that this does not guarantee no leakage of information, as implementations may not be able to fully isolate all caches (e.g.

When set to true, this property will initiate new Connections using as little cached information (such as session tickets or cookies) as  <ins> possible from previous connections that are not in the same Connection Group.  Any state generated by this Connection will only be shared with Connections in the same Connection Group.  Cloned Connections will use saved state from within the Connection Group. This is used for separating Connection Contexts as specified in I- D.ietf-taps-arch. </ins> Note that this does not guarantee no leakage of information, as implementations may not be able to fully isolate all caches (e.g.


prioritization. Note that this property is not a per-message override of the <del> connection Priority - see conn-priority.  The Priority properties may </del>  interact, but can be used independently and be realized by different mechanisms; see priority-in-taps.

prioritization. Note that this property is not a per-message override of the  <ins> Connection Priority - see conn-priority.  The Priority properties may </ins> interact, but can be used independently and be realized by different mechanisms; see priority-in-taps.


example, if reliable delivery was requested for a Message handed over before calling Close, the Closed Event will signify that this Message has indeed been delivered.  This action does not affect any other <del> Connection that is entangled with this one in a Connection Group. </del>  The Closed Event informs the application that the Remote Endpoint has closed the Connection.  There is no guarantee that a remote Close will indeed be signaled. Abort terminates a Connection without delivering any remaining <del> Messages.  This action does not affect any other Connection that is entangled with this one in a Connection Group. </del>  CloseGroup gracefully terminates a Connection and any other <del> Connections that are entangled with this one in a Connection Group. For example, all of the Connections in a group might be streams of a single session for a multistreaming protocol; closing the entire group will close the underlying session.  See also groups.  As with Close, any Messages remaining to be processed on a Connection will be handled prior to closing. AbortGroup terminates a Connection and any other Connections that are entangled with this one in a Connection Group without delivering any remaining Messages. </del>  A ConnectionError informs the application that: 1) data could not be delivered to the peer after a timeout, or 2) the Connection has been

example, if reliable delivery was requested for a Message handed over before calling Close, the Closed Event will signify that this Message has indeed been delivered.  This action does not affect any other  <ins> Connection in the same Connection Group. </ins> The Closed Event informs the application that the Remote Endpoint has closed the Connection.  There is no guarantee that a remote Close will indeed be signaled. Abort terminates a Connection without delivering any remaining  <ins> Messages.  This action does not affect any other Connection that in the same Connection Group. </ins> CloseGroup gracefully terminates a Connection and any other  <ins> Connections in the same Connection Group.  For example, all of the Connections in a group might be streams of a single session for a multistreaming protocol; closing the entire group will close the underlying session.  See also groups.  As with Close, any Messages remaining to be processed on a Connection will be handled prior to closing. AbortGroup terminates a Connection and any other Connections in the same Connection Group without delivering any remaining Messages. </ins> A ConnectionError informs the application that: 1) data could not be delivered to the peer after a timeout, or 2) the Connection has been


during Connection establishment.  This Message can potentially be received multiple times (i.e., multiple copies of the message data may be passed to the Remote Endpoint).  See also msg- <del> safelyreplayable.  Note that disabling this property has no effect for protocols that are not connection-oriented and do not protect against duplicated messages, e.g., UDP. </del>  4.2.6.

The text says What does disabling mean? Does this mean mean that it is valid to set this property to "Prohibit" but still set "safelyReplayable" to "Require" on the PreConnection. Should that not be an error? This is also related to . The use of safelyReplayable for both reliable and unreliable services makes the text a bit confusing. Would it not be sufficient to state that if you do not Require Reliable Data Transfer you have to be prepared to deal with duplication?
We should remove this sentence in 4.2.5:

during Connection establishment.  This Message can potentially be received multiple times (i.e., multiple copies of the message data may be passed to the Remote Endpoint).  See also msg-  <ins> safelyreplayable. </ins> 4.2.6.


a Preconnection allows applications to override this for more detailed control. <del> Transport Properties MUST always be specified while security parameters are OPTIONAL. </del> If Message Framers are used (see framing), they MUST be added to the Preconnection during pre-establishment.

For other parameters you have to pass in an empty object if you do not want to set anything but not for SecurityParameters. Unclear why the parameters are treated differently?
Make security parameters non-optional Remove "Transport Properties MUST always be specified while security parameters are OPTIONAL." Define that there is an implementation-specific security parameter to disable security
LGTM, thanks!

a Preconnection allows applications to override this for more detailed control.  If Message Framers are used (see framing), they MUST be added to the Preconnection during pre-establishment.


Session cache management: Used to tune session cache capacity, lifetime, and other policies.  Representation of Security Parameters in implementations should parallel that chosen for Transport Property names as recommended in scope-of-interface-defn.

Session cache management: Used to tune session cache capacity, lifetime, and other policies. <ins> Connections that use Transport Services SHOULD use security in general.  However, for compatibility with endpoints that do not support transport security protocols (such as a TCP endpoint that does not support TLS), applications can initialize their security parameters to indicate that security can be disabled, or can be opportunistic.  If security is disabled, the Transport Services system will not attempt to add transport security automatically.  If security is opportunistic, it will allow Connections without transport security, but will still attempt to use security if available. </ins> Representation of Security Parameters in implementations should parallel that chosen for Transport Property names as recommended in scope-of-interface-defn.


transport layer, in line with developments in modern platforms and programming languages; <del> Selection between alternate network paths that can be informed by additional information that is available about the networks over </del>  which an endpoint can operate (e.g.  Provisioning Domain (PvD) information RFC7556);

This tries to write something small but specific about Network Policies
LGTM
In Sec 2: Should Section 3.2 also talk about network policies, e.g. a certain network supports a DSCP that the application desires?
As discussed in Feb 2021 interim, this seems like a rabbithole. Gorry will figure out whether it is and write language if not, or close if so.
I'd like to add a couple of lines of text here to bullet 2 that discusses how you might learn that an attached network supports certain features - e.g. through a PvD, routing advertisement, or local discovery method, and suggest that this could enable a system to know which paths have specific characteristics.
I propose this as a starter: When additional information (such as Provisioning Domain (PvD) information {{RFC7556}}) is provided about the networks over which an endpoint can operate, this can inform the selection between alternate network paths. Such information includes network segment PMTU, set of supported DSCPs, expected usage, cost, etc. The usage of this information by the Transport Services API is generally independent of the specific mechanism/protocol that was used to receive the information (e.g. zero-conf, DHCP, or IPv6 RA). Thoughts... ?
Tried to create a PR to add some policy to the path selection
I think the policy is separate from the API.LGTM

transport layer, in line with developments in modern platforms and programming languages;  <ins> Selection between alternate network paths that can be informed when additional information is available about the networks over </ins> which an endpoint can operate (e.g.  Provisioning Domain (PvD) information RFC7556);


A Preconnection Object holds properties reflecting the application's requirements and preferences for the transport.  These include Selection Properties for selecting protocol stacks and paths, as well <del> as Connection Properties for configuration of the detailed operation of the selected Protocol Stacks. </del>  The protocol(s) and path(s) selected as candidates during establishment are determined and configured using these properties.

Applied requested changes
In 3.2 it is clear that you can set Message Properties on Connections and Preconnections, but the rest of the document is not very clear about this. In 4.2 for instance the text only talks about setting Selection Properties and Connection Properties on PreConnections. In 7.1.3 the text says "Connection Properties describe the default behavior for all Messages on a Connection". But I think that Message Properties set on a Connection or PreConnection may also describe the default behavior so this relationship should be clarified. Also, the Message Property defaults given for Properties that do not depend on other Properties sets the defaults for all Messages in a Connection I assume? I also assume that a Message Property set on a Message Context overrides a Message Property set on a PreConnection or Connection, but this is also not specified. In 5.4. Connection Groups, I assume that setting Message Properties on a Connection applies to all Connections in the Connection Group? This is not clear.
In general, I agree. If you set a message property dynamically on a connection, though, I don't think it would affect the group—I could have one stream in a multiplexed protocol have separate behavior, such as priority.
+1 to tfpauly. But this is already written in 5.4, maybe you missed it (Anna)? "Message Properties are also not entangled. For example, changing Lifetime (see Section 7.1.3.1) of a Message will only affect a single Message on a single Connection."
I am fine with an update of a Message Property to a connection applying only to that connection. Priority is not a good example as it is already a special case, but my main issue was that we need to be more clear about the behavior in the specification. NAME the issue is about setting a Message Property on a Connection. The text you refer to talk about setting Message Properties on a message. You could interpret the first part as a general statement, but the way the text is structured it is easy to think it refers to setting Message Properties on a message so I think we should be explicit also about setting Message Properties on a Connection. Could be a second example.
So after looking at the text again, I think the example talking about setting Message Properties on a message should be removed, see comment in the pull request.
Thanks!!

A Preconnection Object holds properties reflecting the application's requirements and preferences for the transport.  These include Selection Properties for selecting protocol stacks and paths, as well  <ins> as Connection Properties and Message Properties for configuration of the detailed operation of the selected Protocol Stacks on a per- Connection and Message level. </ins> The protocol(s) and path(s) selected as candidates during establishment are determined and configured using these properties.


"Connection Priority" on one Connection does not change it on the other Connections in the same Connection Group. <del> Message Properties are also not entangled.  For example, changing "Lifetime" (see msg-lifetime) of a Message will only affect a single Message on a single Connection. </del>  A new Connection created by Clone can have a Message Framer assigned via the optional "framer" parameter of the Clone Action.  If this

"Connection Priority" on one Connection does not change it on the other Connections in the same Connection Group.  <ins> Message Properties set on a Connection also apply only to that Connection. </ins> A new Connection created by Clone can have a Message Framer assigned via the optional "framer" parameter of the Clone Action.  If this


Sending a Message with Message Properties inconsistent with the Selection Properties of the Connection yields an error. <del> Connection Properties describe the default behavior for all Messages on a Connection.  If a Message Property contradicts a Connection Property, and if this per-Message behavior can be supported, it overrides the Connection Property for the specific Message.  For example, if "Reliable Data Transfer (Connection)" is set to "Require" and a protocol with configurable per-Message reliability is used, setting "Reliable Data Transfer (Message)" to "false" for a particular Message will allow this Message to be sent without any reliability guarantees.  Changing the Reliable Data Transfer property on Messages is only possible for Connections that were established enabling the Selection Property "Configure Per-Message Reliability". </del>  The following Message Properties are supported:

Sending a Message with Message Properties inconsistent with the Selection Properties of the Connection yields an error.  <ins> If a Message Property contradicts a Connection Property, and if this per-Message behavior can be supported, it overrides the Connection Property for the specific Message.  For example, if "Reliable Data Transfer (Connection)" is set to "Require" and a protocol with configurable per-Message reliability is used, setting "Reliable Data Transfer (Message)" to "false" for a particular Message will allow this Message to be sent without any reliability guarantees.  Changing the Reliable Data Transfer property on Messages is only possible for Connections that were established enabling the Selection Property "Configure Per-Message Reliability". </ins> The following Message Properties are supported:


IP address (IPv4 or IPv6 address): <del> Interface name (string): </del>  Note that an IPv6 address specified with a scope (e.g. "2001:db8:4920:e29d:a420:7461:7073:0a%en0") is equivalent to

Explain the different use-cases on specifying interfaces to
There's something obvious I'm not seeing: if I want to specify a multicast or link-local address for an endpoint, why not simply specify that address? What's the use of adding the interface into it? Some addresses will inherently limit interfaces; I thought interface preference was about preferring wifi to 4G, etc.
Like, in section 6.1.1:
Multicast groups can be scoped (see RFC7346), thus, may only be available on certain interfaces or even have different meanings based on the interface. So for Interface- and Link-Local scope, specifying the interface is strictly required. For others use cases (like coping with non-unique use of RFC1918 addresses) this may also be helpful.
Why does the API have both the ability to say LocalSpecifier.WithInterface and to set interface requirements on TransportProperties? Why not just keep the latter and remove the former?
There's history here: LocalSpecifier.WithInterface predates explicitly setting the interface name in interface requirements. There might be reasons to keep this: LocalSpecifier is more likely to show up in application UI than in transport properties. Any text proposed to keep both would need to have an unambiguous way to resolve conflicts between the two. This might be a source of hard-to-debug errors.
To discuss next interim

IP address (IPv4 or IPv6 address):  <ins> Interface name (string), e.g., to qualify link-local or multicast addresses (see ifspec for details): </ins> Note that an IPv6 address specified with a scope (e.g. "2001:db8:4920:e29d:a420:7461:7073:0a%en0") is equivalent to


6.1.2.  An Endpoint can have an alternative definition when using different protocols.  For example, a server that supports both TLS/TCP and QUIC may be accessible on two different port numbers depending on which

6.1.2. <ins> Note that this API has multiple ways to constrain and prioritize endpoint candidates based on the network interface: Specifying an interface on a RemoteEndpoint qualifies the scope of the remote endpoint, e.g., for link-local addresses. Specifying an interface on a LocalEndpoint explicitly binds all candidates derived from this endpoint to use the specified interface. Specifying an interface using the "interface" Selection Property (prop-interface) or indirectly via the "pvd" Selection Property (prop-pvd) influences the selection among the available candidates. While specifying an interface on an endpoint restricts the candidates available for connection establishment in the Pre-Establishment Phase, the Selection Properties prioritize and constrain the connection establishment. 6.1.3. </ins> An Endpoint can have an alternative definition when using different protocols.  For example, a server that supports both TLS/TCP and QUIC may be accessible on two different port numbers depending on which