KEP-4962: Standardizing the Representation of Cluster Network Topology #4965
Conversation
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
the
sig/network
|
Welcome @dmitsh! |
k8s-ci-robot
added
the
needs-ok-to-test
|
Hi @dmitsh. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/XL
|
/cc @aojea |
|
@dmitsh: GitHub didn't allow me to request PR reviews from the following users: tardieu, arsenetar, brianhammons. Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
|
||
| ### Network QoS Annotation | ||
| Format: `network.qos.kubernetes.io/switches: <QoS>` | ||
| - `<QoS>`: A JSON object where each key is a switch name (matching the network topology label) with a value containing: |
There was a problem hiding this comment.
So this object contains N items (of below structure), where N is the number of predefined topology units (accelerator, block, datacenter, zone), right?
What I want to ensure is that those needs to be changed/updated when the cluster grows/shrinks (i.e. that the don't define distance between nodes themselves). But given that for a given node its contents don't depend on other nodes (rather on placements in the physical network), this seems to be fine.
There was a problem hiding this comment.
@wojtek-t , that's correct, each node will contain QoS metrics between the node and every reachable switch.
|
/assign @johnbelamaric for sig-architecture |
| - `<switch-name>`: Unique identifier for the switch | ||
|
|
||
| ### Network QoS Annotation | ||
| Format: `network.qos.kubernetes.io/switches: <QoS>` |
There was a problem hiding this comment.
This is not really the switch, right, is the interface in the node that connects to the switch ... and we already have properties to define attributes on the network interfaces with DRA, see slice 14 https://docs.google.com/presentation/d/1Vdr7BhbYXeWjwmLjGmqnUkvJr_eOUdU0x-JxfXWxUT8/edit#slide=id.g2f750386db2_5_0 so I don't feel we need this additional annotation here
There was a problem hiding this comment.
It is not a NIC on the node. These are QoS metrics from the node to every reachable switch. Also, the "switch" in this context could be a physical network device, or an aggregated entity defined by a CSP. For example, AWS returns 3 levels of switches per node, but the actual number of physical switches is unknown.
There was a problem hiding this comment.
These are QoS metrics from the node to every reachable switch
how is the Node connected to the first switch :) ?
| network.qos.kubernetes.io/switches: { | ||
| "nvl10": { | ||
| "latency": "2us", | ||
| "bandwidth": "100Gbps" | ||
| }, | ||
| "sw11": { | ||
| "latency": "50us", | ||
| "bandwidth": "40Gbps" | ||
| }, | ||
| "sw21": { | ||
| "latency": "500us", | ||
| "bandwidth": "20Gbps" | ||
| }, | ||
| "sw31": { | ||
| "latency": "1ms", | ||
| "bandwidth": "10Gbps" | ||
| } |
There was a problem hiding this comment.
These are network interfaces on the node, I think we should better model them with DRA https://github.com/kubernetes/enhancements/pull/4965/files#r1846865095 , that also allows us to provide dynamic capabilities to the interfaces
There was a problem hiding this comment.
Again, as mentioned in the earlier comment, these QoS numbers represent node-to-reachable-switch metrics. They are not per NIC.
There was a problem hiding this comment.
does it mean that some switches may not be connected directly?
how then the latency and bandwidth are obtained to guarantee those values?
| Format: `network.topology.kubernetes.io/<nw-switch-type>: <switch-name>` | ||
| - `<nw-switch-type>`: Logical type of the network switch (can be one of the reserved names or a custom name) | ||
| - Reserved names: `accelerator`, `block`, `datacenter`, `zone` |
There was a problem hiding this comment.
this is the part we need to loop in SIG architecture, I briefly touched on this with @thockin and it seems it took some time to settle on the region/zone.
So my understanding is that we need to model a hierarchy, this KEP suggest to use nested structures: zone > datacenter > block > accelerator , but we should at least describe in alternative why weights are not better than this, it seems to me that weights are more easy to standardize different topologies, as you only focus on the distance provided between layers, and can support different architectures with multiple layers
There was a problem hiding this comment.
This KEP is proposing to use reserved network types for typical network architectures, while allowing to extend network topology using custom network types.
We are providing means for weighted approach by specifying distance and/or bandwidth, latency or other metrics.
These are actual measurable physical characteristics of the network, and will be more accurate than specifying static weights.
Once again, we are providing QoS between a node and a switch, so the distance is the number of hops between the node and the switch. Same goes for bandwidth/latency.
| This proposal is designed with extensibility in mind, enabling the use of custom network types. This ensures that the standard can adapt to future advancements in cluster networking without requiring significant overhauls. | ||
|
|
||
| For custom network types, Network QoS Annotations are required, with distance being the minimum mandatory metric. Specifying latency and bandwidth is optional, but including them can offer a more detailed view of link performance, enabling more efficient scheduling decisions. |
There was a problem hiding this comment.
IIUIC the use of custom network types means that environment that use them may not be compatible with other environments, that practically removes all the benefits of standarization, another thing where I think weighted/distances model can help better
|
|
||
| The same network topology depicted in Example 2 can be represented using custom network types. | ||
|
|
||
| Let's use `tor` for top-of-rack switches, `area` for the second level of switches, and `center` for the third level. |
There was a problem hiding this comment.
After seen this example I'm definitevely not in favor of custom types as it is impossible for a generic tool to infer the distance between these custom types ... it will also create fragmentation and incompatibility as multiple tools can define the same name with different meaning ... the example also talks about levels, that reinforces my idea of weights, so something like network.topology.kubernetes.io/tier: 1
In this example network.topology.kubernetes.io/tor: sw13 will be
- Node:
network.topology.kubernetes.io/tier: 1 - ResourceSlice:
apiVersion: resource.k8s.io/v1alpha3
kind: ResourceSlice
…
spec:
devices:
- basic:
attributes:
tier:
int: 1
type:
string: nvlink
ip:
string: 10.1.1.1/24
latency:
string: "50us"
bandwith:
string: "100gbps"
There was a problem hiding this comment.
When using custom network types, it is mandatory to define distance in the QoS annotation.
We explicitly expressed that in the KEP.
| - Gang-scheduling auto-scaler | ||
| - DRA scheduler plugin | ||
|
|
||
| ### Goals |
There was a problem hiding this comment.
Are there existing topology formats or consumers that we want Kubernetes to integrate with? If so, are these integrations goals or non-goals?
There was a problem hiding this comment.
To the best of my knowledge, only EKS exposes their custom node labels for network layers.
The goal of this KEP is to create a standard way of describing switch network topology.
Ultimately, we want to use this standard in development of Kubernetes-native network-aware scheduler plugin for multi-node workloads. We put this task as a "no goal", as it would be an independent effort.
There was a problem hiding this comment.
Ultimately, we want to use this standard in development of Kubernetes-native network-aware scheduler plugin for multi-node workloads.
I'm (still) not seeing why that development work requires a KEP. An out-of-tree, out-of-project custom scheduler could use labels with keys outside of Kubernetes namespaces.
Let me write my thoughts on this: I see value in a predefined set of labels that describe topology, similar to what we have with 1, It can be other names, but having a number of predefined levels standard across environments will benefit the projects that implement scheduling based on network topology, I think 4 levels should be ok for most environments. And now the friction points:
|
|
OK, but responding to #4965 (comment): if the labels (and annotations?) were inside If we can't articulate that, we should have pause. Maybe some other project should define the standard (not Kubernetes). |
| <!-- | ||
| Why should this KEP _not_ be implemented? | ||
| --> |
There was a problem hiding this comment.
How will we describe multihomed nodes?
For example, a node directly connects to two neighbours, and to two top of rack switches. Pod to pod communication can use either best-effort routing or computed paths with source route metadata added to packets on egress.
Kubernetes doesn't say you can't do that, but this standard implies a hierarchical network layout. Are we tacitly backing a particular network management paradigm?
There was a problem hiding this comment.
| <!-- | |
| Why should this KEP _not_ be implemented? | |
| --> | |
| This proposal assumes that the network topology is hierarchical. |
There was a problem hiding this comment.
This is not accurate.
You can easy describe a node connected to multiple switches:
A simple example, where we assume that performance of switches is comparable:
Labels:
network.topology.kubernetes.io/level_a: sw01
network.topology.kubernetes.io/level_b: sw02Annotations:
network.qos.kubernetes.io/switches: {
"sw01": {
"distance": 1
},
"sw02": {
"distance": 1
}
}An example, where we provide performance metrics:
Labels:
network.topology.kubernetes.io/level_a: sw01
network.topology.kubernetes.io/level_b: sw02Annotations:
network.qos.kubernetes.io/switches: {
"sw01": {
"distance": 1,
"bandwidth": "40Gbps"
},
"sw02": {
"distance": 1,
"bandwidth": "60Gbps"
}
}
I do see value on So I think that aiming for something similar for the new type of workloads that require a more granular definition of network topology makes sense ... I really think we should work on what are the new |
|
Following up #4965 (comment) I still have concerns; I'll explain more. We can encourage cloud providers to define their own labels as well. I agree about the value So this KEP wouldn't persuade providers and / or hardware vendors to define their own parallel, provider-specific labels - but encouraging that kind of guidance would bring something on top of a common denominator of labels that Kubernetes does define. I remain worried about tacitly promoting one network paradigm (a switching hierarchy), especially for a project that might be around another 10 years. This KEP should articulate its alternatives and make clear why we picked the approach we select. |
agree, but I see this is about network hierarchy, switching hierarchy is just an implementation of a network hierarchy Level 1: there are things that are closest , this can be a switch a rack or the PCI bus, the implementer decide this |
|
I guess as an actual alternative, we could define an API for representing network topology. For example, custom resources:
Maybe also:
Remember, alternatives in KEP feedback don't have to be the choice we select. They can instead be viable options we rule out but still list as alternatives we considered. |
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
I want to stress that we are not advocating for or favoring any specific network hierarchy. On the contrary, our goal is to make the system flexible and adaptable to any type of network. To achieve this, we propose using reserved types for common configurations, enabling simplicity and brevity. Additionally, we recommend supporting custom types as a more generic solution, where key details like the number of hops (as a minimum input) and additional performance metrics can be specified. This information will be crucial for optimally scheduling multi-node training jobs or sets of interdependent, data-intensive services. It ensures that network-aware decisions can be made to enhance performance and efficiency. |
Signed-off-by: Dmitry Shmulevich <[email protected]>
k8s-ci-robot
added
size/XXL
aojea
changed the title
k8s-ci-robot
added
size/XL
| @@ -154,13 +154,20 @@ Items marked with (R) are required *prior to targeting to a milestone / release* | |||
|
|
|||
| ## Summary | |||
|
|
|||
| This document proposes a standard for declaring switch network topology in Kubernetes clusters, representing the hierarchy of nodes, switches, and interconnects. In this context, a switch can refer to a physical network device or a collection of such devices with close proximity and functionality. | |||
| This document proposes a standard for declaring switch network topology in Kubernetes clusters, | |||
There was a problem hiding this comment.
I would remove the reference to switch network topology as has connotations, I think we are talking here about "Hierarchical Network Architectures", as we have hierarchy of "networks" with different levels, and the components at the same level should expect the same network properties in terms of performance
| - `latency`: Link latency (e.g., 200 ms), optional | ||
| - `bandwidth`: Link bandwidth (e.g., 100 Gbps), optional | ||
| Format: `network.topology.kubernetes.io/<nw-switch-type>: <switch-name>`, where | ||
| - `<nw-switch-type>` defines the characteristic of the network switch. The term `switch` can refer to a physical network device or a collection of such devices with close proximity and functionality. |
dmitsh
force-pushed
the
ds-topology
branch
5 times, most recently
from
8d2ba06
to
6253878
Compare
|
|
||
| ## Summary | ||
|
|
||
| This document proposes a standard for declaring switch network topology in Kubernetes clusters, representing the hierarchy of nodes, switches, and interconnects. In this context, a switch can refer to a physical network device or a collection of such devices with close proximity and functionality. |
There was a problem hiding this comment.
Kubernetes has generally shied away from describing physical attributes vs user intent.
One of the challenges that this proposal needs to address is how this can be future-proofed against changes in underlying technology.
|
|
||
| Beyond CSPs, certain on-premises clusters support network topology discovery, though this capability depends on the features of the underlying switch network vendors. | ||
|
|
||
| An open-source project, [Topograph](https://github.com/NVIDIA/topograph), has implemented these approaches and is successfully deployed in production environments. |
There was a problem hiding this comment.
It would be good to discuss what this proposal does differently than topograph?
- Is it porting over the general concepts? What worked/did not?
- Does it address the problems with the topograph project?
Signed-off-by: Dmitry Shmulevich <[email protected]>
|
@dmitsh: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| Having these labels available in Kubernetes clusters will help in designing cloud agnostic scheduling systems. | ||
| The scheduler will prioritize switches according to the order outlined above, providing a standardized approach for network-aware scheduling across a range of configurations. | ||
|
|
||
| ### User Stories (Optional) |
There was a problem hiding this comment.
| ### User Stories (Optional) | |
| ### User Stories |
| The scheduler plugin reconstructs the cluster network topology by interpreting `network.topology.kubernetes.io/...` node labels. | ||
| Using this topology information, it optimally binds pods to suitable nodes, reducing overall latency and improving performance. |
There was a problem hiding this comment.
This is the solution more than mart of the story.
Standardizing Cluster Network Topology Representation