Question: <ins> <?xml version="1.0" encoding="utf-8"?> <!-- Generator: Adobe Illustrator 19.2.1, SVG Export Plug-In . SVG Version: 6.00 Build 0)  --> <svg version="1.1" id="svg2" inkscape:export-filename="/usr/local/google/home/thockin/src/kubernetes/docs/services-ipvs-overview.png" inkscape:export-xdpi="76.910004" inkscape:export-ydpi="76.910004" inkscape:version="0.48.4 r9939" sodipodi:docname="services-ipvs-overview.svg" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" xmlns:svg="http://www.w3.org/2000/svg" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" viewBox="0 0 656 486" style="enable-background:new 0 0 656 486;" xml:space="preserve"> <style type="text/css"> .st0{fill:none;stroke:#000000;stroke-width:0.8425;} .st1{fill:none;stroke:#000000;stroke-width:2.0786;} .st2{stroke:#000000;stroke-width:2.0786;stroke-linecap:round;} .st3{fill:none;stroke:#000000;stroke-width:5;} .st4{stroke:#000000;stroke-width:4.7036;stroke-linecap:round;} .st5{fill:none;stroke:#000000;stroke-width:4.2928;} .st6{stroke:#000000;stroke-width:4.2928;stroke-linecap:round;} .st7{fill:#85BFF1;stroke:#000000;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;} .st8{font-family:'ArialMT';} .st9{font-size:32px;} .st10{font-size:24px;} .st11{fill:none;stroke:#000000;stroke-width:5.1965;} .st12{stroke:#000000;stroke-width:5.1965;stroke-linecap:round;} .st13{fill:#F1CB85;stroke:#000000;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;} .st14{fill:#B9F185;stroke:#000000;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;} .st15{fill:#EDC1F8;stroke:#000000;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;} .st16{fill:#FFE680;stroke:#000000;stroke-width:1.7787;} .st17{font-family:'MyriadPro-Regular';} .st18{font-size:40px;} </style> <sodipodi:namedview  bordercolor="#666666" borderopacity="1.0" id="base" inkscape:current-layer="layer1" inkscape:cx="115.9218" inkscape:cy="392.30545" inkscape:document-units="px" inkscape:pageopacity="0.0" inkscape:pageshadow="2" inkscape:window-height="822" inkscape:window-maximized="0" inkscape:window-width="1552" inkscape:window-x="203" inkscape:window-y="50" inkscape:zoom="1.0318369" pagecolor="#ffffff" showgrid="false"> </sodipodi:namedview> <g id="layer1" inkscape:groupmode="layer" inkscape:label="Layer 1"> <rect id="rect3116" height="105.80995" width="645.257723" class="st0" y="350.381592" x="5	.318098"/> <rect id="rect3115" x="104.3" y="72.5" class="st0" width="545.8" height="253.1"/> <g id="g4178-3-0" transform="matrix(1,0,0,-0.92578962,15.303948,1193.1996)"> <path id="path4174-3-7" inkscape:connector-curvature="0" class="st1" d="M485.1,1264.5c0-71.3,0-71.3,0-71.3"/> <path id="path4176-9-1" inkscape:flatsided="true" inkscape:randomized="0" inkscape:rounded="0" inkscape:transform-center-y="-2.5" sodipodi:arg1="2.6179939" sodipodi:arg2="3.6651914" sodipodi:cx="308.85715" sodipodi:cy="753.79077" sodipodi:r1="10" sodipodi:r2="5" sodipodi:sides="3" sodipodi:type="star" class="st2" d=" M476.8,1202.5l8.7-15l8.7,15H476.8z"/> </g> <g id="g4178-3-8" transform="matrix(-0.83212211,-1.1231515,0.89570092,-0.89430772,-72.942206,1678.3161)" inkscape:transform-center-x="84.098741" inkscape:transform-center-y="1.4572787e-05"> <path id="path4174-3-4" inkscape:connector-curvature="0" class="st1" d="M501.5,1075.9c0-71.3,0-71.3,0-71.3"/> <path id="path4176-9-0" inkscape:flatsided="true" inkscape:randomized="0" inkscape:rounded="0" inkscape:transform-center-y="-2.5" sodipodi:arg1="2.6179939" sodipodi:arg2="3.6651914" sodipodi:cx="308.85715" sodipodi:cy="753.79077" sodipodi:r1="10" sodipodi:r2="5" sodipodi:sides="3" sodipodi:type="star" class="st2" d=" M493.2,1013.9l8.7-15l8.7,15L493.2,1013.9z"/> </g> <g id="g4324" transform="matrix(0.96592583,0.25881905,0.25881905,-0.96592583,-272.81074,1126.238)"> <path id="path4174-3-2" inkscape:connector-curvature="0" class="st3" d="M362,988.4C260,874.6,260,874.6,260,874.6"/> <path id="path4176-9-9" inkscape:flatsided="true" inkscape:randomized="0" inkscape:rounded="0" inkscape:transform-center-y="-2.5" sodipodi:arg1="2.6179939" sodipodi:arg2="3.6651914" sodipodi:cx="308.85715" sodipodi:cy="753.79077" sodipodi:r1="10" sodipodi:r2="5" sodipodi:sides="3" sodipodi:type="star" class="st4" d=" M253,881.5l-4.8-18.4l17.8,6.9L253,881.5z"/> </g> <g id="g4324-8" transform="matrix(-0.96592583,0.25881905,-0.25881905,-0.96592583,654.32964,1126.238)"> <path id="path4174-3-2-7" inkscape:connector-curvature="0" class="st3" d="M76.1,911.8C-25.9,798-25.9,798-25.9,798"/> <path id="path4176-9-9-3" inkscape:flatsided="true" inkscape:randomized="0" inkscape:rounded="0" inkscape:transform-center-y="-2.5" sodipodi:arg1="2.6179939" sodipodi:arg2="3.6651914" sodipodi:cx="308.85715" sodipodi:cy="753.79077" sodipodi:r1="10" sodipodi:r2="5" sodipodi:sides="3" sodipodi:type="star" class="st4" d=" M-32.9,804.9l-4.8-18.4l17.8,6.9L-32.9,804.9z"/> </g> <g id="g4178-3-9" transform="matrix(1,0,0,-1.3566066,-154.75999,1749.5431)"> <path id="path4174-3-8" inkscape:connector-curvature="0" class="st5" d="M485.1,1103.7c0-71.3,0-71.3,0-71.3"/> <path id="path4176-9-5" inkscape:flatsided="true" inkscape:randomized="0" inkscape:rounded="0" inkscape:transform-center-y="-2.5" sodipodi:arg1="2.6179939" sodipodi:arg2="3.6651914" sodipodi:cx="308.85715" sodipodi:cy="753.79077" sodipodi:r1="10" sodipodi:r2="5" sodipodi:sides="3" sodipodi:type="star" class="st6" d=" M476.8,1041.6l8.7-15l8.7,15H476.8z"/> </g> <g id="g3937" transform="translate(-212.35646,769.73074)"> <g id="g3868" transform="matrix(0.88792337,0,0,1,43.50975,6.5250001e-6)"> <rect id="rect2985" x="196.7" y="-408.1" class="st7" width="224.3" height="50"/> <g id="g3861"> <text transform="matrix(1 0 0 1 203.3918 -377.1545)" class="st8 st9">Backend Pod 1</text> </g> </g> <g id="g3868-7" transform="matrix(0.88792337,0,0,1,262.00231,6.5250001e-6)"> <rect id="rect2985-1" x="196.7" y="-408.1" class="st7" width="224.3" height="50"/> <g id="g3861-9"> <text transform="matrix(1 0 0 1 203.3919 -377.1545)" class="st8 st9">Backend Pod 2</text> </g> </g> <g id="g3868-3" transform="matrix(0.88792337,0,0,1,480.49489,6.5250001e-6)"> <rect id="rect2985-2" x="196.7" y="-408.1" class="st7" width="224.3" height="50"/> <g id="g3861-3"> <text transform="matrix(1 0 0 1 203.392 -377.1545)" class="st8 st9">Backend Pod 3</text> </g> </g> </g> <g id="g4178-3" transform="matrix(0.91192623,-0.41035418,-0.37990164,-0.84425184,113.60453,1385.4009)"> <path id="path4174-3" inkscape:connector-curvature="0" class="st11" d="M664.6,1154.3c0-71.3,0-71.3,0-71.3"/> <path id="path4176-9" inkscape:flatsided="true" inkscape:randomized="0" inkscape:rounded="0" inkscape:transform-center-y="-2.5" sodipodi:arg1="2.6179939" sodipodi:arg2="3.6651914" sodipodi:cx="308.85715" sodipodi:cy="753.79077" sodipodi:r1="10" sodipodi:r2="5" sodipodi:sides="3" sodipodi:type="star" class="st12" d=" M656.2,1092.3l8.7-15l8.7,15H656.2z"/> </g> <g id="g4090" transform="matrix(0.89067003,0,0,1,-194.97295,-142.36286)"> <rect id="rect2985-4" x="388" y="235.5" class="st13" width="224.3" height="58.6"/> <g id="g3861-6" transform="translate(217.6177,652.82516)"> <text transform="matrix(1 0 0 1 233.7419 -377.2347)" class="st8 st9">Client </text> </g> </g> <g id="g4168" transform="matrix(0.89067003,0,0,1,199.65922,-24.251798)"> <rect id="rect2985-4-0" x="216.9" y="119.8" class="st14" width="250" height="58.6"/> <g id="g3861-6-2" transform="translate(34.747433,534.26287)"> <text transform="matrix(1 0 0 1 226.622 -376.7865)" class="st8 st9">kube-proxy</text> </g> </g> <g id="g4168-5" transform="translate(188.04333,-29.041887)"> <g id="g4238" transform="translate(22.087429,-86.34177)"> <rect id="rect2985-4-0-6" x="199.9" y="119.8" class="st15" width="191.8" height="58.6"/> <g id="g3861-6-2-6" transform="translate(39.107429,534.26287)"> <text transform="matrix(1 0 0 1 184.7109 -377.1544)" class="st8 st9">apiserver</text> </g> </g> </g> <path id="path3884" inkscape:connector-curvature="0" class="st16" d="M332.5,202c-14.5,0-27,8.6-33,21.2c-5-0.9-10.3-1.4-15.8-1.4 c-34.3,0-62.2,19.1-62.2,42.6s27.8,42.6,62.2,42.6c16.9,0,32.2-4.6,43.4-12.2c7.4,16.3,34,28.3,65.7,28.3 c33.5,0,61.4-13.4,66.8-31.1c17-5.3,28.6-15.7,28.6-27.7c0-17.3-24.4-31.4-54.4-31.4c-8.5,0-16.5,1.1-23.6,3.1 c-2-10.5-14.8-18.6-30.3-18.6c-5.8,0-11.2,1.2-15.9,3.1C357.7,209.5,345.9,202,332.5,202L332.5,202z"/> <g id="g3861-6-28" transform="matrix(0.89067003,0,0,1,34.810052,629.61733)"> <text transform="matrix(1 0 0 1 259.8564 -379.173)" class="st8 st9">ServiceIP</text> <text transform="matrix(1 0 0 1 240.8564 -339.173)" class="st8 st9" fill="red">(Virtual Server) </text> </g> <text transform="matrix(1 0 0 1 537.5965 317.8163)" class="st17 st18">Node</text> <text transform="matrix(1 0 0 1 230 450)" class="st8 st9" fill="red">(Real Server) </text> </g> </svg> </ins>

<!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!--Required Information--This is a... [ ] Feature Request [x] Bug Report Problem: For users using K8S versions 1.8 and lower that don't have the fix contained in it may be necessary to set for the OpenStack cloud provider when installing on OpenStack clusters that expose their Cinder endpoint using paths rather than ports to avoid auto-detection errors such as: This includes for users trying out K8S on a devstack environment, and potentially other deployment tools. In this situation it is possible to resolve using: Valid values for are: With being the default. Proposed Solution: Highlight the bs-version parameter as optional (so not minimal configuration) but required in some circumstances. Page to Update:


Question: <ins> --- title: Adding Linux worker nodes content_type: task weight: 50 --- <!-- overview --> This page explains how to add Linux worker nodes to a kubeadm cluster. ## {{% heading "prerequisites" %}} * Each joining worker node has installed the required components from [Installing kubeadm](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/), such as, kubeadm, the kubelet and a {{< glossary_tooltip term_id="container-runtime" text="container runtime" >}}. * A running kubeadm cluster created by `kubeadm init` and following the steps in the document [Creating a cluster with kubeadm](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/). * You need superuser access to the node. <!-- steps --> ## Adding Linux worker nodes To add new Linux worker nodes to your cluster do the following for each machine: 1. Connect to the machine by using SSH or another method. 1. Run the command that was output by `kubeadm init`. For example: ```bash sudo kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash> ``` ### Additional information for kubeadm join {{< note >}} To specify an IPv6 tuple for `<control-plane-host>:<control-plane-port>`, IPv6 address must be enclosed in square brackets, for example: `[2001:db8::101]:2073`. {{< /note >}} If you do not have the token, you can get it by running the following command on the control plane node: ```bash sudo kubeadm token list ``` The output is similar to this: ```console TOKEN                    TTL  EXPIRES              USAGES           DESCRIPTION            EXTRA GROUPS 8ewj1p.9r9hcjoqgajrj4gi  23h  2018-06-12T02:51:28Z authentication,  The default bootstrap  system: signing          token generated by     bootstrappers: 'kubeadm init'.        kubeadm: default-node-token ``` By default, node join tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the control plane node: ```bash sudo kubeadm token create ``` The output is similar to this: ```console 5didvk.d09sbcov8ph2amjw ``` If you don't have the value of `--discovery-token-ca-cert-hash`, you can get it by running the following commands on the control plane node: ```bash sudo cat /etc/kubernetes/pki/ca.crt | openssl x509 -pubkey  | openssl rsa -pubin -outform der 2>/dev/null |  openssl dgst -sha256 -hex | sed 's/^.* //' ``` The output is similar to: ```console 8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78 ``` The output of the `kubeadm join` command should look something like: ``` [preflight] Running pre-flight checks ... (log output of join workflow) ... Node join complete: * Certificate signing request sent to control-plane and response received. * Kubelet informed of new secure connection details. Run 'kubectl get nodes' on control-plane to see this machine join. ``` A few seconds later, you should notice this node in the output from `kubectl get nodes`. (for example, run `kubectl` on a  control plane node). {{< note >}} As the cluster nodes are usually initialized sequentially, the CoreDNS Pods are likely to all run on the first control plane node. To provide higher availability, please rebalance the CoreDNS Pods with `kubectl -n kube-system rollout restart deployment coredns` after at least one new node is joined. {{< /note >}} ## {{% heading "whatsnext" %}} * See how to [add Windows worker nodes](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/). </ins>

Please note: This issue is not suitable for new contributors since you need some familiarity with how Kubernetes works. The Kubernetes website currently lacks a guide on creating Windows/Linux nodes. See for more details. SIG Windows opted to remove the guide that did exist for creating Windows nodes, in PR Write a new task page about creating nodes for all operating systems. This could, potentially, be in a tabbed format as mentions in the issue referenced above. We would also need the entry reflected on the creates the Linux task page by moving / refactoring the existing "join node" contents from the "create a cluster" page: creates the Windows task page by taking the content from at the end of the Windows page we add note/disclaimer whatever that says that CNI on Windows needs special handling and we link to the various CNI choices at: adds tabs in the to choose between OS for joining nodes. /priority backlog /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Problem Statement: /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Thank you for filing this
/sig cluster-lifecycle
If you know how to set up new nodes (with, at least, the tool): you can work on this issue. You don't need anyone's permission to open a pull request. All we ask is that you read our and that you abide by the project's code of conduct.
/assign
Relevant to
Are you still working on this issue?
/assign sending PR soon.


Question: <ins> --- title: Adding Linux worker nodes content_type: task weight: 50 --- <!-- overview --> This page explains how to add Linux worker nodes to a kubeadm cluster. ## {{% heading "prerequisites" %}} * Each joining worker node has installed the required components from [Installing kubeadm](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/), such as, kubeadm, the kubelet and a {{< glossary_tooltip term_id="container-runtime" text="container runtime" >}}. * A running kubeadm cluster created by `kubeadm init` and following the steps in the document [Creating a cluster with kubeadm](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/). * You need superuser access to the node. <!-- steps --> ## Adding Linux worker nodes To add new Linux worker nodes to your cluster do the following for each machine: 1. Connect to the machine by using SSH or another method. 1. Run the command that was output by `kubeadm init`. For example: ```bash sudo kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash> ``` ### Additional information for kubeadm join {{< note >}} To specify an IPv6 tuple for `<control-plane-host>:<control-plane-port>`, IPv6 address must be enclosed in square brackets, for example: `[2001:db8::101]:2073`. {{< /note >}} If you do not have the token, you can get it by running the following command on the control plane node: ```bash sudo kubeadm token list ``` The output is similar to this: ```console TOKEN                    TTL  EXPIRES              USAGES           DESCRIPTION            EXTRA GROUPS 8ewj1p.9r9hcjoqgajrj4gi  23h  2018-06-12T02:51:28Z authentication,  The default bootstrap  system: signing          token generated by     bootstrappers: 'kubeadm init'.        kubeadm: default-node-token ``` By default, node join tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the control plane node: ```bash sudo kubeadm token create ``` The output is similar to this: ```console 5didvk.d09sbcov8ph2amjw ``` If you don't have the value of `--discovery-token-ca-cert-hash`, you can get it by running the following commands on the control plane node: ```bash sudo cat /etc/kubernetes/pki/ca.crt | openssl x509 -pubkey  | openssl rsa -pubin -outform der 2>/dev/null |  openssl dgst -sha256 -hex | sed 's/^.* //' ``` The output is similar to: ```console 8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78 ``` The output of the `kubeadm join` command should look something like: ``` [preflight] Running pre-flight checks ... (log output of join workflow) ... Node join complete: * Certificate signing request sent to control-plane and response received. * Kubelet informed of new secure connection details. Run 'kubectl get nodes' on control-plane to see this machine join. ``` A few seconds later, you should notice this node in the output from `kubectl get nodes`. (for example, run `kubectl` on a  control plane node). {{< note >}} As the cluster nodes are usually initialized sequentially, the CoreDNS Pods are likely to all run on the first control plane node. To provide higher availability, please rebalance the CoreDNS Pods with `kubectl -n kube-system rollout restart deployment coredns` after at least one new node is joined. {{< /note >}} ## {{% heading "whatsnext" %}} * See how to [add Windows worker nodes](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/). </ins>

/priority important-longterm
Relevant to
/retitle Write new task pages about creating Kubernetes nodes
/cc
Done in PR /close
Closing this issue. <detailsIn response to : Done in PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: | ---------------------------------------- | ---------- | ------- | ----------- | | `node.kubernetes.io/not-ready`           | NoExecute  | 1.13+   | DaemonSet pods will not be evicted when there are node problems such as a network partition. | | `node.kubernetes.io/unreachable`         | NoExecute  | 1.13+   | DaemonSet pods will not be evicted when there are node problems such as a network partition. | <del> | `node.kubernetes.io/disk-pressure`       | NoSchedule | 1.8+    | | | `node.kubernetes.io/memory-pressure`     | NoSchedule | 1.8+    | | </del> <ins> | `node.kubernetes.io/disk-pressure`       | NoSchedule | 1.8+    | DaemonSet pods tolerate disk-pressure attributes by default scheduler. | | `node.kubernetes.io/memory-pressure`     | NoSchedule | 1.8+    | DaemonSet pods tolerate memory-pressure attributes by default scheduler. | </ins> | `node.kubernetes.io/unschedulable`       | NoSchedule | 1.12+   | DaemonSet pods tolerate unschedulable attributes by default scheduler. | | `node.kubernetes.io/network-unavailable` | NoSchedule | 1.12+   | DaemonSet pods, who uses host network, tolerate network-unavailable attributes by default scheduler. |

In the table of of there are two descriptions missing of two tolerations kyes, one is another one is , their descriptions are missing.
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question:
Counter	 Counter of OpenAPI v2 spec regeneration count broken down by causing APIService name and reason.
apiservice

reason

   None

aggregator_openapi_v2_regeneration_duration	 ALPHA	 Gauge	 Gauge of OpenAPI v2 spec regeneration duration in seconds.
reason

   None

aggregator_unavailable_apiservice	 ALPHA	 Custom	 Gauge of APIServices which are marked as unavailable broken down by APIService name.
name

   None

aggregator_unavailable_apiservice_total	 ALPHA	 Counter	 Counter of APIServices which are marked as unavailable broken down by APIService name and reason.
name

reason

   None

apiextensions_openapi_v2_regeneration_count	 ALPHA	 Counter	 Counter of OpenAPI v2 spec regeneration count broken down by causing CRD name and reason.
crd

reason

   None

apiextensions_openapi_v3_regeneration_count	 ALPHA	 Counter	 Counter of OpenAPI v3 spec regeneration count broken down by group, version, causing CRD and reason.
crd

group

reason

version

   None

apiserver_admission_step_admission_duration_seconds_summary	 ALPHA	 Summary	 Admission sub-step latency summary in seconds, broken out for each operation and API resource and step type (validate or admit).
operation

rejected

type

   None

apiserver_admission_webhook_fail_open_count	 ALPHA	 Counter	 Admission webhook fail open count, identified by name and broken out for each admission type (validating or mutating).
name

type

   None

apiserver_admission_webhook_rejection_count	 ALPHA	 Counter	 Admission webhook rejection count, identified by name and broken out for each admission type (validating or admit) and operation. Additional labels specify an error type (calling_webhook_error or apiserver_internal_error if an error occurred; no_error otherwise) and optionally a non-zero rejection code if the webhook rejects the request with an HTTP status code (honored by the apiserver when the code is greater or equal to 400). Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
error_type

name

operation

rejection_code

type

   None

apiserver_admission_webhook_request_total	 ALPHA	 Counter	 Admission webhook request total, identified by name and broken out for each admission type (validating or mutating) and operation. Additional labels specify whether the request was rejected or not and an HTTP status code. Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
code

name

operation

rejected

type

   None

apiserver_audit_error_total	 ALPHA	 Counter	 Counter of audit events that failed to be audited properly. Plugin identifies the plugin affected by the error.
plugin

   None

apiserver_audit_event_total	 ALPHA	 Counter	 Counter of audit events generated and sent to the audit backend.
   None	 None

apiserver_audit_level_total	 ALPHA	 Counter	 Counter of policy levels for audit events (1 per request).
level

   None

apiserver_audit_requests_rejected_total	 ALPHA	 Counter	 Counter of apiserver requests rejected due to an error in audit logging backend.
   None	 None

apiserver_cache_list_fetched_objects_total	 ALPHA	 Counter	 Number of objects read from watch cache in the course of serving a LIST request
index

resource_prefix

   None

apiserver_cache_list_returned_objects_total	 ALPHA	 Counter	 Number of objects returned for a LIST request from watch cache
resource_prefix

   None

apiserver_cache_list_total	 ALPHA	 Counter	 Number of LIST requests served from watch cache
index

resource_prefix

   None

apiserver_cel_compilation_duration_seconds	 ALPHA	 Histogram
   None	 None

apiserver_cel_evaluation_duration_seconds	 ALPHA	 Histogram
   None	 None

apiserver_certificates_registry_csr_honored_duration_total	 ALPHA	 Counter	 Total number of issued CSRs with a requested duration that was honored, sliced by signer (only kubernetes.io signer names are specifically identified)
signerName

   None

apiserver_certificates_registry_csr_requested_duration_total	 ALPHA	 Counter	 Total number of issued CSRs with a requested duration, sliced by signer (only kubernetes.io signer names are specifically identified)
signerName

   None

apiserver_client_certificate_expiration_seconds	 ALPHA	 Histogram	 Distribution of the remaining lifetime on the certificate used to authenticate a request.
   None	 None

apiserver_crd_webhook_conversion_duration_seconds	 ALPHA	 Histogram	 CRD webhook conversion duration in seconds
crd_name

from_version

succeeded

to_version

   None

apiserver_current_inqueue_requests	 ALPHA	 Gauge	 Maximal number of queued requests in this apiserver per request kind in last second.
request_kind

   None

apiserver_delegated_authn_request_duration_seconds	 ALPHA	 Histogram	 Request latency in seconds. Broken down by status code.
code

   None

apiserver_delegated_authn_request_total	 ALPHA	 Counter	 Number of HTTP requests partitioned by status code.
code

   None

apiserver_delegated_authz_request_duration_seconds	 ALPHA	 Histogram	 Request latency in seconds. Broken down by status code.
code

   None

apiserver_delegated_authz_request_total	 ALPHA	 Counter	 Number of HTTP requests partitioned by status code.
code

   None

apiserver_egress_dialer_dial_duration_seconds	 ALPHA	 Histogram	 Dial latency histogram in seconds, labeled by the protocol (http-connect or grpc), transport (tcp or uds)
protocol

transport

   None

apiserver_egress_dialer_dial_failure_count	 ALPHA	 Counter	 Dial failure count, labeled by the protocol (http-connect or grpc), transport (tcp or uds), and stage (connect or proxy). The stage indicates at which stage the dial failed
protocol

stage

transport

apiserver_egress_dialer_dial_start_total	 ALPHA	 Counter	 Dial starts, labeled by the protocol (http-connect or grpc) and transport (tcp or uds).
protocol

transport

   None

apiserver_envelope_encryption_dek_cache_fill_percent	 ALPHA	 Gauge	 Percent of the cache slots currently occupied by cached DEKs.
   None	 None

apiserver_envelope_encryption_dek_cache_inter_arrival_time_seconds	 ALPHA	 Histogram	 Time (in seconds) of inter arrival of transformation requests.
transformation_type

   None

apiserver_flowcontrol_current_executing_requests	 ALPHA	 Gauge	 Number of requests in initial (for a WATCH) or any (for a non-WATCH) execution stage in the API Priority and Fairness subsystem
flow_schema

priority_level

   None

apiserver_flowcontrol_current_inqueue_requests	 ALPHA	 Gauge	 Number of requests currently pending in queues of the API Priority and Fairness subsystem
flow_schema

priority_level

apiserver_flowcontrol_current_limit_seats	 ALPHA	 Gauge	 current derived number of execution seats available to each priority level
priority_level

   None

apiserver_flowcontrol_current_r	 ALPHA	 Gauge	 R(time of last change)
priority_level

apiserver_flowcontrol_demand_seats	 ALPHA	 TimingRatioHistogram	 Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level)
priority_level

apiserver_flowcontrol_demand_seats_average	 ALPHA	 Gauge	 Time-weighted average, over last adjustment period, of demand_seats
priority_level

apiserver_flowcontrol_demand_seats_high_watermark	 ALPHA	 Gauge	 High watermark, over last adjustment period, of demand_seats
priority_level

apiserver_flowcontrol_demand_seats_smoothed	 ALPHA	 Gauge	 Smoothed seat demands
priority_level

apiserver_flowcontrol_demand_seats_stdev	 ALPHA	 Gauge	 Time-weighted standard deviation, over last adjustment period, of demand_seats
priority_level

   None

apiserver_flowcontrol_dispatch_r	 ALPHA	 Gauge	 R(time of last dispatch)
priority_level

   None

apiserver_flowcontrol_dispatched_requests_total	 ALPHA	 Counter	 Number of requests executed by API Priority and Fairness subsystem
flow_schema

priority_level

   None

apiserver_flowcontrol_epoch_advance_total	 ALPHA	 Counter	 Number of times the queueset's progress meter jumped backward
priority_level

success

   None

apiserver_flowcontrol_latest_s	 ALPHA	 Gauge	 S(most recently dispatched request)
priority_level

apiserver_flowcontrol_lower_limit_seats	 ALPHA	 Gauge	 Configured lower bound on number of execution seats available to each priority level
priority_level

   None

apiserver_flowcontrol_next_discounted_s_bounds	 ALPHA	 Gauge	 min and max, over queues, of S(oldest waiting request in queue) - estimated work in progress
bound

priority_level

   None

apiserver_flowcontrol_next_s_bounds	 ALPHA	 Gauge	 min and max, over queues, of S(oldest waiting request in queue)
bound

priority_level

apiserver_flowcontrol_nominal_limit_seats	 ALPHA	 Gauge	 Nominal number of execution seats configured for each priority level
priority_level

   None

apiserver_flowcontrol_priority_level_request_utilization	 ALPHA	 TimingRatioHistogram	 Observations, at the end of every nanosecond, of number of requests (as a fraction of the relevant limit) waiting or in any stage of execution (but only initial stage for WATCHes)
phase

priority_level

   None

apiserver_flowcontrol_priority_level_seat_utilization	 ALPHA	 TimingRatioHistogram	 Observations, at the end of every nanosecond, of utilization of seats for any stage of execution (but only initial stage for WATCHes)
priority_level

phase:executing

   map[phase:executing]

apiserver_flowcontrol_read_vs_write_current_requests	 ALPHA	 TimingRatioHistogram	 Observations, at the end of every nanosecond, of the number of requests (as a fraction of the relevant limit) waiting or in regular stage of execution
phase

request_kind

   None

apiserver_flowcontrol_rejected_requests_total	 ALPHA	 Counter	 Number of requests rejected by API Priority and Fairness subsystem
flow_schema

priority_level

reason

   None

apiserver_flowcontrol_request_concurrency_in_use	 ALPHA	 Gauge	 Concurrency (number of seats) occupied by the currently executing (initial stage for a WATCH, any stage otherwise) requests in the API Priority and Fairness subsystem
flow_schema

priority_level

   None

apiserver_flowcontrol_request_concurrency_limit	 ALPHA	 Gauge	 Shared concurrency limit in the API Priority and Fairness subsystem
priority_level

   None

apiserver_flowcontrol_request_dispatch_no_accommodation_total	 ALPHA	 Counter	 Number of times a dispatch attempt resulted in a non accommodation due to lack of available seats
flow_schema

priority_level

   None

apiserver_flowcontrol_request_execution_seconds	 ALPHA	 Histogram	 Duration of initial stage (for a WATCH) or any (for a non-WATCH) stage of request execution in the API Priority and Fairness subsystem
flow_schema

priority_level

type

   None

apiserver_flowcontrol_request_queue_length_after_enqueue	 ALPHA	 Histogram	 Length of queue in the API Priority and Fairness subsystem, as seen by each request after it is enqueued
flow_schema

priority_level

   None

apiserver_flowcontrol_request_wait_duration_seconds	 ALPHA	 Histogram	 Length of time a request spent waiting in its queue
execute

flow_schema

priority_level

apiserver_flowcontrol_seat_fair_frac	 ALPHA	 Gauge	 Fair fraction of server's concurrency to allocate to each priority level that can use it

apiserver_flowcontrol_target_seats	 ALPHA	 Gauge	 Seat allocation targets
priority_level

apiserver_flowcontrol_upper_limit_seats	 ALPHA	 Gauge	 Configured upper bound on number of execution seats available to each priority level
priority_level

   None

apiserver_flowcontrol_watch_count_samples	 ALPHA	 Histogram	 count of watchers for mutating requests in API Priority and Fairness
flow_schema

priority_level

   None

apiserver_flowcontrol_work_estimated_seats	 ALPHA	 Histogram	 Number of estimated seats (maximum of initial and final seats) associated with requests in API Priority and Fairness
flow_schema

priority_level

   None

apiserver_init_events_total	 ALPHA	 Counter	 Counter of init events processed in watch cache broken by resource type.
resource

   None

apiserver_kube_aggregator_x509_insecure_sha1_total	 ALPHA	 Counter	 Counts the number of requests to servers with insecure SHA1 signatures in their serving certificate OR the number of connection failures due to the insecure SHA1 signatures (either/or, based on the runtime environment)
   None	 None

apiserver_kube_aggregator_x509_missing_san_total	 ALPHA	 Counter	 Counts the number of requests to servers missing SAN extension in their serving certificate OR the number of connection failures due to the lack of x509 certificate SAN extension missing (either/or, based on the runtime environment)
   None	 None

apiserver_request_aborts_total	 ALPHA	 Counter	 Number of requests which apiserver aborted possibly due to a timeout, for each group, version, verb, resource, subresource and scope
group

resource

scope

subresource

verb

version

   None

apiserver_request_body_sizes	 ALPHA	 Histogram	 Apiserver request body sizes broken out by size.
resource

verb

   None

apiserver_request_filter_duration_seconds	 ALPHA	 Histogram	 Request filter latency distribution in seconds, for each filter type
filter

   None

apiserver_request_post_timeout_total	 ALPHA	 Counter	 Tracks the activity of the request handlers after the associated requests have been timed out by the apiserver
source

status

apiserver_request_sli_duration_seconds	 ALPHA	 Histogram	 Response latency distribution (not counting webhook duration) in seconds for each verb, group, version, resource, subresource, scope and component.
component

group

resource

scope

subresource

verb

version

   None

apiserver_request_slo_duration_seconds	 ALPHA	 Histogram	 Response latency distribution (not counting webhook duration) in seconds for each verb, group, version, resource, subresource, scope and component.
component

group

resource

scope

subresource

verb

version
	  	 1.27.0
   None

apiserver_request_terminations_total	 ALPHA	 Counter	 Number of requests which apiserver terminated in self-defense.
code

component

group

resource

scope

subresource

verb

version

   None

apiserver_request_timestamp_comparison_time	 ALPHA	 Histogram	 Time taken for comparison of old vs new objects in UPDATE or PATCH requests
code_path

   None

apiserver_selfrequest_total	 ALPHA	 Counter	 Counter of apiserver self-requests broken out for each verb, API resource and subresource.
resource

subresource

verb

   None

apiserver_storage_data_key_generation_duration_seconds	 ALPHA	 Histogram	 Latencies in seconds of data encryption key(DEK) generation operations.
   None	 None

apiserver_storage_data_key_generation_failures_total	 ALPHA	 Counter	 Total number of failed data encryption key(DEK) generation operations.
   None	 None

apiserver_storage_db_total_size_in_bytes	 ALPHA	 Gauge	 Total size of the storage database file physically allocated in bytes.
endpoint

   None

apiserver_storage_envelope_transformation_cache_misses_total	 ALPHA	 Counter	 Total number of cache misses while accessing key decryption key(KEK).
   None	 None

apiserver_storage_list_evaluated_objects_total	 ALPHA	 Counter	 Number of objects tested in the course of serving a LIST request from storage
resource

   None

apiserver_storage_list_fetched_objects_total	 ALPHA	 Counter	 Number of objects read from storage in the course of serving a LIST request
resource

   None

apiserver_storage_list_returned_objects_total	 ALPHA	 Counter	 Number of objects returned for a LIST request from storage
resource

   None

apiserver_storage_list_total	 ALPHA	 Counter	 Number of LIST requests served from storage
resource

   None

apiserver_storage_transformation_duration_seconds	 ALPHA	 Histogram	 Latencies in seconds of value transformation operations.
transformation_type

   None

apiserver_storage_transformation_operations_total	 ALPHA	 Counter	 Total number of transformations.
status

transformation_type

transformer_prefix

   None

apiserver_terminated_watchers_total	 ALPHA	 Counter	 Counter of watchers closed due to unresponsiveness broken by resource type.
resource

   None

apiserver_tls_handshake_errors_total	 ALPHA	 Counter	 Number of requests dropped with 'TLS handshake error from' error
   None	 None

apiserver_validating_admission_policy_check_duration_seconds	 ALPHA	 Histogram	  Validation admission latency for individual validation expressions in seconds, labeled by policy and further including binding, state and enforcement action taken.
enforcement_action

policy

policy_binding

state

   Validation admission latency for individual validation expressions in seconds, labeled by policy and param resource, further including binding, state and enforcement action taken.
enforcement_action

params

policy

policy_binding

state

validation_expression
	 None

apiserver_validating_admission_policy_check_total	 ALPHA	 Counter	  Validation admission policy check total, labeled by policy and further identified by binding, enforcement action taken, and state.
enforcement_action

policy

policy_binding

state

   Validation admission policy check total, labeled by policy and param resource, and further identified by binding, validation expression, enforcement action taken, and state.
enforcement_action

params

policy

policy_binding

state

validation_expression
	 None

apiserver_validating_admission_policy_definition_total	 ALPHA	 Counter	 Validation admission policy count total, labeled by state and enforcement action.
enforcement_action

state

   None

apiserver_watch_cache_events_dispatched_total	 ALPHA	 Counter	 Counter of events dispatched in watch cache broken by resource type.
resource

   None

apiserver_watch_cache_initializations_total	 ALPHA	 Counter	 Counter of watch cache initializations broken by resource type.
resource

   None

apiserver_watch_events_sizes	 ALPHA	 Histogram	 Watch event size distribution in bytes
group

kind

version

   None

apiserver_watch_events_total	 ALPHA	 Counter	 Number of events sent in watch clients
group

kind

version

   None

apiserver_webhooks_x509_insecure_sha1_total	 ALPHA	 Counter	 Counts the number of requests to servers with insecure SHA1 signatures in their serving certificate OR the number of connection failures due to the insecure SHA1 signatures (either/or, based on the runtime environment)
   None	 None

apiserver_webhooks_x509_missing_san_total	 ALPHA	 Counter	 Counts the number of requests to servers missing SAN extension in their serving certificate OR the number of connection failures due to the lack of x509 certificate SAN extension missing (either/or, based on the runtime environment)
   None	 None

attachdetach_controller_forced_detaches	 ALPHA	 Counter	 Number of times the A/D Controller performed a forced detach
   None	 None

attachdetach_controller_total_volumes	 ALPHA	 Custom	 Number of volumes in A/D Controller
plugin_name

state

   None

authenticated_user_requests	 ALPHA	 Counter	 Counter of authenticated requests broken out by username.
username

   None

authentication_attempts	 ALPHA	 Counter	 Counter of authenticated attempts.
result

   None

authentication_duration_seconds	 ALPHA	 Histogram	 Authentication duration in seconds broken out by result.
result

   None

authentication_token_cache_active_fetch_count	 ALPHA	 Gauge
status

   None

authentication_token_cache_fetch_total	 ALPHA	 Counter
status

   None

authentication_token_cache_request_duration_seconds	 ALPHA	 Histogram
status

   None

authentication_token_cache_request_total	 ALPHA	 Counter
status

   None

cloudprovider_aws_api_request_duration_seconds	 ALPHA	 Histogram	 Latency of AWS API calls
request

   None

cloudprovider_aws_api_request_errors	 ALPHA	 Counter	 AWS API errors
request

   None

cloudprovider_aws_api_throttled_requests_total	 ALPHA	 Counter	 AWS API throttled requests
operation_name

   None

cloudprovider_azure_api_request_duration_seconds	 ALPHA	 Histogram	 Latency of an Azure API call
request

resource_group

source

subscription_id

   None

cloudprovider_azure_api_request_errors	 ALPHA	 Counter	 Number of errors for an Azure API call
request

resource_group

source

subscription_id

   None

cloudprovider_azure_api_request_ratelimited_count	 ALPHA	 Counter	 Number of rate limited Azure API calls
request

resource_group

source

subscription_id

   None

cloudprovider_azure_api_request_throttled_count	 ALPHA	 Counter	 Number of throttled Azure API calls
request

resource_group

source

subscription_id

   None

cloudprovider_azure_op_duration_seconds	 ALPHA	 Histogram	 Latency of an Azure service operation
request

resource_group

source

subscription_id

   None

cloudprovider_azure_op_failure_count	 ALPHA	 Counter	 Number of failed Azure service operations
request

resource_group

source

subscription_id

   None

cloudprovider_gce_api_request_duration_seconds	 ALPHA	 Histogram	 Latency of a GCE API call
region

request

version

zone

   None

cloudprovider_gce_api_request_errors	 ALPHA	 Counter	 Number of errors for an API call
region

request

version

zone

   None

cloudprovider_vsphere_api_request_duration_seconds	 ALPHA	 Histogram	 Latency of vsphere api call
request

   None

cloudprovider_vsphere_api_request_errors	 ALPHA	 Counter	 vsphere Api errors
request

   None

cloudprovider_vsphere_operation_duration_seconds	 ALPHA	 Histogram	 Latency of vsphere operation call
operation

   None

cloudprovider_vsphere_operation_errors	 ALPHA	 Counter	 vsphere operation errors
operation

   None

cloudprovider_vsphere_vcenter_versions	 ALPHA	 Custom	 Versions for connected vSphere vCenters
hostname

version

build

   None

container_cpu_usage_seconds_total	 ALPHA	 Custom	 Cumulative cpu time consumed by the container in core-seconds
container

pod

namespace

   None

container_memory_working_set_bytes	 ALPHA	 Custom	 Current working set of the container in bytes
container

pod

namespace

   None

container_start_time_seconds	 ALPHA	 Custom	 Start time of the container since unix epoch in seconds
container

pod

namespace

   None

cronjob_controller_cronjob_job_creation_skew_duration_seconds	 ALPHA	 Histogram	 Time between when a cronjob is scheduled to be run, and when the corresponding job is created	 None	 None

csi_operations_seconds	 ALPHA	 Histogram	 Container Storage Interface operation duration with gRPC error code status total
driver_name

grpc_status_code

method_name

migrated

   None

endpoint_slice_controller_changes	 ALPHA	 Counter	 Number of EndpointSlice changes
operation

   None

endpoint_slice_controller_desired_endpoint_slices	 ALPHA	 Gauge	 Number of EndpointSlices that would exist with perfect endpoint allocation
   None	 None

endpoint_slice_controller_endpoints_added_per_sync	 ALPHA	 Histogram	 Number of endpoints added on each Service sync
   None	 None

endpoint_slice_controller_endpoints_desired	 ALPHA	 Gauge	 Number of endpoints desired
   None	 None

endpoint_slice_controller_endpoints_removed_per_sync	 ALPHA	 Histogram	 Number of endpoints removed on each Service sync
   None	 None

endpoint_slice_controller_endpointslices_changed_per_sync	 ALPHA	 Histogram	 Number of EndpointSlices changed on each Service sync
topology

   None

endpoint_slice_controller_num_endpoint_slices	 ALPHA	 Gauge	 Number of EndpointSlices
   None	 None

endpoint_slice_controller_syncs	 ALPHA	 Counter	 Number of EndpointSlice syncs
result

   None

endpoint_slice_mirroring_controller_addresses_skipped_per_sync	 ALPHA	 Histogram	 Number of addresses skipped on each Endpoints sync due to being invalid or exceeding MaxEndpointsPerSubset
   None	 None

endpoint_slice_mirroring_controller_changes	 ALPHA	 Counter	 Number of EndpointSlice changes
operation

   None

endpoint_slice_mirroring_controller_desired_endpoint_slices	 ALPHA	 Gauge	 Number of EndpointSlices that would exist with perfect endpoint allocation
   None	 None

endpoint_slice_mirroring_controller_endpoints_added_per_sync	 ALPHA	 Histogram	 Number of endpoints added on each Endpoints sync
   None	 None

endpoint_slice_mirroring_controller_endpoints_desired	 ALPHA	 Gauge	 Number of endpoints desired
   None	 None

endpoint_slice_mirroring_controller_endpoints_removed_per_sync	 ALPHA	 Histogram	 Number of endpoints removed on each Endpoints sync
   None	 None

endpoint_slice_mirroring_controller_endpoints_sync_duration	 ALPHA	 Histogram	 Duration of syncEndpoints() in seconds
   None	 None

endpoint_slice_mirroring_controller_endpoints_updated_per_sync	 ALPHA	 Histogram	 Number of endpoints updated on each Endpoints sync
   None	 None

endpoint_slice_mirroring_controller_num_endpoint_slices	 ALPHA	 Gauge	 Number of EndpointSlices
   None	 None

ephemeral_volume_controller_create_failures_total	 ALPHA	 Counter	 Number of PersistenVolumeClaims creation requests
   None	 None

ephemeral_volume_controller_create_total	 ALPHA	 Counter	 Number of PersistenVolumeClaims creation requests
   None	 None

etcd_bookmark_counts	 ALPHA	 Gauge	 Number of etcd bookmarks (progress notify events) split by kind.
resource

   None

etcd_lease_object_counts	 ALPHA	 Histogram	 Number of objects attached to a single etcd lease.
   None	 None

etcd_request_duration_seconds	 ALPHA	 Histogram	 Etcd request latency in seconds for each operation and object type.
operation

type

   None

etcd_version_info	 ALPHA	 Gauge	 Etcd server's binary version
binary_version

   None

field_validation_request_duration_seconds	 ALPHA	 Histogram	 Response latency distribution in seconds for each field validation value and whether field validation is enabled or not
enabled

field_validation

   None

garbagecollector_controller_resources_sync_error_total	 ALPHA	 Counter	 Number of garbage collector resources sync errors
   None	 None

get_token_count	 ALPHA	 Counter	 Counter of total Token() requests to the alternate token source
   None	 None

get_token_fail_count	 ALPHA	 Counter	 Counter of failed Token() requests to the alternate token source
   None	 None

job_controller_job_finished_total	 ALPHA	 Counter	 The number of finished job
completion_mode

reason

result
	 None

job_controller_job_pods_finished_total	 ALPHA	 Counter	 The number of finished Pods that are fully tracked
completion_mode

result
	 None

job_controller_job_sync_duration_seconds	 ALPHA	 Histogram	 The time it took to sync a job
action

completion_mode

result
	 None

job_controller_job_sync_total	 ALPHA	 Counter	 The number of job syncs
action

completion_mode

result
	 None

job_controller_pod_failures_handled_by_failure_policy_total	 ALPHA	 Counter	 `The number of failed Pods handled by failure policy with, 			respect to the failure policy action applied based on the matched, 			rule. Possible values of the action label correspond to the, 			possible values for the failure policy rule action, which are:, 			"FailJob", "Ignore" and "Count".`
action

   None

job_controller_terminated_pods_tracking_finalizer_total	 ALPHA	 Counter	 `The number of terminated pods (phase=Failed|Succeeded), that have the finalizer batch.kubernetes.io/job-tracking, The event label can be "add" or "delete".`
event

   None

kube_apiserver_clusterip_allocator_allocated_ips	 ALPHA	 Gauge	 Gauge measuring the number of allocated IPs for Services
cidr

   None

kube_apiserver_clusterip_allocator_allocation_errors_total	 ALPHA	 Counter	 Number of errors trying to allocate Cluster IPs
cidr

scope

   None

kube_apiserver_clusterip_allocator_allocation_total	 ALPHA	 Counter	 Number of Cluster IPs allocations
cidr

scope

   None

kube_apiserver_clusterip_allocator_available_ips	 ALPHA	 Gauge	 Gauge measuring the number of available IPs for Services
cidr

   None

kube_apiserver_pod_logs_pods_logs_backend_tls_failure_total	 ALPHA	 Counter	 Total number of requests for pods/logs that failed due to kubelet server TLS verification
   None	 None

kube_apiserver_pod_logs_pods_logs_insecure_backend_total	 ALPHA	 Counter	 Total number of requests for pods/logs sliced by usage type: enforce_tls, skip_tls_allowed, skip_tls_denied
usage

   None

kube_pod_resource_limit	 ALPHA	 Custom	 Resources limit for workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any.
namespace

pod

node

scheduler

priority

resource

unit

   None

kube_pod_resource_request	 ALPHA	 Custom	 Resources requested by workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any.
namespace

pod

node

scheduler

priority

resource

unit

   None

kubelet_certificate_manager_client_expiration_renew_errors	 ALPHA	 Counter	 Counter of certificate renewal errors.
   None	 None

kubelet_certificate_manager_client_ttl_seconds	 ALPHA	 Gauge	 Gauge of the TTL (time-to-live) of the Kubelet's client certificate. The value is in seconds until certificate expiry (negative if already expired). If client certificate is invalid or unused, the value will be +INF.
   None	 None

kubelet_certificate_manager_server_rotation_seconds	 ALPHA	 Histogram	 Histogram of the number of seconds the previous certificate lived before being rotated.
   None	 None

kubelet_certificate_manager_server_ttl_seconds	 ALPHA	 Gauge	 Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate. The value is in seconds until certificate expiry (negative if already expired). If serving certificate is invalid or unused, the value will be +INF.
   None	 None

kubelet_cgroup_manager_duration_seconds	 ALPHA	 Histogram	 Duration in seconds for cgroup manager operations. Broken down by method.
operation_type

   None

kubelet_container_log_filesystem_used_bytes	 ALPHA	 Custom	 Bytes used by the container's logs on the filesystem.
uid

namespace

pod

container

   None

kubelet_containers_per_pod_count	 ALPHA	 Histogram	 The number of containers per pod.
   None	 None

kubelet_cpu_manager_pinning_errors_total	 ALPHA	 Counter	 The number of cpu core allocations which required pinning failed.
   None	 None

kubelet_cpu_manager_pinning_requests_total	 ALPHA	 Counter	 The number of cpu core allocations which required pinning.

kubelet_credential_provider_plugin_duration	 ALPHA	 Histogram	 Duration of execution in seconds for credential provider plugin
plugin_name

kubelet_credential_provider_plugin_errors	 ALPHA	 Counter	 Number of errors from credential provider plugin
plugin_name

   None	 None

kubelet_device_plugin_alloc_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to serve a device plugin Allocation request. Broken down by resource name.
resource_name

   None

kubelet_device_plugin_registration_total	 ALPHA	 Counter	 Cumulative number of device plugin registrations. Broken down by resource name.
resource_name

   None

kubelet_eviction_stats_age_seconds	 ALPHA	 Histogram	 Time between when stats are collected, and when pod is evicted based on those stats by eviction signal
eviction_signal

   None

kubelet_evictions	 ALPHA	 Counter	 Cumulative number of pod evictions by eviction signal
eviction_signal

   None

kubelet_graceful_shutdown_end_time_seconds	 ALPHA	 Gauge	 Last graceful shutdown start time since unix epoch in seconds
   None	 None

kubelet_graceful_shutdown_start_time_seconds	 ALPHA	 Gauge	 Last graceful shutdown start time since unix epoch in seconds
   None	 None

kubelet_http_inflight_requests	 ALPHA	 Gauge	 Number of the inflight http requests
long_running

method

path

server_type

   None

kubelet_http_requests_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to serve http requests
long_running

method

path

server_type

   None

kubelet_http_requests_total	 ALPHA	 Counter	 Number of the http requests received since the server started
long_running

method

path

server_type

   None

kubelet_kubelet_credential_provider_plugin_duration	 ALPHA	 Histogram	 Duration of execution in seconds for credential provider plugin
plugin_name
	 None

kubelet_kubelet_credential_provider_plugin_errors	 ALPHA	 Counter	 Number of errors from credential provider plugin
plugin_name
	 None

kubelet_lifecycle_handler_http_fallbacks_total	 ALPHA	 Counter	 The number of times lifecycle handlers successfully fell back to http from https.
   None	 None

kubelet_managed_ephemeral_containers	 ALPHA	 Gauge	 Current number of ephemeral containers in pods managed by this kubelet. Ephemeral containers will be ignored if disabled by the EphemeralContainers feature gate, and this number will be 0.
   None	 None

kubelet_node_name	 ALPHA	 Gauge	 The node's name. The count is always 1.
node

   None

kubelet_pleg_discard_events	 ALPHA	 Counter	 The number of discard events in PLEG.
   None	 None

kubelet_pleg_last_seen_seconds	 ALPHA	 Gauge	 Timestamp in seconds when PLEG was last seen active.
   None	 None

kubelet_pleg_relist_duration_seconds	 ALPHA	 Histogram	 Duration in seconds for relisting pods in PLEG.
   None	 None

kubelet_pleg_relist_interval_seconds	 ALPHA	 Histogram	 Interval in seconds between relisting in PLEG.
   None	 None

kubelet_pod_resources_endpoint_errors_get_allocatable	 ALPHA	 Counter	 Number of requests to the PodResource GetAllocatableResources endpoint which returned error. Broken down by server api version.
server_api_version

   None

kubelet_pod_resources_endpoint_errors_list	 ALPHA	 Counter	 Number of requests to the PodResource List endpoint which returned error. Broken down by server api version.
server_api_version

   None

kubelet_pod_resources_endpoint_requests_get_allocatable	 ALPHA	 Counter	 Number of requests to the PodResource GetAllocatableResources endpoint. Broken down by server api version.
server_api_version

   None

kubelet_pod_resources_endpoint_requests_list	 ALPHA	 Counter	 Number of requests to the PodResource List endpoint. Broken down by server api version.
server_api_version

   None

kubelet_pod_resources_endpoint_requests_total	 ALPHA	 Counter	 Cumulative number of requests to the PodResource endpoint. Broken down by server api version.
server_api_version

   None

kubelet_pod_start_duration_seconds	 ALPHA	 Histogram	 Duration in seconds from kubelet seeing a pod for the first time to the pod starting to run

kubelet_pod_start_sli_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to start a pod, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch
   None	 None

kubelet_pod_status_sync_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to sync a pod status update. Measures time from detection of a change to pod status until the API is successfully updated for that pod, even if multiple intevening changes to pod status occur.
   None	 None

kubelet_pod_worker_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to sync a single pod. Broken down by operation type: create, update, or sync
operation_type

   None

kubelet_pod_worker_start_duration_seconds	 ALPHA	 Histogram	 Duration in seconds from kubelet seeing a pod to starting a worker.
   None	 None

kubelet_preemptions	 ALPHA	 Counter	 Cumulative number of pod preemptions by preemption resource
preemption_signal

   None

kubelet_run_podsandbox_duration_seconds	 ALPHA	 Histogram	 Duration in seconds of the run_podsandbox operations. Broken down by RuntimeClass.Handler.
runtime_handler

   None

kubelet_run_podsandbox_errors_total	 ALPHA	 Counter	 Cumulative number of the run_podsandbox operation errors by RuntimeClass.Handler.
runtime_handler

   None

kubelet_running_containers	 ALPHA	 Gauge	 Number of containers currently running
container_state

   None

kubelet_running_pods	 ALPHA	 Gauge	 Number of pods that have a running pod sandbox
   None	 None

kubelet_runtime_operations_duration_seconds	 ALPHA	 Histogram	 Duration in seconds of runtime operations. Broken down by operation type.
operation_type

   None

kubelet_runtime_operations_errors_total	 ALPHA	 Counter	 Cumulative number of runtime operation errors by operation type.
operation_type

   None

kubelet_runtime_operations_total	 ALPHA	 Counter	 Cumulative number of runtime operations by operation type.
operation_type

   None

kubelet_server_expiration_renew_errors	 ALPHA	 Counter	 Counter of certificate renewal errors.
   None	 None

kubelet_started_containers_errors_total	 ALPHA	 Counter	 Cumulative number of errors when starting containers
code

container_type

   None

kubelet_started_containers_total	 ALPHA	 Counter	 Cumulative number of containers started
container_type

   None

kubelet_started_host_process_containers_errors_total	 ALPHA	 Counter	 Cumulative number of errors when starting hostprocess containers. This metric will only be collected on Windows and requires WindowsHostProcessContainers feature gate to be enabled.
code

container_type

   None

kubelet_started_host_process_containers_total	 ALPHA	 Counter	 Cumulative number of hostprocess containers started. This metric will only be collected on Windows and requires WindowsHostProcessContainers feature gate to be enabled.
container_type

   None

kubelet_started_pods_errors_total	 ALPHA	 Counter	 Cumulative number of errors when starting pods
   None	 None

kubelet_started_pods_total	 ALPHA	 Counter	 Cumulative number of pods started
   None	 None

kubelet_volume_metric_collection_duration_seconds	 ALPHA	 Histogram	 Duration in seconds to calculate volume stats
metric_source

   None

kubelet_volume_stats_available_bytes	 ALPHA	 Custom	 Number of available bytes in the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_capacity_bytes	 ALPHA	 Custom	 Capacity in bytes of the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_health_status_abnormal	 ALPHA	 Custom	 Abnormal volume health status. The count is either 1 or 0. 1 indicates the volume is unhealthy, 0 indicates volume is healthy
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_inodes	 ALPHA	 Custom	 Maximum number of inodes in the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_inodes_free	 ALPHA	 Custom	 Number of free inodes in the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_inodes_used	 ALPHA	 Custom	 Number of used inodes in the volume
namespace

persistentvolumeclaim

   None

kubelet_volume_stats_used_bytes	 ALPHA	 Custom	 Number of used bytes in the volume
namespace

persistentvolumeclaim

   None

kubeproxy_network_programming_duration_seconds	 ALPHA	 Histogram	 In Cluster Network Programming Latency in seconds
   None	 None

kubeproxy_sync_proxy_rules_duration_seconds	 ALPHA	 Histogram	 SyncProxyRules latency in seconds
   None	 None

kubeproxy_sync_proxy_rules_endpoint_changes_pending	 ALPHA	 Gauge	 Pending proxy rules Endpoint changes
   None	 None

kubeproxy_sync_proxy_rules_endpoint_changes_total	 ALPHA	 Counter	 Cumulative proxy rules Endpoint changes

kubeproxy_sync_proxy_rules_iptables_partial_restore_failures_total	 ALPHA	 Counter	 Cumulative proxy iptables partial restore failures
   None	 None

kubeproxy_sync_proxy_rules_iptables_restore_failures_total	 ALPHA	 Counter	 Cumulative proxy iptables restore failures
   None	 None

kubeproxy_sync_proxy_rules_iptables_total	 ALPHA	 Gauge	 Number of proxy iptables rules programmed
table

   None

kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds	 ALPHA	 Gauge	 The last time a sync of proxy rules was queued
   None	 None

kubeproxy_sync_proxy_rules_last_timestamp_seconds	 ALPHA	 Gauge	 The last time proxy rules were successfully synced
   None	 None

kubeproxy_sync_proxy_rules_no_local_endpoints_total	 ALPHA	 Gauge	 Number of services with a Local traffic policy and no endpoints
traffic_policy

   None

kubeproxy_sync_proxy_rules_service_changes_pending	 ALPHA	 Gauge	 Pending proxy rules Service changes
   None	 None

kubeproxy_sync_proxy_rules_service_changes_total	 ALPHA	 Counter	 Cumulative proxy rules Service changes
   None	 None

kubernetes_build_info	 ALPHA	 Gauge	 A metric with a constant '1' value labeled by major, minor, git version, git commit, git tree state, build date, Go version, and compiler from which Kubernetes was built, and platform on which it is running.
build_date

compiler

git_commit

git_tree_state

git_version

go_version

major

minor

platform

   None

kubernetes_feature_enabled	 ALPHA	 Gauge	 This metric records the data about the stage and enablement of a k8s feature.
name

stage

   None

kubernetes_healthcheck	 ALPHA	 Gauge	 This metric records the result of a single healthcheck.
name

type

   None

kubernetes_healthchecks_total	 ALPHA	 Counter	 This metric records the results of all healthcheck.
name

status

type

   None

leader_election_master_status	 ALPHA	 Gauge	 Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
name

   None

node_authorizer_graph_actions_duration_seconds	 ALPHA	 Histogram	 Histogram of duration of graph actions in node authorizer.
operation

   None

node_collector_evictions_number	 ALPHA	 Counter	 Number of Node evictions that happened since current instance of NodeController started, This metric is replaced by node_collector_evictions_total.
zone
	  	 1.24.0
   None

node_collector_unhealthy_nodes_in_zone	 ALPHA	 Gauge	 Gauge measuring number of not Ready Nodes per zones.
zone

   None

node_collector_zone_health	 ALPHA	 Gauge	 Gauge measuring percentage of healthy nodes per zone.
zone

   None

node_collector_zone_size	 ALPHA	 Gauge	 Gauge measuring number of registered Nodes per zones.
zone

   None

node_cpu_usage_seconds_total	 ALPHA	 Custom	 Cumulative cpu time consumed by the node in core-seconds
   None	 None

node_ipam_controller_cidrset_allocation_tries_per_request	 ALPHA	 Histogram	 Number of endpoints added on each Service sync
clusterCIDR

   None

node_ipam_controller_cidrset_cidrs_allocations_total	 ALPHA	 Counter	 Counter measuring total number of CIDR allocations.
clusterCIDR

   None

node_ipam_controller_cidrset_cidrs_releases_total	 ALPHA	 Counter	 Counter measuring total number of CIDR releases.
clusterCIDR

   None

node_ipam_controller_cidrset_usage_cidrs	 ALPHA	 Gauge	 Gauge measuring percentage of allocated CIDRs.
clusterCIDR

   None

node_ipam_controller_multicidrset_allocation_tries_per_request	 ALPHA	 Histogram	 Histogram measuring CIDR allocation tries per request.
clusterCIDR

   None

node_ipam_controller_multicidrset_cidrs_allocations_total	 ALPHA	 Counter	 Counter measuring total number of CIDR allocations.
clusterCIDR

   None

node_ipam_controller_multicidrset_cidrs_releases_total	 ALPHA	 Counter	 Counter measuring total number of CIDR releases.
clusterCIDR

   None

node_ipam_controller_multicidrset_usage_cidrs	 ALPHA	 Gauge	 Gauge measuring percentage of allocated CIDRs.
clusterCIDR

   None

node_memory_working_set_bytes	 ALPHA	 Custom	 Current working set of the node in bytes
   None	 None

number_of_l4_ilbs	 ALPHA	 Gauge	 Number of L4 ILBs
feature

   None

plugin_manager_total_plugins	 ALPHA	 Custom	 Number of plugins in Plugin Manager
socket_path

state

   None

pod_cpu_usage_seconds_total	 ALPHA	 Custom	 Cumulative cpu time consumed by the pod in core-seconds
pod

namespace

pod_gc_collector_force_delete_pod_errors_total	 ALPHA	 Counter	 Number of errors encountered when forcefully deleting the pods since the Pod GC Controller started.

pod_gc_collector_force_delete_pods_total	 ALPHA	 Counter	 Number of pods that are being forcefully deleted since the Pod GC Controller started.
   None

pod_memory_working_set_bytes	 ALPHA	 Custom	 Current working set of the pod in bytes
pod

namespace

   None

pod_security_errors_total	 ALPHA	 Counter	 Number of errors preventing normal evaluation. Non-fatal errors may result in the latest restricted profile being used for evaluation.
fatal

request_operation

resource

subresource

   None

pod_security_evaluations_total	 ALPHA	 Counter	 Number of policy evaluations that occurred, not counting ignored or exempt requests.
decision

mode

policy_level

policy_version

request_operation

resource

subresource

   None

pod_security_exemptions_total	 ALPHA	 Counter	 Number of exempt requests, not counting ignored or out of scope requests.
request_operation

resource

subresource

   None

prober_probe_duration_seconds	 ALPHA	 Histogram	 Duration in seconds for a probe response.
container

namespace

pod

probe_type

   None

prober_probe_total	 ALPHA	 Counter	 Cumulative number of a liveness, readiness or startup probe for a container by result.
container

namespace

pod

pod_uid

probe_type

result

   None

pv_collector_bound_pv_count	 ALPHA	 Custom	 Gauge measuring number of persistent volume currently bound
storage_class

   None

pv_collector_bound_pvc_count	 ALPHA	 Custom	 Gauge measuring number of persistent volume claim currently bound
namespace

   None

pv_collector_total_pv_count	 ALPHA	 Custom	 Gauge measuring total number of persistent volumes
plugin_name

volume_mode

   None

pv_collector_unbound_pv_count	 ALPHA	 Custom	 Gauge measuring number of persistent volume currently unbound
storage_class

   None

pv_collector_unbound_pvc_count	 ALPHA	 Custom	 Gauge measuring number of persistent volume claim currently unbound
namespace

   None

replicaset_controller_sorting_deletion_age_ratio	 ALPHA	 Histogram	 The ratio of chosen deleted pod's ages to the current youngest pod's age (at the time). Should be <2.The intent of this metric is to measure the rough efficacy of the LogarithmicScaleDown feature gate's effect onthe sorting (and deletion) of pods when a replicaset scales down. This only considers Ready pods when calculating and reporting.
   None	 None

rest_client_exec_plugin_call_total	 ALPHA	 Counter	 Number of calls to an exec plugin, partitioned by the type of event encountered (no_error, plugin_execution_error, plugin_not_found_error, client_internal_error) and an optional exit code. The exit code will be set to 0 if and only if the plugin call was successful.
call_status

code

   None

rest_client_exec_plugin_certificate_rotation_age	 ALPHA	 Histogram	 Histogram of the number of seconds the last auth exec plugin client certificate lived before being rotated. If auth exec plugin client certificates are unused, histogram will contain no data.
   None	 None

rest_client_exec_plugin_ttl_seconds	 ALPHA	 Gauge	 Gauge of the shortest TTL (time-to-live) of the client certificate(s) managed by the auth exec plugin. The value is in seconds until certificate expiry (negative if already expired). If auth exec plugins are unused or manage no TLS certificates, the value will be +INF.
   None	 None

rest_client_rate_limiter_duration_seconds	 ALPHA	 Histogram	 Client side rate limiter latency in seconds. Broken down by verb, and host.
host

verb

   None

rest_client_request_duration_seconds	 ALPHA	 Histogram	 Request latency in seconds. Broken down by verb, and host.
host

verb

   None

rest_client_request_size_bytes	 ALPHA	 Histogram	 Request size in bytes. Broken down by verb and host.
host

verb

   None

rest_client_requests_total	 ALPHA	 Counter	 Number of HTTP requests, partitioned by status code, method, and host.
code

host

method

   None

rest_client_response_size_bytes	 ALPHA	 Histogram	 Response size in bytes. Broken down by verb and host.
host

verb

   None

retroactive_storageclass_errors_total	 ALPHA	 Counter	 Total number of failed retroactive StorageClass assignments to persistent volume claim
   None	 None

retroactive_storageclass_total	 ALPHA	 Counter	 Total number of retroactive StorageClass assignments to persistent volume claim
   None	 None

root_ca_cert_publisher_sync_duration_seconds	 ALPHA	 Histogram	 Number of namespace syncs happened in root ca cert publisher.
code

   None

root_ca_cert_publisher_sync_total	 ALPHA	 Counter	 Number of namespace syncs happened in root ca cert publisher.
code

   None

running_managed_controllers	 ALPHA	 Gauge	 Indicates where instances of a controller are currently running
manager

name

   None

scheduler_e2e_scheduling_duration_seconds	 ALPHA	 Histogram	 E2e scheduling latency in seconds (scheduling algorithm + binding). This metric is replaced by scheduling_attempt_duration_seconds.
profile

result
	  	 1.23.0
   None

scheduler_goroutines	 ALPHA	 Gauge	 Number of running goroutines split by the work they do such as binding.
operation

   None

scheduler_permit_wait_duration_seconds	 ALPHA	 Histogram	 Duration of waiting on permit.
result

   None

scheduler_plugin_execution_duration_seconds	 ALPHA	 Histogram	 Duration for running a plugin at a specific extension point.
extension_point

plugin

status

   None

scheduler_scheduler_cache_size	 ALPHA	 Gauge	 Number of nodes, pods, and assumed (bound) pods in the scheduler cache.
type

   None

scheduler_scheduler_goroutines	 ALPHA	 Gauge	 Number of running goroutines split by the work they do such as binding. This metric is replaced by the "goroutines" metric.
work
	  	 1.26.0
   None

scheduler_scheduling_algorithm_duration_seconds	 ALPHA	 Histogram	 Scheduling algorithm latency in seconds
   None	 None

scheduler_unschedulable_pods	 ALPHA	 Gauge	 The number of unschedulable pods broken down by plugin name. A pod will increment the gauge for all plugins that caused it to not schedule and so this metric have meaning only when broken down by plugin.
plugin

profile

   None

scheduler_volume_binder_cache_requests_total	 ALPHA	 Counter	 Total number for request volume binding cache
operation

   None

scheduler_volume_scheduling_stage_error_total	 ALPHA	 Counter	 Volume scheduling stage error count
operation

   None

scrape_error	 ALPHA	 Custom	 1 if there was an error while getting container metrics, 0 otherwise
   None	 None

service_controller_nodesync_latency_seconds	 ALPHA	 Histogram	 A metric measuring the latency for nodesync which updates loadbalancer hosts on cluster node updates.
   None	 None

service_controller_update_loadbalancer_host_latency_seconds	 ALPHA	 Histogram	 A metric measuring the latency for updating each load balancer hosts.
   None	 None

serviceaccount_legacy_tokens_total	 ALPHA	 Counter	 Cumulative legacy service account tokens used
   None	 None

serviceaccount_stale_tokens_total	 ALPHA	 Counter	 Cumulative stale projected service account tokens used
   None	 None

serviceaccount_valid_tokens_total	 ALPHA	 Counter	 Cumulative valid projected service account tokens used
   None	 None

storage_count_attachable_volumes_in_use	 ALPHA	 Custom	 Measure number of volumes in use
node

volume_plugin

   None

storage_operation_duration_seconds	 ALPHA	 Histogram	 Storage operation duration
migrated

operation_name

status

volume_plugin

   None

ttl_after_finished_controller_job_deletion_duration_seconds	 ALPHA	 Histogram	 The time it took to delete the job since it became eligible for deletion
   None	 None

volume_manager_selinux_container_errors_total	 ALPHA	 Gauge	 Number of errors when kubelet cannot compute SELinux context for a container. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of containers.
   None	 None

volume_manager_selinux_container_warnings_total	 ALPHA	 Gauge	 Number of errors when kubelet cannot compute SELinux context for a container that are ignored. They will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
   None	 None

volume_manager_selinux_pod_context_mismatch_errors_total	 ALPHA	 Gauge	 Number of errors when a Pod defines different SELinux contexts for its containers that use the same volume. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of Pods.
   None	 None

volume_manager_selinux_pod_context_mismatch_warnings_total	 ALPHA	 Gauge	 Number of errors when a Pod defines different SELinux contexts for its containers that use the same volume. They are not errors yet, but they will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
   None	 None

volume_manager_selinux_volume_context_mismatch_errors_total	 ALPHA	 Gauge	 Number of errors when a Pod uses a volume that is already mounted with a different SELinux context than the Pod needs. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of Pods.
   None	 None

volume_manager_selinux_volume_context_mismatch_warnings_total	 ALPHA	 Gauge	 Number of errors when a Pod uses a volume that is already mounted with a different SELinux context than the Pod needs. They are not errors yet, but they will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
   None	 None

volume_manager_selinux_volumes_admitted_total	 ALPHA	 Gauge	 Number of volumes whose SELinux context was fine and will be mounted with mount -o context option.
   None	 None

volume_manager_total_volumes	 ALPHA	 Custom	 Number of volumes in Volume Manager
plugin_name

state

   None

volume_operation_total_errors	 ALPHA	 Counter	 Total volume operation errors
operation_name

plugin_name

   None

volume_operation_total_seconds	 ALPHA	 Histogram	 Storage operation end to end duration in seconds
operation_name

plugin_name

   None

watch_cache_capacity	 ALPHA	 Gauge	 Total capacity of watch cache broken by resource type.
resource

   None

watch_cache_capacity_decrease_total	 ALPHA	 Counter	 Total number of watch cache capacity decrease events broken by resource type.
resource

   None

watch_cache_capacity_increase_total	 ALPHA	 Counter	 Total number of watch cache capacity increase events broken by resource type.
resource

   None

workqueue_adds_total	 ALPHA	 Counter	 Total number of adds handled by workqueue
name

   None

workqueue_depth	 ALPHA	 Gauge	 Current depth of workqueue
name

   None

workqueue_longest_running_processor_seconds	 ALPHA	 Gauge	 How many seconds has the longest running processor for workqueue been running.
name

   None

workqueue_queue_duration_seconds	 ALPHA	 Histogram	 How long in seconds an item stays in workqueue before being requested.
name

   None

workqueue_retries_total	 ALPHA	 Counter	 Total number of retries handled by workqueue
name

   None

workqueue_unfinished_work_seconds	 ALPHA	 Gauge	 How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
name

   None

workqueue_work_duration_seconds	 ALPHA	 Histogram	 How long in seconds processing an item from workqueue takes.
name

   None

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: documents metrics for Kubernetes v1.26 but that's incorrect because at the time I'm filing this the live documentation is for Kubernetes v1.25 Proposed Solution: Revert whichever PR bumped the document too early or do a careful partial revert that only omits the v1.26 changes Page to Update: <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--Additional Information: /sig instrumentation /language en If v1.26 gets released before we fix this, it's OK to close the issue. Consider a backport to the v1.25 docs, though.
/assign
/triage accepted
/assign


Question: ## Usando Labels - Defina e use [labels](/docs/concepts/overview/working-with-objects/labels/) que identifiquem _atributos semânticos_ da sua aplicação ou Deployment, como `{ app: myapp, tier: frontend, phase: test, deployment: v3 }`. Você pode usar essas labels para selecionar os Pods apropriados para outros recursos; por exemplo, um Service que seleciona todos os Pods `tier: frontend`, ou todos <del> os componentes de `app: myapp`. Veja o app [guestbook](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/guestbook/) para exemplos dessa abordagem. </del> <ins> os componentes de `app: myapp`. Veja o app [guestbook](https://github.com/kubernetes/examples/tree/master/guestbook/) para exemplos dessa abordagem. </ins> Um Service pode ser feito para abranger vários Deployments, omitindo labels específicas de lançamento de seu seletor. Quando você precisar atualizar um serviço em execução sem _downtime_, use um [Deployment](/docs/concepts/workloads/controllers/deployment/).

This is a Bug Report Problem: Regarding , I found the similar broken links on other pages. English pages have already been fixed. ref: pt pages also need to be fixed. Proposed Solution: Fixes broken links including {{< param "githubbranch" }} to master. In the case of link to , fixes to {{< param "fullversion" }}. ref:
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: {{< note >}} If the `SizeMemoryBackedVolumes` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled, you can specify a size for memory backed volumes.  If no size is specified, memory <del> backed volumes are sized to 50% of the memory on a Linux host. </del> <ins> backed volumes are sized to node allocatable memory. </ins> {{< /note>}} #### emptyDir configuration example

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: The description about emptyDir sizeLimit in page mentioned that: But the following code shows that if no size is specified, size limit defaults to node allocatable memory. Here is the information about my local environment and the result: Proposed Solution: Change this description to the following: Page to Update: <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--<!--Additional Information:--/assign /language en /sig storage
Could I ask that why it says that if no size is specified, memory backed volumes are sized to 50% of the memory on a Linux host in the document? Why would memory backed volumes be sized 50% of the memory on a Linux host? Is it because the tmpfs default size=50% of physical RAM?
/remove-sig storage /sig node
/triage accepted
The memory-backed EmptyDir by default consumes a very large amount of memory (documented as , but actually ). I think it's too large for the default. In fact, one pod can occupy most of a node's memory. Is there any way to limits the size of memory-backed EmptyDir by cluster admin?
I'm not aware of any existing mechanisms to automate the setting of for volumes. You may have to devise your own mutating webhook or wait for the mutating admission policy to land. -- my 2 cents.
Thank you for your advice! But it's problem that the whole of cluster can be down easily by few pods and the cluster admin can not control this, IMHO. I have a few idea for this problem: Set the default size of memory-backed EmptyDir fewer. e.g. 50Mi. Enable to be controlled the total size of memory-backed EmptyDir Add taint to node for that memory-backed EmptyDir can use WDYT?
I'd like to discuss about this in . Thanks!!


Question: node-3       Ready    v1.19.6   containerd://1.4.1 ``` <del> コンテナランタイムについては、[コンテナランタイム](/docs/setup/production-environment/container-runtimes/)のページで詳細を確認することができます。 </del> <ins> コンテナランタイムについては、[コンテナランタイム](/ja/docs/setup/production-environment/container-runtimes/)のページで詳細を確認することができます。 </ins>

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be <!-- Describe as precisely as possible how this feature/enhancement should work from the user perspective. What should be changed, etc. --Update Japanese translation Why is this needed Comments <!-- Any additional related comments that might help. Drawings/mockups would be extremely helpful (if required). --ref. /language ja
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I want to work on this. Have you already started working on it? It seems that some pages linked from here are not translated yet: dockershim-removal template For now, I'm going to start working on dockershim related contents. Feel free to notify me if this conflicts with your work.
I'll assign myself since you haven't assigned youself to this issue. Please tell me if this conflicts with you work. /assign


Question: # Last commit for the localized file LASTCOMMIT=`git log -n 1 --pretty=format:%h -- $LOCALIZED` <del> git diff $LASTCOMMIT...HEAD $EN_VERSION </del> <ins> git diff --exit-code $LASTCOMMIT...HEAD $EN_VERSION </ins> if [ "$?" -eq 0 ]; then echo "$LOCALIZED is still in sync"

Problem: Wasn't sure how to report this so selected bug as it seemed closest to what to report Running from the directory should return the diff and then close without output. But it shows the diff then outputs that the page is still in sync. Output Proposed Solution: Remove output from showing after the diff is shown when a page is not in sync. Page to Update: N/A Thanks for the script (for whoever wrote this). :]
/triage accepted /assign


Question: ## {{% heading "whatsnext" %}} - [Secretのコンセプト](/ja/docs/concepts/configuration/secret/)を読む <del> - [設定ファイルを使用してSecretを管理する](/docs/tasks/configmap-secret/managing-secret-using-config-file/)方法を知る </del> <ins> - [設定ファイルを使用してSecretを管理する](/ja/docs/tasks/configmap-secret/managing-secret-using-config-file/)方法を知る </ins> - [kustomizeを使用してSecretを管理する](/docs/tasks/configmap-secret/managing-secret-using-kustomize/)方法を知る

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be <!-- Describe as precisely as possible how this feature/enhancement should work from the user perspective. What should be changed, etc. --Translate into Japanese Why is this needed This page has not been translated. Comments <!-- Any additional related comments that might help. Drawings/mockups would be extremely helpful (if required). --/language ja /assign /triage accepted
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: app: hello tier: backend track: stable <del> replicas: 7 </del> <ins> replicas: 3 </ins> template: metadata: labels:

This is a Feature Request Korean Localization for branch. What would you like to be Any localization contributions for Korean. Comments This issue is to notify the schedule of Korean localization to contributors. is the current work branch for Korean localization. Important dates A Korean localization work branch is merged to branch after we complete around 3~4 weeks job. PR Freeze: 01/16 (PR opened after freeze will be requested to rebase to next work branch) Branch Merge: 01/19 (merge to main branch)  2~3 , main . PR : 01/16 ( PR , PR ) : 01/19 (main , k8s.io ) /language ko /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ## In English is the current work branch for Korean localization.A Korean localization work branch is merged to branch after we complete around 3~4 weeks job. PR Freeze: 01/16 (PR opened after freeze will be requested to rebase to next work branch) Branch Merge: 01/19 (merge to main branch) PR : 01/16 ( PR , PR ) : 01/19 (main , k8s.io ) /help </details
/milestone dev-1.26-ko.1
/triage accepted
Outdated Korean contents in the dev-1.26-ko.1


Question: <del> --- approvers: - bprashanth - enisoc - erictune - foxish - janetkuo - kow3ns - smarterclayton title: Upgrade from PetSets to StatefulSets --- {% capture overview %} This page shows how to upgrade from PetSets (Kubernetes version 1.3 or 1.4) to *StatefulSets* (Kubernetes version 1.5 or later). {% endcapture %} {% capture prerequisites %} * If you don't have PetSets in your current cluster, or you don't plan to upgrade your master to Kubernetes 1.5 or later, you can skip this task. {% endcapture %} {% capture steps %} ## Differences between alpha PetSets and beta StatefulSets PetSet was introduced as an alpha resource in Kubernetes release 1.3, and was renamed to StatefulSet as a beta resource in 1.5. Here are some notable changes: * **StatefulSet is the new PetSet**: PetSet is no longer available in Kubernetes release 1.5 or later. It becomes beta StatefulSet. To understand why the name was changed, see this [discussion thread](https://github.com/kubernetes/kubernetes/issues/27430). * **StatefulSet guards against split brain**: StatefulSets guarantee at most one Pod for a given ordinal index can be running anywhere in a cluster, to guard against split brain scenarios with distributed applications. *TODO: Link to doc about fencing.* * **Flipped debug annotation behavior**: The default value of the debug annotation (`pod.alpha.kubernetes.io/initialized`) is `true` in 1.5 through 1.7. The annotation is completely ignored in 1.8 and above, which always behave as if it were `true`. The absence of this annotation will pause PetSet operations, but will NOT pause StatefulSet operations. In most cases, you no longer need this annotation in your StatefulSet manifests. ## Upgrading from PetSets to StatefulSets Note that these steps need to be done in the specified order. You **should NOT upgrade your Kubernetes master, nodes, or `kubectl` to Kubernetes version 1.5 or later**, until told to do so. ### Find all PetSets and their manifests First, find all existing PetSets in your cluster: ```shell kubectl get petsets --all-namespaces ``` If you don't find any existing PetSets, you can safely upgrade your cluster to Kubernetes version 1.5 or later. If you find existing PetSets and you have all their manifests at hand, you can continue to the next step to prepare StatefulSet manifests. Otherwise, you need to save their manifests so that you can recreate them as StatefulSets later. Here's an example command for you to save all existing PetSets as one file. ```shell # Save all existing PetSets in all namespaces into a single file. Only needed when you don't have their manifests at hand. kubectl get petsets --all-namespaces -o yaml > all-petsets.yaml ``` ### Prepare StatefulSet manifests Now, for every PetSet manifest you have, prepare a corresponding StatefulSet manifest: 1. Change `apiVersion` from `apps/v1alpha1` to `apps/v1beta1`. 2. Change `kind` from `PetSet` to `StatefulSet`. 3. If you have the debug hook annotation `pod.alpha.kubernetes.io/initialized` set to `true`, you can remove it because it's redundant. If you don't have this annotation or have it set to `false`, be aware that StatefulSet operations might resume after the upgrade. If you are upgrading to 1.6 or 1.7, you can set the annotation explicitly to `false` to maintain the paused behavior. If you are upgrading to 1.8 or above, there's no longer any debug annotation to pause StatefulSets. It's recommended that you keep both PetSet manifests and StatefulSet manifests, so that you can safely roll back and recreate your PetSets, if you decide not to upgrade your cluster. ### Delete all PetSets without cascading If you find existing PetSets in your cluster in the previous step, you need to delete all PetSets *without cascading*. You can do this from `kubectl` with `--cascade=false`. Note that if the flag isn't set, **cascading deletion will be performed by default**, and all Pods managed by your PetSets will be gone. Delete those PetSets by specifying file names. This only works when the files contain only PetSets, but not other resources such as Services: ```shell # Delete all existing PetSets without cascading # Note that <pet-set-file> should only contain PetSets that you want to delete, but not any other resources kubectl delete -f <pet-set-file> --cascade=false ``` Alternatively, delete them by specifying resource names: ```shell # Alternatively, delete them by name and namespace without cascading kubectl delete petsets <pet-set-name> -n=<pet-set-namespace> --cascade=false ``` Make sure you've deleted all PetSets in the system: ```shell # Get all PetSets again to make sure you deleted them all # This should return nothing kubectl get petsets --all-namespaces ``` At this moment, you've deleted all PetSets in your cluster, but not their Pods, Persistent Volumes, or Persistent Volume Claims. However, since the Pods are not managed by PetSets anymore, they will be vulnerable to node failures until you finish the master upgrade and recreate StatefulSets. ### Upgrade your master to Kubernetes version 1.5 or later Now, you can [upgrade your Kubernetes master](/docs/admin/cluster-management/#upgrading-a-cluster) to Kubernetes version 1.5 or later. Note that **you should NOT upgrade Nodes at this time**, because the Pods (that were once managed by PetSets) are now vulnerable to node failures. ### Upgrade kubectl to Kubernetes version 1.5 or later Upgrade `kubectl` to Kubernetes version 1.5 or later, following [the steps for installing and setting up kubectl](/docs/tasks/kubectl/install/). ### Create StatefulSets Make sure you have both master and `kubectl` upgraded to Kubernetes version 1.5 or later before continuing: ```shell kubectl version ``` The output is similar to this: ```shell Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.0", GitCommit:"0776eab45fe28f02bbeac0f05ae1a203051a21eb", GitTreeState:"clean", BuildDate:"2016-11-24T22:35:03Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.0", GitCommit:"0776eab45fe28f02bbeac0f05ae1a203051a21eb", GitTreeState:"clean", BuildDate:"2016-11-24T22:30:23Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} ``` If both `Client Version` (`kubectl` version) and `Server Version` (master version) are 1.5 or later, you are good to go. Create StatefulSets to adopt the Pods belonging to the deleted PetSets with the StatefulSet manifests generated in the previous step: ```shell kubectl create -f <stateful-set-file> ``` Make sure all StatefulSets are created and running as expected in the newly-upgraded cluster: ```shell kubectl get statefulsets --all-namespaces ``` ### Upgrade nodes to Kubernetes version 1.5 or later (optional) You can now [upgrade Kubernetes nodes](/docs/admin/cluster-management/#upgrading-a-cluster) to Kubernetes version 1.5 or later. This step is optional, but needs to be done after all StatefulSets are created to adopt PetSets' Pods. You should be running Node version >= 1.1.0 to run StatefulSets safely. Older versions do not support features which allow the StatefulSet to guarantee that at any time, there is **at most** one Pod with a given identity running in a cluster. {% endcapture %} {% capture whatsnext %} Learn more about [scaling a StatefulSet](/docs/tasks/manage-stateful-set/scale-stateful-set/). {% endcapture %} {% include templates/task.md %} </del>

This is a... [ ] Feature Request [x] Bug Report Problem: There is a link missing that was supposed to be in section Differences between alpha PetSets and beta StatefulSets under it in the second point StatefulSet guards against split brain which ends with following text TODO: Link to doc about fencing. Change it and add the link that is required. Proposed Solution: Add link which is missing Page to Update:
PetSet has long been replaced by StatefuSet and StatefulSet has been GA'ed in 1.9. So we are going to remove this page to avoid further confusion.


Question: The value is a comma-separated list of the names of namespaces other than the parent's namespace in which objects are found. <del> ### applyset.kubernetes.io/contains-group-resources (alpha) {#applyset-kubernetes-io-contains-group-resources} </del> <ins> ### applyset.kubernetes.io/contains-group-kinds (alpha) {#applyset-kubernetes-io-contains-group-kinds} </ins> Type: Annotation <del> Example: `applyset.kubernetes.io/contains-group-resources: "certificates.cert-manager.io,configmaps,deployments.apps,secrets,services"` </del> <ins> Example: `applyset.kubernetes.io/contains-group-kinds: "certificates.cert-manager.io,configmaps,deployments.apps,secrets,services"` </ins> Used on: Objects being used as ApplySet parents.

I'm not entirely sure if this is a bug, or if the documentation is just inaccurate/lacking some context, but it's at the very least a bit confusing. If you check the documentation for the well-known labels, etc., you can find the documentation for the annotation . ! However, when I try to reference a parent object with the option to without this annotation, it complains about the annotation missing: At this point my parent resource is declared as: It appears that I can set either one of the two ( or ) on the resource and will accept it; only in the latter case it appears to be copying the "resources" annotation to the add the "kinds" annotation. To phrase it another way: Setting the "kinds" annotation seems like the right way to go (but not what the documentation suggests), and setting the "resources" annotation works as well but you'll also end up with the "kinds" annotation afterwards. I expect the documentation and kubectl to align. Point at a resource that doesn't have the required annotations. No response <details</detailsnocloud No response No response No response No response
/sig docs /sig cli
This was changed in the code by and it seems that document needs changes to reflect this. Thank you bringing this issue /triage accepted
I think this should be transferred to the k/website so that the right annotation is recorded. /transfer wbsite
Something went wrong or the destination repo kubernetes/wbsite does not exist. <detailsIn response to : I think this should be transferred to the k/website so that the right annotation is recorded. /transfer wbsite Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
my bad !! /transfer website
/assign
Due to an implementation error with , in k8s v1.29 release this annotation was deprecated and the same is replaced by annotation. made this changes.
PR Raised .


Question: * コンテナランタイムがcgroup v2をサポートしていること。例えば、 * [containerd](https://containerd.io/) v1.4以降 * [cri-o](https://cri-o.io/) v1.20以降 <del> * kubeletとコンテナランタイムが[systemd cgroupドライバー](/docs/setup/production-environment/container-runtimes#systemd-cgroup-driver)を使うように設定されていること </del> <ins> * kubeletとコンテナランタイムが[systemd cgroupドライバー](/ja/docs/setup/production-environment/container-runtimes#systemd-cgroup-driver)を使うように設定されていること </ins> ### Linuxディストリビューションのcgroup v2サポート