Question: <ins> --- title: 分散システムコーディネーターZooKeeperの実行 content_type: tutorial weight: 40 --- <!-- overview --> このチュートリアルでは、[StatefulSet](/ja/docs/concepts/workloads/controllers/statefulset/)、[PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#pod-disruption-budget)、[Podアンチアフィニティ](/ja/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)を使って、Kubernetes上での[Apache Zookeeper](https://zookeeper.apache.org)の実行をデモンストレーションします。 ## {{% heading "prerequisites" %}} このチュートリアルを始める前に、以下のKubernetesの概念について理解しておく必要があります。 - [Pod](/ja/docs/concepts/workloads/pods/) - [クラスターDNS](/ja/docs/concepts/services-networking/dns-pod-service/) - [Headless Service](/ja/docs/concepts/services-networking/service/#headless-service) - [PersistentVolume](/ja/docs/concepts/storage/volumes/) - [PersistentVolume Provisioning](https://github.com/kubernetes/examples/tree/master/staging/persistent-volume-provisioning/) - [StatefulSet](/ja/docs/concepts/workloads/controllers/statefulset/) - [PodDisruptionBudgets](/docs/concepts/workloads/pods/disruptions/#pod-disruption-budget) - [Podアンチアフィニティ](/ja/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) - [kubectl CLI](/docs/reference/kubectl/kubectl/) 少なくとも4つのノードのクラスターが必要で、各ノードは少なくとも2つのCPUと4GiBのメモリが必須です。このチュートリアルでは、クラスターのノードをcordonおよびdrainします。 **つまり、クラスターがそのノードの全てのPodを終了して退去させ、ノードが一時的にスケジュールできなくなる、ということです。** このチュートリアル専用のクラスターを使うか、起こした破壊がほかのテナントに干渉しない確証を得ることをお勧めします。 このチュートリアルでは、クラスターがPersistentVolumeの動的なプロビジョニングが行われるように設定されていることを前提としています。 クラスターがそのように設定されていない場合、チュートリアルを始める前に20GiBのボリュームを3つ、手動でプロビジョニングする必要があります。 ## {{% heading "objectives" %}} このチュートリアルを終えると、以下の知識を得られます。 - StatefulSetを使ってZooKeeperアンサンブルをデプロイする方法。 - アンサンブルを一貫して設定する方法。 - ZooKeeperサーバーのデプロイをアンサンブルに広げる方法。 - 計画されたメンテナンス中もサービスが利用可能であることを保証するためにPodDisruptionBudgetsを使う方法。 <!-- lessoncontent --> ### ZooKeeper [Apache ZooKeeper](https://zookeeper.apache.org/doc/current/)は、分散アプリケーションのための、分散型オープンソースコーディネーションサービスです。 ZooKeeperでは、データの読み書き、および更新の監視ができます。 データは階層化されてファイルシステム内に編成され、アンサンブル(ZooKeeperサーバーのセット)内の全てのZooKeeperサーバーに複製されます。 データへの全ての操作はアトミックかつ逐次的に一貫性を持ちます。 ZooKeeperは、アンサンブル内の全てのサーバー間でステートマシンを複製するために[Zab](https://pdfs.semanticscholar.org/b02c/6b00bd5dbdbd951fddb00b906c82fa80f0b3.pdf)合意プロトコルを使ってこれを保証します。 アンサンブルはリーダーを選出するのにZabプロトコルを使い、選出が完了するまでデータを書き出しません。 完了すると、アンサンブルは複製するのにZabを使い、書き込みが承認されてクライアントに可視化されるより前に、全ての書き込みをクォーラムに複製することを保証します。 重み付けされたクォーラムでなければ、クォーラムは現在のリーダーを含むアンサンブルの多数を占めるコンポーネントです。 例えばアンサンブルが3つのサーバーを持つ時、リーダーとそれ以外のもう1つのサーバーを含むコンポーネントが、クォーラムを構成します。 アンサンブルがクォーラムに達しない場合、アンサンブルはデータを書き出せません。 ZooKeeperサーバー群はそれらの全てのステートマシンをメモリに保持し、それぞれの変化をストレージメディア上の永続的なWAL(Write Ahead Log)に書き出します。 サーバーがクラッシュした時には、WALをリプレーすることで以前のステートに回復できます。 WALを際限のない増加から防ぐために、ZooKeeperサーバーは、メモリステートにあるものをストレージメディアに定期的にスナップショットします。 これらのスナップショットはメモリに直接読み込むことができ、スナップショットより前の全てのWALエントリは破棄され得ます。 ## ZooKeeperアンサンブルの作成 以下のマニフェストは[Headless Service](/ja/docs/concepts/services-networking/service/#headless-services)、[Service](/ja/docs/concepts/services-networking/service/)、[PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets)、[StatefulSet](/ja/docs/concepts/workloads/controllers/statefulset/)を含んでいます。 {{< codenew file="application/zookeeper/zookeeper.yaml" >}} ターミナルを開き、マニフェストを作成するために [`kubectl apply`](/docs/reference/generated/kubectl/kubectl-commands/#apply)コマンドを使います。 ```shell kubectl apply -f https://k8s.io/examples/application/zookeeper/zookeeper.yaml ``` これは`zk-hs` Headless Service、`zk-cs` Service、`zk-pdb` PodDisruptionBudget、 `zk` StatefulSetを作成します。 ``` service/zk-hs created service/zk-cs created poddisruptionbudget.policy/zk-pdb created statefulset.apps/zk created ``` StatefulSetのPodを作成するStatefulSetコントローラーを監視するため、[`kubectl get`](/docs/reference/generated/kubectl/kubectl-commands/#get)を使います。 ```shell kubectl get pods -w -l app=zk ``` `zk-2` PodがRunningおよびReadyになったら、`CTRL-C`でkubectlを終了してください。 ``` NAME      READY     STATUS    RESTARTS   AGE zk-0      0/1       Pending   0          0s zk-0      0/1       Pending   0         0s zk-0      0/1       ContainerCreating   0         0s zk-0      0/1       Running   0         19s zk-0      1/1       Running   0         40s zk-1      0/1       Pending   0         0s zk-1      0/1       Pending   0         0s zk-1      0/1       ContainerCreating   0         0s zk-1      0/1       Running   0         18s zk-1      1/1       Running   0         40s zk-2      0/1       Pending   0         0s zk-2      0/1       Pending   0         0s zk-2      0/1       ContainerCreating   0         0s zk-2      0/1       Running   0         19s zk-2      1/1       Running   0         40s ``` StatefulSetコントローラーが3つのPodを作成し、各Podは[ZooKeeper](https://archive.apache.org/dist/zookeeper/stable/)サーバー付きのコンテナを持ちます。 ### リーダーの選出のファシリテート {#facilitating-leader-election} 匿名のネットワークにおいてリーダー選出を終了するアルゴリズムがないので、Zabはリーダー選出のための明示的なメンバーシップ設定を要します。 アンサンブルの各サーバーはユニーク識別子を持つ必要があり、全てのサーバーは識別子のグローバルセットを知っている必要があり、各識別子はネットワークアドレスと関連付けられている必要があります。 `zk` StatefulSetのPodのホスト名を取得するために[`kubectl exec`](/docs/reference/generated/kubectl/kubectl-commands/#exec)を使います。 ```shell for i in 0 1 2; do kubectl exec zk-$i -- hostname; done ``` StatefulSetコントローラーは各Podに、その順序インデックスに基づくユニークなホスト名を提供します。 ホスト名は`<statefulset名>-<順序インデックス>`という形をとります。 `zk` StatefulSetの`replicas`フィールドが`3`にセットされているので、このセットのコントローラーは、ホスト名にそれぞれ`zk-0`、`zk-1`、`zk-2`が付いた3つのPodを作成します。 ``` zk-0 zk-1 zk-2 ``` ZooKeeperアンサンブルのサーバーは、ユニーク識別子として自然数を使い、それぞれのサーバーの識別子をサーバーのデータディレクトリ内の`myid`というファイルに格納します。 各サーバーの`myid`ファイルの内容を調べるには、以下のコマンドを使います。 ```shell for i in 0 1 2; do echo "myid zk-$i";kubectl exec zk-$i -- cat /var/lib/zookeeper/data/myid; done ``` 識別子が自然数で順序インデックスは正の整数なので、順序に1を加算することで識別子を生成できます。 ``` myid zk-0 1 myid zk-1 2 myid zk-2 3 ``` `zk` StatefulSet内の各Podの完全修飾ドメイン名(FQDN)を取得するには、以下のコマンドを使います。 ```shell for i in 0 1 2; do kubectl exec zk-$i -- hostname -f; done ``` `zk-hs` Serviceは、全Podのためのドメイン`zk-hs.default.svc.cluster.local`を作成します。 ``` zk-0.zk-hs.default.svc.cluster.local zk-1.zk-hs.default.svc.cluster.local zk-2.zk-hs.default.svc.cluster.local ``` [Kubernetes DNS](/ja/docs/concepts/services-networking/dns-pod-service/)のAレコードは、FQDNをPodのIPアドレスに解決します。 KubernetesがPodを再スケジュールした場合、AレコードはPodの新しいIPアドレスに更新されますが、Aレコードの名前は変更されません。 ZooKeeperはそのアプリケーション設定を`zoo.cfg`という名前のファイルに格納します。 `zk-0` Pod内の`zoo.cfg`ファイルの内容を見るには、`kubectl exec`を使います。 ```shell kubectl exec zk-0 -- cat /opt/zookeeper/conf/zoo.cfg ``` ファイル末尾にある`server.1`、`server.2`、`server.3`のプロパティの、`1`、`2`、`3`はZooKeeperサーバーの`myid`ファイル内の識別子に対応します。 これらは`zk` StatefulSet内のPodのFQDNにセットされます。 ``` clientPort=2181 dataDir=/var/lib/zookeeper/data dataLogDir=/var/lib/zookeeper/log tickTime=2000 initLimit=10 syncLimit=2000 maxClientCnxns=60 minSessionTimeout= 4000 maxSessionTimeout= 40000 autopurge.snapRetainCount=3 autopurge.purgeInterval=0 server.1=zk-0.zk-hs.default.svc.cluster.local:2888:3888 server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888 server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888 ``` ### 合意形成 {#achieving-consensus} 合意(consensus)プロトコルは、各参加者の識別子がユニークであることを要件としています。 Zabプロトコル内で同じユニーク識別子を主張する2つの参加者はないものとします。 これは、システム内のプロセスが、どのプロセスがどのデータをコミットしたかを同意できるようにするために必須です。 2つのPodが同じ順序値で起動されたなら、2つのZooKeeperサーバーはどちらもそれら自身を同じサーバーとして認識してしまうでしょう。 ```shell kubectl get pods -w -l app=zk ``` ``` NAME      READY     STATUS    RESTARTS   AGE zk-0      0/1       Pending   0          0s zk-0      0/1       Pending   0         0s zk-0      0/1       ContainerCreating   0         0s zk-0      0/1       Running   0         19s zk-0      1/1       Running   0         40s zk-1      0/1       Pending   0         0s zk-1      0/1       Pending   0         0s zk-1      0/1       ContainerCreating   0         0s zk-1      0/1       Running   0         18s zk-1      1/1       Running   0         40s zk-2      0/1       Pending   0         0s zk-2      0/1       Pending   0         0s zk-2      0/1       ContainerCreating   0         0s zk-2      0/1       Running   0         19s zk-2      1/1       Running   0         40s ``` 各PodのAレコードは、PodがReadyになった時に記入されます。そのため、ZooKeeperサーバー群のFQDNはある1つのエンドポイント、すなわち`myid`ファイルで設定された識別子を主張するユニークなZooKeeperサーバーに解決されます。 ``` zk-0.zk-hs.default.svc.cluster.local zk-1.zk-hs.default.svc.cluster.local zk-2.zk-hs.default.svc.cluster.local ``` これは、ZooKeeperの`zzo.cfg`ファイル内の`servers`プロパティが正しく設定されたアンサンブルを表していることを保証します。 ``` server.1=zk-0.zk-hs.default.svc.cluster.local:2888:3888 server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888 server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888 ``` サーバーが値のコミットを試みるためにZabプロトコルを使う時、(リーダー選出が成功していて、少なくともPodのうちの2つがRunningおよびReadyならば)それぞれのサーバーは双方の合意をとって値をコミット、あるいは、(もし双方の状態が合わなければ)それを行うことに失敗します。 あるサーバーが別のサーバーを代行して書き込みを承認する状態は発生しません。 ### アンサンブルの健全性テスト {#sanity-testing-the-ensemble} 最も基本的な健全性テストは、データを1つのZooKeeperサーバーに書き込み、そのデータを別のサーバーで読み取ることです。 以下のコマンドは、`world`をアンサンブル内の`zk-0` Podのパス`/hello`に書き込むのに、`zkCli.sh`スクリプトを実行します。 ```shell kubectl exec zk-0 -- zkCli.sh create /hello world ``` ``` WATCHER:: WatchedEvent state:SyncConnected type:None path:null Created /hello ``` `zk-1` Podからデータを取得するには、以下のコマンドを使います。 ```shell kubectl exec zk-1 -- zkCli.sh get /hello ``` `zk-0`に作成したデータは、アンサンブル内の全てのサーバーで利用できます。 ``` WATCHER:: WatchedEvent state:SyncConnected type:None path:null world cZxid = 0x100000002 ctime = Thu Dec 08 15:13:30 UTC 2016 mZxid = 0x100000002 mtime = Thu Dec 08 15:13:30 UTC 2016 pZxid = 0x100000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 5 numChildren = 0 ``` ### 永続的なストレージの提供 [ZooKeeperの概要](#zookeeper)のセクションで言及したように、 ZooKeeperは全てのエントリを永続的なWALにコミットし、定期的にメモリ状態のスナップショットをストレージメディアに書き出します。 永続性を提供するためにWALを使用するのは、複製されたステートマシンを立てるために合意プロトコルを使うアプリケーションでよくあるテクニックです。 `zk` StatefulSetを削除するために、[`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands/#delete)コマンドを使います。 ```shell kubectl delete statefulset zk ``` ``` statefulset.apps "zk" deleted ``` StatefulSet内のPodの終了を観察します。 ```shell kubectl get pods -w -l app=zk ``` `zk-0`が完全に終了したら、`CTRL-C`でkubectlを終了します。 ``` zk-2      1/1       Terminating   0         9m zk-0      1/1       Terminating   0         11m zk-1      1/1       Terminating   0         10m zk-2      0/1       Terminating   0         9m zk-2      0/1       Terminating   0         9m zk-2      0/1       Terminating   0         9m zk-1      0/1       Terminating   0         10m zk-1      0/1       Terminating   0         10m zk-1      0/1       Terminating   0         10m zk-0      0/1       Terminating   0         11m zk-0      0/1       Terminating   0         11m zk-0      0/1       Terminating   0         11m ``` `zookeeper.yaml`のマニフェストを再適用します。 ```shell kubectl apply -f https://k8s.io/examples/application/zookeeper/zookeeper.yaml ``` これは`zk` StatefulSetオブジェクトを作成しますが、マニフェストのその他のAPIオブジェクトはすでに存在しているので変更されません。 StatefulSetコントローラーがStatefulSetのPodを再作成するのを見てみます。 ```shell kubectl get pods -w -l app=zk ``` `zk-2` PodがRunningおよびReadyになったら、`CTRL-C`でkubectlを終了します。 ``` NAME      READY     STATUS    RESTARTS   AGE zk-0      0/1       Pending   0          0s zk-0      0/1       Pending   0         0s zk-0      0/1       ContainerCreating   0         0s zk-0      0/1       Running   0         19s zk-0      1/1       Running   0         40s zk-1      0/1       Pending   0         0s zk-1      0/1       Pending   0         0s zk-1      0/1       ContainerCreating   0         0s zk-1      0/1       Running   0         18s zk-1      1/1       Running   0         40s zk-2      0/1       Pending   0         0s zk-2      0/1       Pending   0         0s zk-2      0/1       ContainerCreating   0         0s zk-2      0/1       Running   0         19s zk-2      1/1       Running   0         40s ``` [健全性テスト](#sanity-testing-the-ensemble)で入力した値を`zk-2` Podから取得するには、以下のコマンドを使います。 ```shell kubectl exec zk-2 zkCli.sh get /hello ``` `zk` StatefulSet内の全てのPodを終了して再作成したにもかかわらず、アンサンブルは元の値をなおも供給します。 ``` WATCHER:: WatchedEvent state:SyncConnected type:None path:null world cZxid = 0x100000002 ctime = Thu Dec 08 15:13:30 UTC 2016 mZxid = 0x100000002 mtime = Thu Dec 08 15:13:30 UTC 2016 pZxid = 0x100000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 5 numChildren = 0 ``` `zk` StatefulSetの`spec`の`volumeClaimTemplates`フィールドは、各PodにプロビジョニングされるPersistentVolumeを指定します。 ```yaml volumeClaimTemplates: - metadata: name: datadir annotations: volume.alpha.kubernetes.io/storage-class: anything spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 20Gi ``` `StatefulSet`コントローラーは、`StatefulSet`内の各Podのために`PersistentVolumeClaim`を生成します。 `StatefulSet`の`PersistentVolumeClaims`を取得するために、以下のコマンドを使います。 ```shell kubectl get pvc -l app=zk ``` `StatefulSet`がそのPodを再作成した時、`StatefulSet`はPodのPersistentVolumeを再マウントします。 ``` NAME           STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE datadir-zk-0   Bound     pvc-bed742cd-bcb1-11e6-994f-42010a800002   20Gi       RWO           1h datadir-zk-1   Bound     pvc-bedd27d2-bcb1-11e6-994f-42010a800002   20Gi       RWO           1h datadir-zk-2   Bound     pvc-bee0817e-bcb1-11e6-994f-42010a800002   20Gi       RWO           1h ``` `StatefulSet`のコンテナ`template`の`volumeMounts`セクションは、ZooKeeperサーバーのデータディレクトリにあるPersistentVolumeをマウントします。 ```yaml volumeMounts: - name: datadir mountPath: /var/lib/zookeeper ``` `zk` `StatefulSet`内のPodが(再)スケジュールされると、ZooKeeperサーバーのデータディレクトリにマウントされた同じ`PersistentVolume`を常に得ます。 Podが再スケジュールされたとしても、全ての書き込みはZooKeeperサーバーのWALおよび全スナップショットに行われ、永続性は残ります。 ## 一貫性のある設定の保証 [リーダーの選出のファシリテート](#facilitating-leader-election)および[合意形成](#achieving-consensus)のセクションで記したように、ZooKeeperのアンサンブル内のサーバー群は、リーダーを選出しクォーラムを形成するための一貫性のある設定を必要とします。 また、プロトコルがネットワーク越しで正しく動作するために、Zabプロトコルの一貫性のある設定も必要です。 この例では、設定を直接マニフェストに埋め込むことで一貫性のある設定を達成します。 `zk` StatefulSetを取得しましょう。 ```shell kubectl get sts zk -o yaml ``` ``` … command: - sh - -c - "start-zookeeper  --servers=3  --data_dir=/var/lib/zookeeper/data  --data_log_dir=/var/lib/zookeeper/data/log  --conf_dir=/opt/zookeeper/conf  --client_port=2181  --election_port=3888  --server_port=2888  --tick_time=2000  --init_limit=10  --sync_limit=5  --heap=512M  --max_client_cnxns=60  --snap_retain_count=3  --purge_interval=12  --max_session_timeout=40000  --min_session_timeout=4000  --log_level=INFO" … ``` このcommandでは、ZooKeeperサーバーを開始するためにコマンドラインパラメータで設定を渡しています。 設定をアンサンブルへ渡すのには環境変数を使うこともできます。 ### ログの設定 `zkGenConfig.sh`スクリプトで生成されたファイルの1つは、ZooKeeperのログを制御します。 ZooKeeperは[Log4j](https://logging.apache.org/log4j/2.x/)を使い、デフォルトではログの設定に基づいて、ログ設定に時間およびサイズベースでのローリングファイルアペンダー(ログのローテーション)を使用します。 `zk` `StatefulSet`内のPodの1つからログ設定を取得するには、以下のコマンドを使います。 ```shell kubectl exec zk-0 cat /usr/etc/zookeeper/log4j.properties ``` 以下のログ設定は、ZooKeeperにログの全てを標準出力ファイルストリームに書き出す処理をさせます。 ``` zookeeper.root.logger=CONSOLE zookeeper.console.threshold=INFO log4j.rootLogger=${zookeeper.root.logger} log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender log4j.appender.CONSOLE.Threshold=${zookeeper.console.threshold} log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n ``` これはログコンテナ内のログを安全にとるための、最もシンプルと思われる方法です。 アプリケーションはログを標準出力に書き出し、Kubernetesがログのローテーションを処理してくれます。 Kubernetesは、標準出力と標準エラー出力に書き出されるアプリケーションのログがローカルストレージメディアを使い尽くさないことを保証する、健全維持ポリシーも実装しています。 Podの1つから末尾20行を取得するために、[`kubectl logs`](/docs/reference/generated/kubectl/kubectl-commands/#logs)を使ってみます。 ```shell kubectl logs zk-0 --tail 20 ``` `kubectl logs`を利用するか、Kubernetes Dashboardから、標準出力または標準エラーに書き出されたアプリケーションログを参照できます。 ``` 2016-12-06 19:34:16,236 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52740 2016-12-06 19:34:16,237 [myid:1] - INFO  [Thread-1136:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52740 (no session established for client) 2016-12-06 19:34:26,155 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52749 2016-12-06 19:34:26,155 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52749 2016-12-06 19:34:26,156 [myid:1] - INFO  [Thread-1137:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52749 (no session established for client) 2016-12-06 19:34:26,222 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52750 2016-12-06 19:34:26,222 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52750 2016-12-06 19:34:26,226 [myid:1] - INFO  [Thread-1138:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52750 (no session established for client) 2016-12-06 19:34:36,151 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52760 2016-12-06 19:34:36,152 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52760 2016-12-06 19:34:36,152 [myid:1] - INFO  [Thread-1139:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52760 (no session established for client) 2016-12-06 19:34:36,230 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52761 2016-12-06 19:34:36,231 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52761 2016-12-06 19:34:36,231 [myid:1] - INFO  [Thread-1140:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52761 (no session established for client) 2016-12-06 19:34:46,149 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52767 2016-12-06 19:34:46,149 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52767 2016-12-06 19:34:46,149 [myid:1] - INFO  [Thread-1141:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52767 (no session established for client) 2016-12-06 19:34:46,230 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52768 2016-12-06 19:34:46,230 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52768 2016-12-06 19:34:46,230 [myid:1] - INFO  [Thread-1142:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52768 (no session established for client) ``` Kubernetesは多くのログソリューションを統合しています。 クラスターおよびアプリケーションに最も適合するログソリューションを選べます。 クラスターレベルのロギングとアグリゲーションとして、ログをローテートおよび輸送するための[サイドカーコンテナ](/ja/docs/concepts/cluster-administration/logging#sidecar-container-with-logging-agent)をデプロイすることを検討してください。 ### 非特権ユーザーの設定 コンテナ内で特権ユーザーとしての実行をアプリケーションに許可するベストプラクティスは、議論の的です。 アプリケーションが非特権ユーザーとして動作することを組織で必須としているなら、エントリポイントがそのユーザーとして実行できるユーザーを制御する[セキュリティコンテキスト](/ja/docs/tasks/configure-pod-container/security-context/)を利用できます。 `zk` `StatefulSet`のPod `template`は、`SecurityContext`を含んでいます。 ```yaml securityContext: runAsUser: 1000 fsGroup: 1000 ``` Podのコンテナ内で、UID 1000はzookeeperユーザーに、GID 1000はzookeeperグループにそれぞれ相当します。 `zk-0` PodからのZooKeeperプロセス情報を取得してみましょう。 ```shell kubectl exec zk-0 -- ps -elf ``` `securityContext`オブジェクトの`runAsUser`フィールドが1000にセットされているとおり、ZooKeeperプロセスは、rootとして実行される代わりにzookeeperユーザーとして実行されています。 ``` F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD 4 S zookeep+     1     0  0  80   0 -  1127 -      20:46 ?        00:00:00 sh -c zkGenConfig.sh && zkServer.sh start-foreground 0 S zookeep+    27     1  0  80   0 - 1155556 -    20:46 ?        00:00:19 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg ``` デフォルトでは、PodのPersistentVolumeがZooKeeperサーバーのデータディレクトリにマウントされている時、rootユーザーのみがそこにアクセス可能です。 この設定はZooKeeperのプロセスがそのWALに書き込んだりスナップショットに格納したりするのを妨げることになります。 `zk-0` PodのZooKeeperデータディレクトリのファイルパーミッションを取得するには、以下のコマンドを使います。 ```shell kubectl exec -ti zk-0 -- ls -ld /var/lib/zookeeper/data ``` `securityContext`オブジェクトの`fsGroup`フィールドが1000にセットされているので、PodのPersistentVolumeの所有権はzookeeperグループにセットされ、ZooKeeperのプロセスがそのデータを読み書きできるようになります。 ``` drwxr-sr-x 3 zookeeper zookeeper 4096 Dec  5 20:45 /var/lib/zookeeper/data ``` ## ZooKeeperプロセスの管理 [ZooKeeperドキュメント](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_supervision)では、「You will want to have a supervisory process that manages each of your ZooKeeper server processes (JVM).(各ZooKeeperサーバープロセス(JVM)を管理する監督プロセスを持ちたくなります)」と述べています。 分散型システム内で失敗したプロセスを再起動するのにwatchdog(監督プロセス)を使うのは、典型的パターンです。 アプリケーションをKubernetesにデプロイする時には、監督プロセスのような外部ユーティリティを使うよりもむしろ、アプリケーションのwatchdogとしてKubernetesを使うべきです。 ### アンサンブルのアップデート `zk` `StatefulSet`は`RollingUpdate`アップデート戦略を使うように設定されています。 サーバーに割り当てられる`cpus`の数を更新するのに、`kubectl patch`を利用できます。 ```shell kubectl patch sts zk --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value":"0.3"}]' ``` ``` statefulset.apps/zk patched ``` 更新の状況を見るには、`kubectl rollout status`を使います。 ```shell kubectl rollout status sts/zk ``` ``` waiting for statefulset rolling update to complete 0 pods at revision zk-5db4499664... Waiting for 1 pods to be ready... Waiting for 1 pods to be ready... waiting for statefulset rolling update to complete 1 pods at revision zk-5db4499664... Waiting for 1 pods to be ready... Waiting for 1 pods to be ready... waiting for statefulset rolling update to complete 2 pods at revision zk-5db4499664... Waiting for 1 pods to be ready... Waiting for 1 pods to be ready... statefulset rolling update complete 3 pods at revision zk-5db4499664... ``` これはPod群を終了し、逆の順番で1つずつそれらを新しい設定で再作成します。 これはクォーラムがローリングアップデート中に維持されることを保証します。 履歴や過去の設定を見るには、`kubectl rollout history`コマンドを使います。 ```shell kubectl rollout history sts/zk ``` 出力は次のようになります: ``` statefulsets "zk" REVISION 1 2 ``` 変更をロールバックするには、`kubectl rollout undo`コマンドを使います。 ```shell kubectl rollout undo sts/zk ``` 出力は次のようになります: ``` statefulset.apps/zk rolled back ``` ### プロセスの失敗の取り扱い [再起動ポリシー](/ja/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy)は、Pod内のコンテナのエントリポイントへのプロセスの失敗をKubernetesがどのように取り扱うかを制御します。 `StatefulSet`内のPodにおいて唯一妥当な`RestartPolicy`はAlwaysで、これはデフォルト値です。 ステートフルなアプリケーションでは、このデフォルトポリシーの上書きは**絶対にすべきではありません**。 `zk-0` Pod内で実行されているZooKeeperサーバーのプロセスツリーを調査するには、以下のコマンドを使います。 ```shell kubectl exec zk-0 -- ps -ef ``` コンテナのエントリポイントとして使われるコマンドはPID 1、エントリポイントの子であるZooKeeperプロセスはPID 27となっています。 ``` UID        PID  PPID  C STIME TTY          TIME CMD zookeep+     1     0  0 15:03 ?        00:00:00 sh -c zkGenConfig.sh && zkServer.sh start-foreground zookeep+    27     1  0 15:03 ?        00:00:03 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg ``` 別のターミナルで、以下のコマンドを使って`zk` `StatefulSet`内のPodを見てみます。 ```shell kubectl get pod -w -l app=zk ``` 別のターミナルで、以下のコマンドを使ってPod `zk-0`内のZooKeeperプロセスを終了します。 ```shell kubectl exec zk-0 -- pkill java ``` ZooKeeperプロセスの終了は、その親プロセスの終了を引き起こします。 コンテナの`RestartPolicy`はAlwaysなので、親プロセスが再起動(restart)されます。 ``` NAME      READY     STATUS    RESTARTS   AGE zk-0      1/1       Running   0          21m zk-1      1/1       Running   0          20m zk-2      1/1       Running   0          19m NAME      READY     STATUS    RESTARTS   AGE zk-0      0/1       Error     0          29m zk-0      0/1       Running   1         29m zk-0      1/1       Running   1         29m ``` アプリケーションが、そのビジネスロジックを実装するプロセスを立ち上げるのにスクリプト(`zkServer.sh`など)を使っている場合、スクリプトは子プロセスとともに終了する必要があります。 これは、Kubernetesがアプリケーションのコンテナを、そのビジネスロジックを実装しているプロセスが失敗した時に再起動することを保証します。 ### 生存性テスト 失敗したプロセスを再起動するための設定をアプリケーションに施すのは、分散型システムの健全さを保つのに十分ではありません。 システムのプロセスが生きていることもあれば無反応なこともあり、あるいはそうでなく不健全という状況もあります。 アプリケーションのプロセスが不健全で再起動すべきであることをKubernetesに通知するには、liveness probeを使うのがよいでしょう。 `zk` `StatefulSet`のPod `template`でliveness probeを指定します。 ```yaml livenessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 15 timeoutSeconds: 5 ``` プローブはサーバーの健全さをテストするのに、ZooKeeperの`ruok` 4文字コマンドを使うbashスクリプトを呼び出します。 ``` OK=$(echo ruok | nc 127.0.0.1 $1) if [ "$OK" == "imok" ]; then exit 0 else exit 1 fi ``` ターミナルウィンドウで、`zk` StatefulSet内のPodを見るのに以下のコマンドを使います。 ```shell kubectl get pod -w -l app=zk ``` 別のウィンドウで、Pod `zk-0`のファイルシステムから`zookeeper-ready`スクリプトを削除するために以下のコマンドを使います。 ```shell kubectl exec zk-0 -- rm /opt/zookeeper/bin/zookeeper-ready ``` ZooKeeperプロセスの失敗のためにliveness probeを使う時、アンサンブル内の不健全なプロセスが再起動されることを保証するために、Kubernetesは自動的にプロセスを再起動します。 ```shell kubectl get pod -w -l app=zk ``` ``` NAME      READY     STATUS    RESTARTS   AGE zk-0      1/1       Running   0          1h zk-1      1/1       Running   0          1h zk-2      1/1       Running   0          1h NAME      READY     STATUS    RESTARTS   AGE zk-0      0/1       Running   0          1h zk-0      0/1       Running   1         1h zk-0      1/1       Running   1         1h ``` ### 準備性テスト 準備性は生存性と同じではありません。 プロセスが生きているのであれば、スケジュールされ健全です。 プロセスの準備ができたら、入力を処理できます。 生存性はなくてはならないものですが、準備性の状態には十分ではありません。 プロセスは生きてはいるが準備はできていない時、特に初期化および終了の間がそのケースに相当します。 準備性プローブを指定するとKubernetesは、準備性チェックに合格するまで、アプリケーションのプロセスがネットワークトラフィックを受け取らないことを保証します。 ZooKeeperサーバーにとって、健全性は準備性を意味します。 そのため、`zookeeper.yaml`マニフェストからの準備性プローブは、生存プローブと同一です。 ```yaml readinessProbe: exec: command: - sh - -c - "zookeeper-ready 2181" initialDelaySeconds: 15 timeoutSeconds: 5 ``` 健全性プローブと準備性プローブが同一だとしても、両方を指定することが重要です。 これは、ZooKeeperアンサンブル内の健全なサーバーだけがネットワークトラフィックを受け取ることを保証します。 ## ノードの失敗の許容 ZooKeeperはデータの変更を正しくコミットするのにサーバーのクォーラムを必要とします。 3つのサーバーのアンサンブルにおいては、書き込みの成功のために2つのサーバーは健全でなければなりません。 クォーラムベースのシステムにおいて、可用性を保証するために、メンバーは障害ドメインにデプロイされます。 個々のマシンを失ってしまうので、故障を避けるためのベストプラクティスは、同じマシン上でアプリケーションの複数のインスタンスがコロケート(同じ場所に配置)されないようにすることです。 デフォルトでKubernetesは、同じノードの`StatefulSet`にPodをコロケートします。 3つのサーバーアンサンブルを作成していたとして、2つのサーバーが同じノードにあり、そのノードが障害を起こした場合、ZooKeeperサービスのクライアントは、少なくともPodの1つが再スケジュールされるまで障害に見舞われることになります。 クリティカルシステムのプロセスがノードの失敗イベントで再スケジュールできるよう、追加のキャパシティを常にプロビジョンしておくべきです。 そうしておけば、障害は単にKubernetesのスケジューラーがZooKeeperのサーバーの1つを再スケジュールするまでの辛抱です。 ただし、ダウンタイムなしでノードの障害への耐性をサービスに持たせたいなら、`podAntiAffinity`をセットすべきです。 `zk` `StatefulSet`内のPodのノードを取得するには、以下のコマンドを使います。 ```shell for i in 0 1 2; do kubectl get pod zk-$i --template {{.spec.nodeName}}; echo ""; done ``` `zk` `StatefulSet`内の全てのPodは、別々のノードにデプロイされます。 ``` kubernetes-node-cxpk kubernetes-node-a5aq kubernetes-node-2g2d ``` これは`zk` `StatefulSet`内のPodに`PodAntiAffinity`の指定があるからです。 ```yaml affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - zk topologyKey: "kubernetes.io/hostname" ``` `requiredDuringSchedulingIgnoredDuringExecution`フィールドは、`topologyKey`で定義されたドメイン内で`app`ラベルの値が`zk`の2つのPodが絶対にコロケートすべきでないことを、Kubernetes Schedulerに指示します。 `topologyKey`の`kubernetes.io/hostname`は、ドメインが固有ノードであることを示しています。 異なるルール、ラベル、セレクターを使って、物理・ネットワーク・電源といった障害ドメイン全体に広がるアンサンブルにこのテクニックを広げることができます。 ## メンテナンスの存続 このセクションでは、ノードをcordon(スケジュール不可化)およびdorain(解放)します。もし共有クラスターでこのチュートリアルを試しているのであれば、これがほかのテナントに有害な影響を及ぼさないことを確認してください。 前のセクションでは、計画外のノード障害に備えてどのようにPodをノード全体に広げるかを示しましたが、計画されたメンテナンスのため引き起こされる一時的なノード障害に対して計画する必要もあります。 クラスター内のノードを取得するために、以下のコマンドを使います。 ```shell kubectl get nodes ``` このチュートリアルでは、4つのノードのあるクラスターを仮定しています。 クラスターが4つよりも多くある場合には、4つのノード以外全てをcordonするために[`kubectl cordon`](/docs/reference/generated/kubectl/kubectl-commands/#cordon)を使ってください。 ノードを4つに制約することで、以下のメンテナンスシミュレーションにおいてzookeeper Podをスケジュールした時に、KubernetesがアフィニティとPodDisruptionBudget制約に遭遇することを保証します。 ```shell kubectl cordon <ノード名> ``` `zk-pdb`の`PodDisruptionBudget`を取得するために、以下のコマンドを使います。 ```shell kubectl get pdb zk-pdb ``` `max-unavailable`フィールドは、`zk` `StatefulSet`の最大で1つのPodがいつでも利用できなくなる可能性があるということを、Kubernetesに指示します。 ``` NAME      MIN-AVAILABLE   MAX-UNAVAILABLE   ALLOWED-DISRUPTIONS   AGE zk-pdb    N/A             1                 1 ``` 1つ目のターミナルで、`zk` `StatefulSet`内のPodを見るのに以下のコマンドを使います。 ```shell kubectl get pods -w -l app=zk ``` 次に別のターミナルで、Podが現在スケジュールされているノードを取得するために、以下のコマンドを使います。 ```shell for i in 0 1 2; do kubectl get pod zk-$i --template {{.spec.nodeName}}; echo ""; done ``` 出力は次のようになります: ``` kubernetes-node-pb41 kubernetes-node-ixsl kubernetes-node-i4c4 ``` `zk-0` Podがスケジュールされているノードをcordonおよびdrainするには、[`kubectl drain`](/docs/reference/generated/kubectl/kubectl-commands/#drain)を使います。 ```shell kubectl drain $(kubectl get pod zk-0 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data ``` 出力は次のようになります: ``` node "kubernetes-node-pb41" cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-pb41, kube-proxy-kubernetes-node-pb41; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-o5elz pod "zk-0" deleted node "kubernetes-node-pb41" drained ``` クラスターに4つのノードがあるので、`kubectl drain`は成功し、`zk-0`が別のノードに再スケジュールされます。 ``` NAME      READY     STATUS    RESTARTS   AGE zk-0      1/1       Running   2          1h zk-1      1/1       Running   0          1h zk-2      1/1       Running   0          1h NAME      READY     STATUS        RESTARTS   AGE zk-0      1/1       Terminating   2          2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Pending   0         0s zk-0      0/1       Pending   0         0s zk-0      0/1       ContainerCreating   0         0s zk-0      0/1       Running   0         51s zk-0      1/1       Running   0         1m ``` 最初のターミナルで`StatefulSet`のPodを見守り、`zk-1`がスケジュールされたノードをdrainします。 ```shell kubectl drain $(kubectl get pod zk-1 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data ``` 出力は次のようになります: ``` "kubernetes-node-ixsl" cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-ixsl, kube-proxy-kubernetes-node-ixsl; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-voc74 pod "zk-1" deleted node "kubernetes-node-ixsl" drained ``` `zk` `StatefulSet`がPodのコロケーションを抑止する`PodAntiAffinity`ルールを含んでいるので、`zk-1` Podはスケジュールされず、またスケジュール可能なのは2つのノードだけなので、PodはPendingの状態のままになっています。 ```shell kubectl get pods -w -l app=zk ``` 出力は次のようになります: ``` NAME      READY     STATUS    RESTARTS   AGE zk-0      1/1       Running   2          1h zk-1      1/1       Running   0          1h zk-2      1/1       Running   0          1h NAME      READY     STATUS        RESTARTS   AGE zk-0      1/1       Terminating   2          2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Pending   0         0s zk-0      0/1       Pending   0         0s zk-0      0/1       ContainerCreating   0         0s zk-0      0/1       Running   0         51s zk-0      1/1       Running   0         1m zk-1      1/1       Terminating   0         2h zk-1      0/1       Terminating   0         2h zk-1      0/1       Terminating   0         2h zk-1      0/1       Terminating   0         2h zk-1      0/1       Pending   0         0s zk-1      0/1       Pending   0         0s ``` StatefulSetのPodを見続け、`zk-2`がスケジュールされているノードをdrainします。 ```shell kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data ``` 出力は次のようになります: ``` node "kubernetes-node-i4c4" cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-i4c4, kube-proxy-kubernetes-node-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog WARNING: Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog; Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-i4c4, kube-proxy-kubernetes-node-i4c4 There are pending pods when an error occurred: Cannot evict pod as it would violate the pod's disruption budget. pod/zk-2 ``` kubectlを終了するために`CTRL-C`を押します。 `zk-2`を退去させると`zk-budget`違反になってしまうので、3つ目のノードはdrainできません。ただし、ノードはcordonされたままとなります。 健全性テスト中に入力した値を`zk-0`から取得するには、`zkCli.sh`を使います。 ```shell kubectl exec zk-0 zkCli.sh get /hello ``` `PodDisruptionBudget`が遵守されているので、サービスはまだ利用可能です。 ``` WatchedEvent state:SyncConnected type:None path:null world cZxid = 0x200000002 ctime = Wed Dec 07 00:08:59 UTC 2016 mZxid = 0x200000002 mtime = Wed Dec 07 00:08:59 UTC 2016 pZxid = 0x200000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 5 numChildren = 0 ``` 最初のノードをuncordon(スケジュール可能化)するには、[`kubectl uncordon`](/docs/reference/generated/kubectl/kubectl-commands/#uncordon)を使います。 ```shell kubectl uncordon kubernetes-node-pb41 ``` 出力は次のようになります: ``` node "kubernetes-node-pb41" uncordoned ``` `zk-1`はこのノードで再スケジュールされます。`zk-1`がRunningおよびReadyになるまで待ちます。 ```shell kubectl get pods -w -l app=zk ``` 出力は次のようになります: ``` NAME      READY     STATUS    RESTARTS   AGE zk-0      1/1       Running   2          1h zk-1      1/1       Running   0          1h zk-2      1/1       Running   0          1h NAME      READY     STATUS        RESTARTS   AGE zk-0      1/1       Terminating   2          2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Terminating   2         2h zk-0      0/1       Pending   0         0s zk-0      0/1       Pending   0         0s zk-0      0/1       ContainerCreating   0         0s zk-0      0/1       Running   0         51s zk-0      1/1       Running   0         1m zk-1      1/1       Terminating   0         2h zk-1      0/1       Terminating   0         2h zk-1      0/1       Terminating   0         2h zk-1      0/1       Terminating   0         2h zk-1      0/1       Pending   0         0s zk-1      0/1       Pending   0         0s zk-1      0/1       Pending   0         12m zk-1      0/1       ContainerCreating   0         12m zk-1      0/1       Running   0         13m zk-1      1/1       Running   0         13m ``` 試しに`zk-2`がスケジュールされているノードをdrainしてみます。 ```shell kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data ``` 出力は次のようになります: ``` node "kubernetes-node-i4c4" already cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-i4c4, kube-proxy-kubernetes-node-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog pod "heapster-v1.2.0-2604621511-wht1r" deleted pod "zk-2" deleted node "kubernetes-node-i4c4" drained ``` 今度は`kubectl drain`は成功しました。 `zk-2`の再スケジュールができるように、2つ目のノードをuncordonします。 ```shell kubectl uncordon kubernetes-node-ixsl ``` 出力は次のようになります: ``` node "kubernetes-node-ixsl" uncordoned ``` サービスがメンテナンス中も利用可能なままであることを保証するために、`PodDisruptionBudgets`とあわせて`kubectl drain`を利用できます。 メンテナンスでノードがオフラインになる前にノードをcordonして、Podを退去させるのにdrainが使われている場合、Disruption Budget(停止状態の予算)を表すサービスは遵守すべきバジェットを持ちます。 クリティカルサービスでは、Podをすぐに再スケジュールできるよう、追加のキャパティを常に割り当てておくべきです。 ## {{% heading "cleanup" %}} - クラスターの全てのノードをuncordonするために、`kubectl uncordon`を実行してください。 - このチュートリアルで使ったPersistentVolumeの永続的なストレージメディアを削除する必要があります。 全てのストレージが回収されたことを確実とするために、お使いの環境、ストレージ設定、プロビジョニング方法に基いて必要な手順に従ってください。 </ins>

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be <!-- Describe as precisely as possible how this feature/enhancement should work from the user perspective. What should be changed, etc. --Translate into Japanese Why is this needed Comments <!-- Any additional related comments that might help. Drawings/mockups would be extremely helpful (if required). --Estimated completion in a weekend. /language ja /assign
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale (PR is already under reviewing)


Question: <ins> apiVersion: v1 kind: Pod metadata: name: dapi-test-pod spec: containers: - name: test-container image: k8s.gcr.io/busybox command: [ "/bin/sh", "-c", "env" ] env: - name: SPECIAL_LEVEL_KEY valueFrom: configMapKeyRef: name: special-config key: special.how - name: LOG_LEVEL valueFrom: configMapKeyRef: name: env-config key: log_level restartPolicy: Never </ins>

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be Translate tasks/configure-pod-container/configure-pod-configmap/ into Japanese. Why is this needed This page has not been translated. Comments /language ja
/assign
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: 401: Unauthorized <del> 404: Not Found (if the `CheckpointContainer` feature gate is disabled) </del> <ins> 404: Not Found (if the `ContainerCheckpoint` feature gate is disabled) </ins> 404: Not Found (if the specified `namespace`, `pod` or `container` cannot be found)

Feature Gate ChecpointContainer is unrecognized by kubelet. I've tried to set up a new 1.25 cluster with kubeadm with kubelet config And it fails with I've also tried to set up another cluster without this feature gate and enable it after, and it goes the same. Kubelet is started and checkpoint api is accessible. To reproduce try to install a new cluster with kubeadm v1.25.0 and use next kubelet config Or try to enable this feature gate on already installed 1.25.0 cluster No response <details</details<detailsBare Metal </details<details</details<detailskubeadm 1.25.0 </details<detailscontainerd </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The feature gate name is according to
/sig node
/kind documentation
doesn't list a feature gate for kubelets. has a feature gate named . I'm guessing that this is a docs bug and is incorrect (possibly other docs pages too).
Thank you Yep, using as feature gate name resolves the issue. So this is a doc bug.
/retitle Feature gate misnamed in docs
/transfer website
This issue is now and should fix it. /language en /triage accepted
/sig docs
/priority backlog


Question: <!--more--> <del> kubeadmを使用して、コントロールプレーンとワーカーノード{{< glossary_tooltip text="ワーカーノード" term_id="node" >}}コンポーネントの両方をインストールできます。 </del> <ins> kubeadmを使用して、コントロールプレーンと{{< glossary_tooltip text="ワーカーノード" term_id="node" >}}コンポーネントの両方をインストールできます。 </ins>

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: There is typo in Following is the text with a typo. kubeadm Proposed Solution: fix "" to "" Page to Update: <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--<!--Additional Information:--
Page link for reported issue: Markdown of glossary entry reported to have issue: /language ja
/area localization
/assign
/triage accepted


Question:
 --- title: Autoscaling Workloads description: >- With autoscaling, you can automatically update your workloads in one way or another. This allows your cluster to react to changes in resource demand more elastically and efficiently. content_type: concept weight: 40 ---  In Kubernetes, you can _scale_ a workload depending on the current demand of resources. This allows your cluster to react to changes in resource demand more elastically and efficiently. When you scale a workload, you can either increase or decrease the number of replicas managed by the workload, or adjust the resources available to the replicas in-place. The first approach is referred to as _horizontal scaling_, while the second is referred to as _vertical scaling_. There are manual and automatic ways to scale your workloads, depending on your use case.  ## Scaling workloads manually Kubernetes supports _manual scaling_ of workloads. Horizontal scaling can be done using the `kubectl` CLI. For vertical scaling, you need to _patch_ the resource definition of your workload. See below for examples of both strategies. - **Horizontal scaling**: [Running multiple instances of your app](/docs/tutorials/kubernetes-basics/scale/scale-intro/) - **Vertical scaling**: [Resizing CPU and memory resources assigned to containers](/docs/tasks/configure-pod-container/resize-container-resources) ## Scaling workloads automatically Kubernetes also supports _automatic scaling_ of workloads, which is the focus of this page. The concept of _Autoscaling_ in Kubernetes refers to the ability to automatically update an object that manages a set of Pods (for example a {{< glossary_tooltip text="Deployment" term_id="deployment" >}}. ### Scaling workloads horizontally In Kubernetes, you can automatically scale a workload horizontally using a _HorizontalPodAutoscaler_ (HPA). It is implemented as a Kubernetes API resource and a {{< glossary_tooltip text="controller" term_id="controller" >}} and periodically adjusts the number of {{< glossary_tooltip text="replicas" term_id="replica" >}} in a workload to match observed resource utilization such as CPU or memory usage. There is a [walkthrough tutorial](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough) of configuring a HorizontalPodAutoscaler for a Deployment. ### Scaling workloads vertically {{< feature-state for_k8s_version="v1.25" state="stable" >}} You can automatically scale a workload vertically using a _VerticalPodAutoscaler_ (VPA). Different to the HPA, the VPA doesn't come with Kubernetes by default, but is a separate project that can be found [on GitHub](https://github.com/kubernetes/autoscaler/tree/9f87b78df0f1d6e142234bb32e8acbd71295585a/vertical-pod-autoscaler). Once installed, it allows you to create {{< glossary_tooltip text="CustomResourceDefinitions" term_id="customresourcedefinition" >}} (CRDs) for your workloads which define _how_ and _when_ to scale the resources of the managed replicas. {{< note >}} You will need to have the [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) installed to your cluster for the HPA to work. {{< /note >}} At the moment, the VPA can operate in four different modes: {{
}} Mode | Description :----|:----------- `Auto` | Currently `Recreate`, might change to in-place updates in the future `Recreate` | The VPA assigns resource requests on pod creation as well as updates them on existing pods by evicting them when the requested resources differ significantly from the new recommendation `Initial` | The VPA only assigns resource requests on pod creation and never changes them later. `Off` | The VPA does not automatically change the resource requirements of the pods. The recommendations are calculated and can be inspected in the VPA object. {{
}} #### Requirements for in-place resizing {{< feature-state for_k8s_version="v1.27" state="alpha" >}} Resizing a workload in-place **without** restarting the {{< glossary_tooltip text="Pods" term_id="pod" >}} or its {{< glossary_tooltip text="Containers" term_id="container" >}} requires Kubernetes version 1.27 or later.
 Additionally, the `InPlaceVerticalScaling` feature gate needs to be enabled. {{< feature-gate-description name="InPlacePodVerticalScaling" >}} ### Autoscaling based on cluster size For workloads that need to be scaled based on the size of the cluster (for example `cluster-dns` or other system components), you can use the _Cluster Proportional Autoscaler_.
 Just like the VPA, it is not part of the Kubernetes core, but hosted in its own repository [on GitHub](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler). The Cluster Proportional Autoscaler watches the number of schedulable {{< glossary_tooltip text="nodes" term_id="node" >}} and cores and scales the number of replicas of the target workload accordingly. ### Event driven Autoscaling It is also possible to scale workloads based on events, for example using the [_Kubernetes Event Driven Autoscaler_ (**KEDA**)](https://keda.sh/). KEDA is a CNCF graduated enabling you to scale your workloads based on the number of events to be processed, for example the amount of messages in a queue. There exists a wide range of adapters for different event sources to choose from. ### Autoscaling based on schedules Another strategy for scaling your workloads is to **schedule** the scaling operations, for example in order to reduce resource consumption during off-peak hours. Similar to event driven autoscaling, such behavior can be achieved using KEDA in conjunction with its [`Cron` scaler](https://keda.sh/docs/2.13/scalers/cron/). The `Cron` scaler allows you to define schedules (and time zones) for scaling your workloads in or out. ## Scaling cluster infrastructure If scaling workloads isn't enough to meet your needs, you can also scale your cluster infrastructure itself. Scaling the cluster infrastructure normally means adding or removing {{< glossary_tooltip text="nodes" term_id="node" >}}. This can be done using one of two available autoscalers: - [**Cluster Autoscaler**](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) - [**Karpenter**](https://github.com/kubernetes-sigs/karpenter?tab=readme-ov-file) Both scalers work by watching for pods marked as _unschedulable_ or _underutilized_ nodes and then adding or removing nodes as needed. ## {{% heading "whatsnext" %}} - Learn more about scaling horizontally - [Scale a StatefulSet](/docs/tasks/run-application/scale-stateful-set/) - [HorizontalPodAutoscaler Walkthrough](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) - [Resize Container Resources In-Place](/docs/tasks/configure-pod-container/resize-container-resources/) - [Autoscale the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/)

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be Within , add a new and brief page that has an overview of autoscaling. This means adding a new Markdown file. The new page should mention: (aim to make it clear that this is a thing that autoscales nodes, not Pods) (aim to make it clear that this is a thing that autoscales nodes, not Pods) Optionally, also mention: - marked as third party software as it's a separate project from Kubernetes - marked as third party software as it's a separate project from Kubernetes For this issue, all that's needed is a few words about each item. Later PRs could add more detail. Why is this needed our documentation on autoscaling is incomplete readers would like to have their workloads automatically sized to match demand. Comments This helps with /language en /kind feature PS. I didn't mark this as a good first issue. However, if you're looking for something to work on, and haven't already contributed to Kubernetes' docs, there may be some issues available marked as good first issues. See this . If you have already worked on an issue, please leave these for people who are new to contributing.
/triage accepted /priority important-longterm /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /triage accepted /priority important-longterm /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work on this issue /assign
Should I Make a new file inside Workloads!!


Question:
 --- title: Autoscaling Workloads description: >- With autoscaling, you can automatically update your workloads in one way or another. This allows your cluster to react to changes in resource demand more elastically and efficiently. content_type: concept weight: 40 ---  In Kubernetes, you can _scale_ a workload depending on the current demand of resources. This allows your cluster to react to changes in resource demand more elastically and efficiently. When you scale a workload, you can either increase or decrease the number of replicas managed by the workload, or adjust the resources available to the replicas in-place. The first approach is referred to as _horizontal scaling_, while the second is referred to as _vertical scaling_. There are manual and automatic ways to scale your workloads, depending on your use case.  ## Scaling workloads manually Kubernetes supports _manual scaling_ of workloads. Horizontal scaling can be done using the `kubectl` CLI. For vertical scaling, you need to _patch_ the resource definition of your workload. See below for examples of both strategies. - **Horizontal scaling**: [Running multiple instances of your app](/docs/tutorials/kubernetes-basics/scale/scale-intro/) - **Vertical scaling**: [Resizing CPU and memory resources assigned to containers](/docs/tasks/configure-pod-container/resize-container-resources) ## Scaling workloads automatically Kubernetes also supports _automatic scaling_ of workloads, which is the focus of this page. The concept of _Autoscaling_ in Kubernetes refers to the ability to automatically update an object that manages a set of Pods (for example a {{< glossary_tooltip text="Deployment" term_id="deployment" >}}. ### Scaling workloads horizontally In Kubernetes, you can automatically scale a workload horizontally using a _HorizontalPodAutoscaler_ (HPA). It is implemented as a Kubernetes API resource and a {{< glossary_tooltip text="controller" term_id="controller" >}} and periodically adjusts the number of {{< glossary_tooltip text="replicas" term_id="replica" >}} in a workload to match observed resource utilization such as CPU or memory usage. There is a [walkthrough tutorial](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough) of configuring a HorizontalPodAutoscaler for a Deployment. ### Scaling workloads vertically {{< feature-state for_k8s_version="v1.25" state="stable" >}} You can automatically scale a workload vertically using a _VerticalPodAutoscaler_ (VPA). Different to the HPA, the VPA doesn't come with Kubernetes by default, but is a separate project that can be found [on GitHub](https://github.com/kubernetes/autoscaler/tree/9f87b78df0f1d6e142234bb32e8acbd71295585a/vertical-pod-autoscaler). Once installed, it allows you to create {{< glossary_tooltip text="CustomResourceDefinitions" term_id="customresourcedefinition" >}} (CRDs) for your workloads which define _how_ and _when_ to scale the resources of the managed replicas. {{< note >}} You will need to have the [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) installed to your cluster for the HPA to work. {{< /note >}} At the moment, the VPA can operate in four different modes: {{
}} Mode | Description :----|:----------- `Auto` | Currently `Recreate`, might change to in-place updates in the future `Recreate` | The VPA assigns resource requests on pod creation as well as updates them on existing pods by evicting them when the requested resources differ significantly from the new recommendation `Initial` | The VPA only assigns resource requests on pod creation and never changes them later. `Off` | The VPA does not automatically change the resource requirements of the pods. The recommendations are calculated and can be inspected in the VPA object. {{
}} #### Requirements for in-place resizing {{< feature-state for_k8s_version="v1.27" state="alpha" >}} Resizing a workload in-place **without** restarting the {{< glossary_tooltip text="Pods" term_id="pod" >}} or its {{< glossary_tooltip text="Containers" term_id="container" >}} requires Kubernetes version 1.27 or later.
 Additionally, the `InPlaceVerticalScaling` feature gate needs to be enabled. {{< feature-gate-description name="InPlacePodVerticalScaling" >}} ### Autoscaling based on cluster size For workloads that need to be scaled based on the size of the cluster (for example `cluster-dns` or other system components), you can use the _Cluster Proportional Autoscaler_.
 Just like the VPA, it is not part of the Kubernetes core, but hosted in its own repository [on GitHub](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler). The Cluster Proportional Autoscaler watches the number of schedulable {{< glossary_tooltip text="nodes" term_id="node" >}} and cores and scales the number of replicas of the target workload accordingly. ### Event driven Autoscaling It is also possible to scale workloads based on events, for example using the [_Kubernetes Event Driven Autoscaler_ (**KEDA**)](https://keda.sh/). KEDA is a CNCF graduated enabling you to scale your workloads based on the number of events to be processed, for example the amount of messages in a queue. There exists a wide range of adapters for different event sources to choose from. ### Autoscaling based on schedules Another strategy for scaling your workloads is to **schedule** the scaling operations, for example in order to reduce resource consumption during off-peak hours. Similar to event driven autoscaling, such behavior can be achieved using KEDA in conjunction with its [`Cron` scaler](https://keda.sh/docs/2.13/scalers/cron/). The `Cron` scaler allows you to define schedules (and time zones) for scaling your workloads in or out. ## Scaling cluster infrastructure If scaling workloads isn't enough to meet your needs, you can also scale your cluster infrastructure itself. Scaling the cluster infrastructure normally means adding or removing {{< glossary_tooltip text="nodes" term_id="node" >}}. This can be done using one of two available autoscalers: - [**Cluster Autoscaler**](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) - [**Karpenter**](https://github.com/kubernetes-sigs/karpenter?tab=readme-ov-file) Both scalers work by watching for pods marked as _unschedulable_ or _underutilized_ nodes and then adding or removing nodes as needed. ## {{% heading "whatsnext" %}} - Learn more about scaling horizontally - [Scale a StatefulSet](/docs/tasks/run-application/scale-stateful-set/) - [HorizontalPodAutoscaler Walkthrough](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) - [Resize Container Resources In-Place](/docs/tasks/configure-pod-container/resize-container-resources/) - [Autoscale the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/)

In 1 PR I will be creating 2 topics like (HorizontalPodAutoscaler, VerticalPodAutoscaler) and will it be sufficient, because It will be taking 2-3 PR's!
For advice on how to contribute to the Kubernetes docs, please ask in #sig-docs on the Kubernetes Slack workspace. If you're not already a member of that workspace, you can visit for an invitation. Anyone is welcome to work on this issue. it's slightly unhelpful to assign an issue to yourself and also to ask questions about how to work on it; ideally, people only assign an issue once they are confident that they know how to resolve it.
Our no cookie-licking policy means that anyone is welcome to open a PR that closes an open issue. It's still friendly to reach out to people who might already be working on the same thing. See for an explanation of the jargon.
Thanks I was just asking it. I can fix this issue!
Hey This has been pending for more than a month. So, May I go ahead and create a PR? I have a basic markdown structure for file is ready and I can add more details as needed after raising a PR.
PRs are welcome, and because this project is open source, nobody needs permission.
In light of our policy on cookie licking: /unassign I think we'll make progress better without any assignee. If anyone wants to share that they have started work, feel free to open a draft pull request that mentions this issue.


Question:
The list of phases can be obtained with the kubeadm init --help command. The flag "--skip-phases" takes precedence over this field.
 -->
skipPhases 是命令执行过程中药略过的阶段（Phases）。
skipPhases 是命令执行过程中要略过的阶段（Phases）。  通过执行命令 kubeadm init --help 可以获得阶段的列表。 参数标志 "--skip-phases" 优先于此字段的设置。

skipPhases Phases skipPhases Phases
/triage accepted /kind bug PRs welcomed.
/language zh
/assign


Question: <ins> --- # reviewers: # - enisoc # - erictune # - foxish # - janetkuo # - kow3ns # - smarterclayton title: 복제 스테이트풀 애플리케이션 실행하기 content_type: tutorial weight: 30 --- <!-- overview --> 이 페이지에서는 {{< glossary_tooltip term_id="statefulset" >}} 으로 복제 스테이트풀 애플리케이션을 실행하는 방법에 대해 소개한다. 이 애플리케이션은 복제 MySQL 데이터베이스이다. 이 예제의 토폴로지는 단일 주 서버와 여러 복제 서버로 이루어져있으며, row-based 비동기 복제 방식을 사용한다. {{< note >}} **해당 설정은 프로덕션 설정이 아니다**. 쿠버네티스에서 스테이트풀한 애플리케이션을 실행하기 위한 일반적인 패턴에 집중하기 위해 MySQL 세팅이 안전하지 않은 기본 설정으로 되어 있다. {{< /note >}} ## {{% heading "prerequisites" %}} * {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} * {{< include "default-storage-class-prereqs.md" >}} * 이 튜토리얼은 [퍼시스턴트볼륨](/ko/docs/concepts/storage/persistent-volumes/) 그리고 [스테이트풀셋](/ko/docs/concepts/workloads/controllers/statefulset/), [파드](/ko/docs/concepts/workloads/pods/), [서비스](/ko/docs/concepts/services-networking/service/), [컨피그맵(ConfigMap)](/docs/tasks/configure-pod-container/configure-pod-configmap/)와 같은 핵심 개념들에 대해 알고 있다고 가정한다. * MySQL에 대한 지식이 있으면 도움이 되지만, 이 튜토리얼은 다른 시스템을 활용하였을 때도 도움이 되는 일반적인 패턴을 다루는데 중점을 둔다. * default 네임스페이스를 사용하거나, 다른 오브젝트들과 충돌이 나지 않는 다른 네임스페이스를 사용한다. ## {{% heading "objectives" %}} * 스테이트풀셋을 이용한 복제 MySQL 토폴로지를 배포한다. * MySQL 클라이언트에게 트래픽을 보낸다. * 다운타임에 대한 저항력을 관찰한다. * 스테이트풀셋을 확장/축소한다. <!-- lessoncontent --> ## MySQL 배포하기 MySQL 디플로이먼트 예시는 컨피그맵과, 2개의 서비스, 그리고 스테이트풀셋으로 구성되어 있다. ### 컨피그맵 생성하기 {#configmap} 다음 YAML 설정 파일로부터 컨피그맵을 생성한다. {{< codenew file="application/mysql/mysql-configmap.yaml" >}} ```shell kubectl apply -f https://k8s.io/examples/application/mysql/mysql-configmap.yaml ``` 이 컨피그맵은 당신이 독립적으로 주 MySQL 서버와 레플리카들의 설정을 컨트롤할 수 있도록 `my.cnf` 을 오버라이드한다. 이 경우에는, 주 서버는 복제 로그를 레플리카들에게 제공하고 레플리카들은 복제를 통한 쓰기가 아닌 다른 쓰기들은 거부하도록 할 것이다. 컨피그맵 자체가 다른 파드들에 서로 다른 설정 영역이 적용되도록 하는 것이 아니다. 스테이트풀셋 컨트롤러가 제공해주는 정보에 따라서, 각 파드들은 초기화되면서 설정 영역을 참조할지 결정한다. ### 서비스 생성하기 {#services} 다음 YAML 설정 파일로부터 서비스를 생성한다. {{< codenew file="application/mysql/mysql-services.yaml" >}} ```shell kubectl apply -f https://k8s.io/examples/application/mysql/mysql-services.yaml ``` 헤드리스 서비스는 스테이트풀셋 {{< glossary_tooltip text="컨트롤러" term_id="controller" >}}가 집합의 일부분인 파드들을 위해 생성한 DNS 엔트리들(entries)의 위한 거점이 된다. 헤드리스 서비스의 이름이 `mysql`이므로, 파드들은 같은 쿠버네티스 클러스터나 네임스페이스에 존재하는 다른 파드들에게 `<pod-name>.mysql`라는 이름으로 접근될 수 있다. `mysql-read`라고 불리우는 클라이언트 서비스는 고유의 클러스터 IP를 가지며, Ready 상태인 모든 MySQL 파드들에게 커넥션을 분배하는 일반적인 서비스이다. 잠재적인 엔드포인트들의 집합은 주 MySQL 서버와 해당 레플리카들을 포함한다. 오직 읽기 쿼리들만 로드-밸런싱된 클라이언트 서비스를 이용할 수 있다는 사실에 주목하자. 하나의 주 MySQL 서버만이 존재하기 떄문에, 클라이언트들은 쓰기 작업을 실행하기 위해서 주 MySQL 파드에 (헤드리스 서비스 안에 존재하는 DNS 엔트리를 통해) 직접 접근해야 한다. ### 스테이트풀셋 생성하기 {#statefulset} 마지막으로, 다음 YAML 설정 파일로부터 스테이트풀셋을 생성한다. {{< codenew file="application/mysql/mysql-statefulset.yaml" >}} ```shell kubectl apply -f https://k8s.io/examples/application/mysql/mysql-statefulset.yaml ``` 다음을 실행하여, 초기화되는 프로세스들을 확인할 수 있다. ```shell kubectl get pods -l app=mysql --watch ``` 잠시 뒤에, 3개의 파드들이 `Running` 상태가 되는 것을 볼 수 있다. ``` NAME      READY     STATUS    RESTARTS   AGE mysql-0   2/2       Running   0          2m mysql-1   2/2       Running   0          1m mysql-2   2/2       Running   0          1m ``` **Ctrl+C**를 입력하여 watch를 종료하자. {{< note >}} 만약 아무 진행 상황도 보이지 않는다면, [시작하기 전에](#before-you-begin) 에 언급된 동적 퍼시스턴트볼륨 프로비저너(provisioner)가 활성화되어 있는지 확인한다. {{< /note >}} 해당 메니페스트에는 스테이트풀셋의 일부분인 스테이트풀 파드들을 관리하기 위한 다양한 기법들이 적용되어 있다. 다음 섹션에서는 스테트풀셋이 파드들을 생성할 때 일어나는 일들을 이해할 수 있도록 일부 기법들을 강조하여 설명한다. ## 스테이트풀 파드 초기화 이해하기 스테이트풀셋 컨트롤러는 파드들의 인덱스에 따라 순차적으로 시작시킨다. 컨트롤러는 다음 파드 생성 이전에 각 파드가 Ready 상태가 되었다고 알려줄 때까지 기다린다. 추가적으로, 컨트롤러는 각 파드들에게 `<스테이트풀셋 이름>-<순차적 인덱스>` 형태의 고유하고 안정적인 이름을 부여하는데, 결과적으로 파드들은 `mysql-0`, `mysql-1`, 그리고 `mysql-2` 라는 이름을 가지게 된다. 스테이트풀셋 매니페스트의 파드 템플릿은 해당 속성들을 통해 순차적인 MySQL 복제의 시작을 수행한다. ### 설정 생성하기 파드 스펙의 컨테이너를 시작하기 전에, 파드는 순서가 정의되어 있는 [초기화 컨테이너](/ko/docs/concepts/workloads/pods/init-containers/)들을 먼저 실행시킨다. `init-mysql`라는 이름의 첫 번째 초기화 컨테이너는, 인덱스에 따라 특별한 MySQL 설정 파일을 생성한다. 스크립트는 인덱스를 `hostname` 명령으로 반환되는 파드 이름의 마지막 부분에서 추출하여 결정한다. 그리고 인덱스(이미 사용된 값들을 피하기 위한 오프셋 숫자와 함께)를 MySQL의 `conf.d` 디렉토리의 `server-id.cnf` 파일에 저장한다. 이는 스테이트풀셋에게서 제공된 고유하고, 안정적인 신원을 같은 속성을 필요로 하는 MySQL 서버 ID의 형태로 바꾸어준다. 또한 `init-mysql` 컨테이너의 스크립트는 컨피그맵을 `conf.d`로 복사하여, `primary.cnf` 또는 `replica.cnf`을 적용한다. 이 예제의 토폴로지가 하나의 주 MySQL 서버와 일정 수의 레플리카들로 이루어져 있기 때문에, 스크립트는 `0` 인덱스를 주 서버로, 그리고 나머지 값들은 레플리카로 지정한다. 스테이트풀셋 컨트롤러의 [디플로이먼트와 스케일링 보증](/ko/docs/concepts/workloads/controllers/statefulset/#디플로이먼트와-스케일링-보증)과 합쳐지면, 복제를 위한 레플리카들을 생성하기 전에 주 MySQL 서버가 Ready 상태가 되도록 보장할 수 있다. ### 기존 데이터 복제(cloning) 일반적으로, 레플리카에 새로운 파드가 추가되는 경우, 주 MySQL 서버가 이미 데이터를 가지고 있다고 가정해야 한다. 또한 복제 로그가 첫 시작점부터의 로그들을 다 가지고 있지는 않을 수 있다고 가정해야 한다. 이러한 보수적인 가정들은 스테이트풀셋이 초기 크기로 고정되어 있는 것보다, 시간에 따라 확장/축소하게 할 수 있도록 하는 중요한 열쇠가 된다. `clone-mysql`라는 이름의 두 번째 초기화 컨테이너는, 퍼시스턴트볼륨에서 처음 초기화된 레플리카 파드에 복제 작업을 수행한다. 이 말은 다른 실행 중인 파드로부터 모든 데이터들을 복제하기 때문에, 로컬의 상태가 충분히 일관성을 유지하고 있어서 주 서버에서부터 복제를 시작할 수 있다는 의미이다. MySQL 자체가 이러한 메커니즘을 제공해주지는 않기 때문에, 이 예제에서는 XtraBackup이라는 유명한 오픈소스 도구를 사용한다. 복제 중에는, 복제 대상 MySQL 서버가 성능 저하를 겪을 수 있다. 주 MySQL 서버의 이러한 충격을 최소화하기 위해, 스크립트는 각 파드가 자신의 인덱스보다 하나 작은 파드로부터 복제하도록 지시한다. 이것이 정상적으로 동작하는 이유는 스테이트풀셋 컨트롤러가 파드 `N+1`을 실행하기 전에 항상 파드 `N`이 Ready 상태라는 것을 보장하기 때문이다. ### 복제(replication) 시작하기 초기화 컨테이너들의 성공적으로 완료되면, 일반적인 컨테이너가 실행된다. MySQL 파드는 `mysqld` 서버를 구동하는 `mysql` 컨테이너로 구성되어 있으며, `xtrabackup` 컨테이너는 [사이드카(sidecar)](/blog/2015/06/the-distributed-system-toolkit-patterns)로서 작동한다. `xtrabackup` 사이드카는 복제된 데이터 파일들을 보고 레플리카에 MySQL 복제를 시작해야 할 필요가 있는지 결정한다. 만약 그렇다면, `mysqld`이 준비될 때까지 기다린 후 `CHANGE MASTER TO`, 그리고 `START SLAVE`를 XtraBackup 복제(clone) 파일들에서 추출한 복제(replication) 파라미터들과 함께 실행시킨다. 레플리카가 복제를 시작하면, 먼저 주 MySQL 서버를 기억하고, 서버가 재시작되거나 커넥션이 끊어지면 다시 연결한다. 또한 레플리카들은 주 서버를 안정된 DNS 이름 (`mysql-0.mysql`)으로 찾기 때문에, 주 서버가 리스케쥴링에 의해 새로운 파드 IP를 받아도 주 서버를 자동으로 찾는다. 마지막으로, 복제를 시작한 후에는, `xtrabackup` 컨테이너는 데이터 복제를 요청하는 다른 파드들의 커넥션을 리스닝한다. 이 서버는 스테이트풀셋이 확장하거나, 다음 파드가 퍼시스턴트볼륨클레임을 잃어서 다시 복제를 수행해햐 할 경우를 대비하여 독립적으로 존재해야 한다. ## 클라이언트 트래픽 보내기 임시 컨테이너를 `mysql:5.7` 이미지로 실행하고 `mysql` 클라이언트 바이너리를 실행하는 것으로 테스트 쿼리를 주 MySQL 서버(`mysql-0.mysql` 호스트네임)로 보낼 수 있다. ```shell kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never -- mysql -h mysql-0.mysql <<EOF CREATE DATABASE test; CREATE TABLE test.messages (message VARCHAR(250)); INSERT INTO test.messages VALUES ('hello'); EOF ``` `mysql-read` 호스트네임을 사용하여 Ready 상태가 되었다고 보고하는 어느 서버에나 시험 쿼리를 보낼 수 있다. ```shell kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never -- mysql -h mysql-read -e "SELECT * FROM test.messages" ``` 그러면 다음과 같은 출력를 얻을 것이다. ``` Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false +---------+ | message | +---------+ | hello   | +---------+ pod "mysql-client" deleted ``` `mysql-read` 서비스가 커넥션을 서버들에게 분배하고 있다는 사실을 확인하기 위해서, `SELECT @@server_id`를 루프로 돌릴 수 있다. ```shell kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never -- bash -ic "while sleep 1; do mysql -h mysql-read -e 'SELECT @@server_id,NOW()'; done" ``` 보고된 `@@server_id`가 무작위로 바뀌는 것을 알 수 있다. 왜냐하면 매 커넥션 시도마다 다른 엔드포인트가 선택될 수 있기 때문이다. ``` +-------------+---------------------+ | @@server_id | NOW()               | +-------------+---------------------+ |         100 | 2006-01-02 15:04:05 | +-------------+---------------------+ +-------------+---------------------+ | @@server_id | NOW()               | +-------------+---------------------+ |         102 | 2006-01-02 15:04:06 | +-------------+---------------------+ +-------------+---------------------+ | @@server_id | NOW()               | +-------------+---------------------+ |         101 | 2006-01-02 15:04:07 | +-------------+---------------------+ ``` 루프를 끝내고 싶다면 **Ctrl+C**를 입력하면 되지만, 다음 단계에서의 영향을 보기 위해서 다른 창에 계속 실행시켜 놓는 것이 좋다. ## 노드와 파드 실패 시뮬레이션하기 {#simulate-pod-and-node-downtime} 단일 서버 대비 레플리카들의 풀에 의한 읽기의 가용성 향상을 시연하기 위해, 파드를 Ready 상태로 강제하면서 `SELECT @@server_id` 루프를 돌리자. ### 준비성 프로브 고장내기 `mysql` 컨테이너의 [준비성 프로브](/ko/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes)는 쿼리를 실행할 수 있는지를 확인하기 위해 `mysql -h 127.0.0.1 -e 'SELECT 1'` 명령을 실행한다. 준비성 프로브를 강제로 실패시키는 방법은 해당 명령을 고장내는 것이다. ```shell kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql /usr/bin/mysql.off ``` 이것은 `mysql-2` 파드의 실제 컨테이너의 파일시스템에 접근해서 `mysql` 명령의 이름을 변경하여, 준비성 프로브가 찾을 수 없도록 한다. 몇 초 뒤에, 파드는 컨터이너 중 하나가 준비되지 않았다고 보고할 것이다. 다음 명령을 실행시켜서 확인할 수 있다. ```shell kubectl get pod mysql-2 ``` `READY` 항목의 `1/2`를 확인하자. ``` NAME      READY     STATUS    RESTARTS   AGE mysql-2   1/2       Running   0          3m ``` 이 시점에서, `102`를 보고하지 않지만, 계속해서 `SELECT @@server_id` 루프가 실행되는 것을 확인할 수 있을 것이다. `init-mysql` 스크립트에서 `server-id`를 `100 + $ordinal`로 정의했기 때문에, 서버 ID `102`는 `mysql-2` 파드에 대응된다는 사실을 상기하자. 이제 파드를 고치면 몇 초 뒤에 루프 출력에 나타날 것이다. ```shell kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql.off /usr/bin/mysql ``` ### 파드 삭제하기 레플리카셋이 스테이트리스 파드들에게 하는 것처럼, 스테이트풀셋 또한 파드들이 삭제되면 파드들을 다시 생성한다. ```shell kubectl delete pod mysql-2 ``` 스테이트풀셋 컨트롤러는 `mysql-2` 파드가 존재하지 않는 것을 인지하고, 같은 이름으로 새로운 파드를 생성하고 같은 퍼시스턴트볼륨클레임에 연결한다. 잠시동안 서버 ID `102`가 루프 출력에서 사라지고 다시 나타나는 것을 확인할 수 있을 것이다. ### 노드 드레인하기 만약 당신의 쿠버네티스 클러스터가 여러 노드들을 가지고 있으면, [드레인](/docs/reference/generated/kubectl/kubectl-commands/#drain)을 사용하여 노드 다운타임을 시뮬레이션할 수 있다. 먼저 MySQL 파드가 존재하고 있는 노드를 확인하자. ```shell kubectl get pod mysql-2 -o wide ``` 노드 이름은 마지막 열에서 나타날 것이다. ``` NAME      READY     STATUS    RESTARTS   AGE       IP            NODE mysql-2   2/2       Running   0          15m       10.244.5.27   kubernetes-node-9l2t ``` 그 후에, 다음 명령을 실행하여 노드를 드레인한다. 그러면 새로운 파드가 스케줄되지 않게 방지하고(cordon), 기존의 파드들을 추방(evict)한다. `<node-name>`를 이전 단계에서 찾았던 노드의 이름으로 바꾸자. {{< caution >}} 노드 드레인은 해당 노드에서 실행 중인 다른 워크로드와 애플리케이션들에게 영향을 줄 수 있다. 테스트 클러스터에만 다음 단계를 수행하자. {{< /caution >}} ```shell # 위에 명시된 다른 워크로드들이 받는 영향에 대한 주의사항을 확인한다. kubectl drain <node-name> --force --delete-emptydir-data --ignore-daemonsets ``` 이제 파드가 다른 노드에 리스케줄링되는 것을 관찰할 수 있다. ```shell kubectl get pod mysql-2 -o wide --watch ``` 출력은 다음과 비슷할 것이다. ``` NAME      READY   STATUS          RESTARTS   AGE       IP            NODE mysql-2   2/2     Terminating     0          15m       10.244.1.56   kubernetes-node-9l2t [...] mysql-2   0/2     Pending         0          0s        <none>        kubernetes-node-fjlm mysql-2   0/2     Init:0/2        0          0s        <none>        kubernetes-node-fjlm mysql-2   0/2     Init:1/2        0          20s       10.244.5.32   kubernetes-node-fjlm mysql-2   0/2     PodInitializing 0          21s       10.244.5.32   kubernetes-node-fjlm mysql-2   1/2     Running         0          22s       10.244.5.32   kubernetes-node-fjlm mysql-2   2/2     Running         0          30s       10.244.5.32   kubernetes-node-fjlm ``` 그리고, 서버 ID `102`가 `SELECT @@server_id` 루프 출력에서 잠시 사라진 후 다시 보이는 것을 확인할 수 있을 것이다. 이제 노드의 스케줄 방지를 다시 해제(uncordon)해서 정상으로 돌아가도록 조치한다. ```shell kubectl uncordon <node-name> ``` ## 레플리카 스케일링하기 MySQL 레플리케이션을 사용하면, 레플리카를 추가하는 것으로 읽기 쿼리 용량을 키울 수 있다. 스테이트풀셋을 사용하면, 단 한 줄의 명령으로 달성할 수 있다. ```shell kubectl scale statefulset mysql  --replicas=5 ``` 명령을 실행시켜서 새로운 파드들이 올라오는 것을 관찰하자. ```shell kubectl get pods -l app=mysql --watch ``` 파드들이 올라오면, `SELECT @@server_id` 루프 출력에 서버 ID `103` 과 `104`가 나타나기 시작할 것이다. 그리고 해당 파드들이 존재하기 전에 추가된 데이터들이 해당 새 서버들에게도 존재하는 것을 확인할 수 있다. ```shell kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never -- mysql -h mysql-3.mysql -e "SELECT * FROM test.messages" ``` ``` Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false +---------+ | message | +---------+ | hello   | +---------+ pod "mysql-client" deleted ``` 축소하는 것도 간단하게 할 수 있다. ```shell kubectl scale statefulset mysql --replicas=3 ``` {{< note >}} 확장은 퍼시스턴트볼륨클레임을 자동으로 생성하지만, 축소에서는 해당 PVC들이 자동으로 삭제되지 않는다. 이로써 확장을 빠르게 하기 위해 초기화된 PVC들을 보관해 두거나, 삭제하기 전에 데이터를 추출하는 선택을 할 수 있다. {{< /note >}} 다음 명령을 실행하여 확인할 수 있다. ```shell kubectl get pvc -l app=mysql ``` 스테이트풀셋을 3으로 축소했음에도 PVC 5개가 아직 남아있음을 보여준다. ``` NAME           STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE data-mysql-0   Bound     pvc-8acbf5dc-b103-11e6-93fa-42010a800002   10Gi       RWO           20m data-mysql-1   Bound     pvc-8ad39820-b103-11e6-93fa-42010a800002   10Gi       RWO           20m data-mysql-2   Bound     pvc-8ad69a6d-b103-11e6-93fa-42010a800002   10Gi       RWO           20m data-mysql-3   Bound     pvc-50043c45-b1c5-11e6-93fa-42010a800002   10Gi       RWO           2m data-mysql-4   Bound     pvc-500a9957-b1c5-11e6-93fa-42010a800002   10Gi       RWO           2m ``` 만약 여분의 PVC들을 재사용하지 않을 것이라면, 이들을 삭제할 수 있다. ```shell kubectl delete pvc data-mysql-3 kubectl delete pvc data-mysql-4 ``` ## {{% heading "cleanup" %}} 1. `SELECT @@server_id` 루프를 끝내기 위해, 터미널에 **Ctrl+C**를 입력하거나, 해당 명령을 다른 터미널에서 실행시키자. ```shell kubectl delete pod mysql-client-loop --now ``` 1. 스테이트풀셋을 삭제한다. 이것은 파드들 또한 삭제할 것이다. ```shell kubectl delete statefulset mysql ``` 1. 파드들의 삭제를 확인한다. 삭제가 완료되기까지 시간이 걸릴 수 있다. ```shell kubectl get pods -l app=mysql ``` 위와 같은 메세지가 나타나면 파드들이 삭제되었다는 것을 알 수 있다. ``` No resources found. ``` 1. 컨피그맵, 서비스, 그리고 퍼시스턴트볼륨클레임들을 삭제한다. ```shell kubectl delete configmap,service,pvc -l app=mysql ``` 1. 만약 수동으로 퍼시스턴스볼륨들을 프로비저닝했다면, 수동으로 삭제하면서, 그 밑에 존재하는 리소스들을 또한 삭제해야 한다. 만약 동적 프로비저너를 사용했다면, 당신이 퍼시스턴트볼륨클레임으로 삭제하면 자동으로 퍼시스턴트볼륨을 삭제한다. 일부 (EBS나 PD와 같은) 동적 프로비저너들은 퍼시스턴트볼륨을 삭제 하면 그 뒤에 존재하는 리소스들도 삭제한다. ## {{% heading "whatsnext" %}} * [스테이트풀셋(StatefulSet) 확장하기](/ko/docs/tasks/run-application/scale-stateful-set/)에 대해 더 배워보기. * [스테이트풀셋 디버깅하기](/ko/docs/tasks/debug/debug-application/debug-statefulset/)에 대해 더 배워보기. * [스테이트풀셋(StatefulSet) 삭제하기](/ko/docs/tasks/run-application/delete-stateful-set/)에 대해 더 배워보기. * [스테이트풀셋(StatefulSet) 파드 강제 삭제하기](/ko/docs/tasks/run-application/force-delete-stateful-set-pod/)에 대해 더 배워보기. * [Helm Charts 저장소](https://artifacthub.io/)를 통해 다른 스테이트풀 애플리케이션 예제들을 확인해보기. </ins>

This is a Feature Request Korean Localization for branch. What would you like to be Any localization contributions for Korean. Comments This issue is to notify the schedule of Korean localization to contributors. is the current work branch for Korean localization. Important dates A Korean localization work branch is merged to branch after we complete around 3~4 weeks job. PR Freeze: 01/16 (PR opened after freeze will be requested to rebase to next work branch) Branch Merge: 01/19 (merge to main branch)  2~3 , main . PR : 01/16 ( PR , PR ) : 01/19 (main , k8s.io ) /language ko /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ## In English is the current work branch for Korean localization.A Korean localization work branch is merged to branch after we complete around 3~4 weeks job. PR Freeze: 01/16 (PR opened after freeze will be requested to rebase to next work branch) Branch Merge: 01/19 (merge to main branch) PR : 01/16 ( PR , PR ) : 01/19 (main , k8s.io ) /help </details
/milestone dev-1.26-ko.1
/triage accepted
Outdated Korean contents in the dev-1.26-ko.1


Question: | `app.kubernetes.io/managed-by`      | The tool being used to manage the operation of an application | `helm` | string | | `app.kubernetes.io/created-by`      | The controller/user who created this resource | `controller-manager` | string | <del> To illustrate these labels in action, consider the following StatefulSet object: </del> <ins> To illustrate these labels in action, consider the following {{< glossary_tooltip text="StatefulSet" term_id="statefulset" >}} object: </ins> ```yaml <ins> # This is an excerpt </ins> apiVersion: apps/v1 kind: StatefulSet metadata:

The concept of the has not been introduced at this point yet, but is used at multiple locations on this page. It would make sense to briefly explain what that is and link to a page that covers that topic.
/triage accepted /language en /help I would use a here. We should also mention that StatefulSet is just an example. Related, the example manifest is incomplete. We should add a comment in YAML to make it clear that this is an excerpt.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /triage accepted /language en /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to fix this /assign
I think is already defined .
My idea is to reference that definition into (see PR )


Question: gcloud components install kubectl ``` <del> Do check that the version is sufficiently up-to-date using `kubectl version`. </del> <ins> Check that the version is sufficiently up-to-date using `kubectl version`. </ins> <del> ## Install with brew </del> <ins> ## Install with Homebrew on macOS </ins> <del> If you are on MacOS and using brew, you can install with: </del> <ins> If you are on macOS and using [Homebrew](https://brew.sh/) package manager, you can install with: </ins> ```shell brew install kubectl

This is a... [x] Bug Report Problem: There is no directory in the tar bundle. Example: Proposed Solution: Recommend people to execute in the bundle? Page to Update:


Question: body.cid-community #gallery { display: flex; <del> max-width: 100vw; </del> <ins> max-width: 100%; </ins> gap: 0.75rem; justify-content: center; margin-left: auto;

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: Proposed Solution: Page to Update: /assign
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
During my browsing of the docs on a mobile device, I came across a similar situation on another page, specifically in the . It seems this issue might be widespread and could be affecting multiple pages, not limited to just the community page. Perhaps we could explore identifying a CSS fix that enhances the rendering of pages with wider elements. This way, we can efficiently resolve the problem and improve the overall user experience.
/area web-development
I tried making some changes on the source, A simple overflow - x in the right places seems to solve the problem. ! However, I'd often come across such problems when the scaling isn't managed properly. I'm sure there are better ways to get this problem fixed.
I see, In that case I'll try to find anymore pages that might be having the same issue.
Similar issue is present in case study


Question: | Golang marker | OpenAPI extension | Accepted values | Description | Introduced in | |---|---|---|---|---| <del> | `//+listType` | `x-kubernetes-list-type` | `atomic`/`set`/`map` | Applicable to lists. `atomic` and `set` apply to lists with scalar elements only. `map` applies to lists of nested types only. If configured as `atomic`, the entire list is replaced during merge; a single manager manages the list as a whole at any one time. If `granular`, different managers can manage entries separately. | 1.16          | </del> <ins> | `//+listType` | `x-kubernetes-list-type` | `atomic`/`set`/`map` | Applicable to lists. `atomic` and `set` apply to lists with scalar elements only. `map` applies to lists of nested types only. If configured as `atomic`, the entire list is replaced during merge; a single manager manages the list as a whole at any one time. If `set` or `map`, different managers can manage entries separately. | 1.16          | </ins> | `//+listMapKey` | `x-kubernetes-list-map-keys` | Slice of map keys that uniquely identify entries for example `["port", "protocol"]` | Only applicable when `+listType=map`. A slice of strings whose values in combination must uniquely identify list entries. While there can be multiple keys, `listMapKey` is singular because keys need to be specified individually in the Go type. | 1.16 | | `//+mapType` | `x-kubernetes-map-type` | `atomic`/`granular` | Applicable to maps. `atomic` means that the map can only be entirely replaced by a single manager. `granular` means that the map supports separate managers updating individual fields. | 1.17 | | `//+structType` | `x-kubernetes-map-type` | `atomic`/`granular` | Applicable to structs; otherwise same usage and OpenAPI annotation as `//+mapType`.| 1.17 |

In the section, the annotation documentation gives possible values of //, but the description describes and . There's a more-detailed behaviour description in which might be useful to draw from.
Good point, let me send a quick fix.
/triage accepted
/kind cleanup /language en


Question: title: "Overview" weight: 20 description: Get a high-level outline of Kubernetes and the components it is built from. <ins> sitemap: priority: 0.9 </ins> ---

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be The Kubernetes website generates a sitemap. Let's highlight specific pages using priority weighting: (and any localization that copies the priority weight) (and any localization that copies the priority weight) (and any localization that copies the priority weight) Why is this needed: Making this change helps search engines understand what pages the Kubernetes project sees as important. Comments /kind feature


Question: id: statefulset name: StatefulSet full-link: /docs/concepts/workloads/controllers/statefulset/ <del> aka: - PetSet </del> related: - deployment - pod

This is a... [ ] Feature Request [x] Bug Report Problem: There is a link missing that was supposed to be in section Differences between alpha PetSets and beta StatefulSets under it in the second point StatefulSet guards against split brain which ends with following text TODO: Link to doc about fencing. Change it and add the link that is required. Proposed Solution: Add link which is missing Page to Update:
PetSet has long been replaced by StatefuSet and StatefulSet has been GA'ed in 1.9. So we are going to remove this page to avoid further confusion.


Question: If you want to use minikube again to learn more about Kubernetes, you don't need to delete it. <ins> ## Conclusion This page covered the basic aspects to get a minikube cluster up and running. You are now ready to deploy applications. </ins> ## {{% heading "whatsnext" %}}

The basic tutorial starts at , which leads to , which leads to Since one does not know what is exactly meant with in the second site, one does also the deploy step. Because it's not clearly said "" or so. Then you go back to the second site, to read all through, only to do create/expose again (with little extras), only to go back to the first site and do create/expose again. Back to the first site on step 2, pretty much at the beginning it says: But I did it already 2 times in the current tutorial flow. Now I have 3 deployments, still in the basics deployment tutorial. Its like, there are 3 tutorials to do, which are the same, but also not the same. Instead of leading the user back and forth, I suggest to do one single sufficient tutorial.
Page related to issue: /language en
/retitle Consolidate and streamline Kubernetes basics tutorials to avoid redundancy and confusion
/sig docs
the three tutorial have different purpose. What I understood the purpose of having these as three.: Yes, Its leading to back and forth of the users but I think First link showing the general functionality of the cluster's component. It Should have the link To and then link will be helpful for readers to traverse. This sequence is lacking in the page. Thats the reason why i thing its taking user back and forth several times .
There was a previous attempt at de-duplicating, but it probably has to be done again - after the removal of the Katacoda tutorials, which were using a hosted setup of to run through the tutorials "inline" with the docs
I'll change the title to frame this as a bug. /retitle Multiple, similar Kubernetes basics tutorials that cause confusion /kind bug /triage accepted /priority important-longterm
/lifecycle frozen
Yes You are right! +1
/assign
I will start working on this.


Question: `Pending`   | The Pod has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run. This includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network. `Running`   | The Pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. `Succeeded` | All containers in the Pod have terminated in success, and will not be restarted. <del> `Failed`    | All containers in the Pod have terminated, and at least one container has terminated in failure. That is, the container either exited with non-zero status or was terminated by the system. </del> <ins> `Failed`    | All containers in the Pod have terminated, and at least one container has terminated in failure. That is, the container either exited with non-zero status or was terminated by the system, and is not set for automatic restarting. </ins> `Unknown`   | For some reason the state of the Pod could not be obtained. This phase typically occurs due to an error in communicating with the node where the Pod should be running. {{< note >}}

This is a Bug Report Problem: An explanation in the Pod phase section is misleading. A pod doesn't have a state until it is configured not to be restarted. If the is not set, the pod will still be in the state. Proposed Solution: All containers in the Pod have terminated, and at least one container has terminated in failure. That is, the container either exited with non-zero status or was terminated by the system, and will not be restarted. Page to Update: Kubernetes Version: v1.30.1 I'll be happy to work on it if it's accepted.
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/language en /sig docs
/sig node
so long as you're confident your understanding is correct, you don't need anyone to tell you it's OK to start work. Kubernetes is open source.
That’s true, thanks. /assign


Question: /docs/reference/generated/kube-scheduler/     /docs/reference/command-line-tools-reference/kube-scheduler/ 301 /docs/reference/generated/kubectl/kubectl-options/     /docs/reference/kubectl/kubectl/ 301 /docs/reference/generated/kubectl/kubectl/     /docs/reference/generated/kubectl/kubectl-commands/ 301 <del> /docs/reference/generated/kubectl/kubectl/kubectl_*.md    /docs/reference/generated/kubectl/kubectl-commands#:splat 301 </del> <ins> /docs/reference/generated/kubectl/kubectl/kubectl_*    /docs/reference/generated/kubectl/kubectl-commands#:splat 301 </ins> /docs/reference/glossary/maintainer/    /docs/reference/glossary/approver/ 301 /docs/reference/kubectl/kubectl-cmds/     /docs/reference/generated/kubectl/kubectl-commands/ 301! <del> /docs/reference/kubectl/kubectl/kubectl_*.md    /docs/reference/generated/kubectl/kubectl-commands#:splat 301 </del> <ins> /docs/reference/kubectl/kubectl/kubectl_*    /docs/reference/generated/kubectl/kubectl-commands#:splat 301 </ins> /docs/reference/scheduling/profiles/    /docs/reference/scheduling/config/#profiles 301 /docs/reference/generated/kubernetes-api/v1.15/    https://v1-15.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/  301

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: In , There are some redirection rules with wildcard(``). Expectation: Netlify redirects to Reality: gives Instead, Netlify redirects (literally) to (literally) Expectation: Netlify redirects to Reality: Netlify redirects to (with trailing slash) which is not a valid anchor URL (because of the trailing slash). Proposed Solution: File to Update:  <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--<!--Additional Information:--
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to give this a try /assign
how can I test these redirect things after I made the changes? Do I have to make a PR to get the netlify link to test the things?
Are you working on this issue?
Ya I was trying but couldn't figure out how can I test these redirects after making the changes
Thanks, I will unassign myself. Please consider commenting before. It helps others to know about the progress of issue. /unassign
/assign
It was my bad.
I think the problem is with the link itself. Here may be is the splat. Maybe that's why it's unable to redirect. If I change the redirect to something like below without the .md it works fine And for this link , the splat is get/ and hence it redirects to This can be resolve if we remove the trailing slash from this link So, where are all these links present if we want to modify the link itself?
What a great research! I think so.


Question: /docs/reference/generated/kube-scheduler/     /docs/reference/command-line-tools-reference/kube-scheduler/ 301 /docs/reference/generated/kubectl/kubectl-options/     /docs/reference/kubectl/kubectl/ 301 /docs/reference/generated/kubectl/kubectl/     /docs/reference/generated/kubectl/kubectl-commands/ 301 <del> /docs/reference/generated/kubectl/kubectl/kubectl_*.md    /docs/reference/generated/kubectl/kubectl-commands#:splat 301 </del> <ins> /docs/reference/generated/kubectl/kubectl/kubectl_*    /docs/reference/generated/kubectl/kubectl-commands#:splat 301 </ins> /docs/reference/glossary/maintainer/    /docs/reference/glossary/approver/ 301 /docs/reference/kubectl/kubectl-cmds/     /docs/reference/generated/kubectl/kubectl-commands/ 301! <del> /docs/reference/kubectl/kubectl/kubectl_*.md    /docs/reference/generated/kubectl/kubectl-commands#:splat 301 </del> <ins> /docs/reference/kubectl/kubectl/kubectl_*    /docs/reference/generated/kubectl/kubectl-commands#:splat 301 </ins> /docs/reference/scheduling/profiles/    /docs/reference/scheduling/config/#profiles 301 /docs/reference/generated/kubernetes-api/v1.15/    https://v1-15.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/  301

To do tests, you should changes on  file, a PR, and the redirection rules on the deploy preview: https://deploy-preview-xxxxx--kubernetes-io-main- And regarding your research, I suggest to make changes like these: and test on the deploy preview of the PR you will post. I did some searching and found these: content/en/blog/posts/2016-08-00-Kubernetes-Namespaces-Use-Cases- content/en/blog/posts/2016-12-00-Statefulset-Run-Scale-Stateful-Applications-In-
I have made the PR. Let me know if I have to do some more changes or anything


Question: --> ### 仅在某些节点上运行 Pod <del> 如果指定了 `.spec.template.spec.nodeSelector`，DaemonSet Controller 将在能够与 [Node Selector](/zh/docs/concepts/scheduling-eviction/assign-pod-node/) 匹配的节点上创建 Pod。类似这种情况，可以指定 `.spec.template.spec.affinity`，然后 DaemonSet Controller 将在能够与 [node Affinity](/zh/docs/concepts/scheduling-eviction/assign-pod-node/) 匹配的节点上创建 Pod。 </del> <ins> 如果指定了 `.spec.template.spec.nodeSelector`，DaemonSet 控制器将在能够与 [Node 选择算符](/zh/docs/concepts/scheduling-eviction/assign-pod-node/) 匹配的节点上创建 Pod。 类似这种情况，可以指定 `.spec.template.spec.affinity`，之后 DaemonSet 控制器 将在能够与[节点亲和性](/zh/docs/concepts/scheduling-eviction/assign-pod-node/) 匹配的节点上创建 Pod。 </ins> 如果根本就没有指定，则 DaemonSet Controller 将在所有节点上创建 Pod。 <!--

“ DaemonSet spec.template.“ “”“”
Mind propose a fix?
/triage accepted
/language zh (I presume)
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /language zh (I presume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes,this is chinese languge,“”change to“”
/assign


Question: sudo mv ./kubectl /usr/local/bin/kubectl ``` <del> ## Extract kubectl from Release .tar.gz or Compiled Source If you downloaded a pre-compiled [release](https://github.com/kubernetes/kubernetes/releases), kubectl will be under `platforms/<os>/<arch>` from the tar bundle. If you compiled Kubernetes from source, kubectl should be either under `_output/local/bin/<os>/<arch>` or `_output/dockerized/bin/<os>/<arch>`. Copy or move kubectl into a directory already in your PATH (e.g. `/usr/local/bin`). For example: ```shell # OS X sudo cp platforms/darwin/amd64/kubectl /usr/local/bin/kubectl # Linux sudo cp platforms/linux/amd64/kubectl /usr/local/bin/kubectl ``` Next make it executable with the following command: ```shell sudo chmod +x /usr/local/bin/kubectl ``` The kubectl binary doesn't have to be installed to be executable, but the rest of the walkthrough will assume that it's in your PATH. If you prefer not to copy kubectl, you need to ensure it is in your path: ```shell # OS X export PATH=<path/to/kubernetes-directory>/platforms/darwin/amd64:$PATH # Linux export PATH=<path/to/kubernetes-directory>/platforms/linux/amd64:$PATH ``` </del> ## Download as part of the Google Cloud SDK kubectl can be installed as part of the Google Cloud SDK:


Question: <ins> apiVersion: v1 kind: Pod metadata: name: dapi-test-pod spec: containers: - name: test-container image: k8s.gcr.io/busybox command: [ "/bin/sh", "-c", "env" ] env: # 環境変数を定義します - name: SPECIAL_LEVEL_KEY valueFrom: configMapKeyRef: # SPECIAL_LEVEL_KEYに割り当てる値をConfigMapが保持します name: special-config # 値に紐付けるキーを指定します key: special.how restartPolicy: Never </ins>

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be Translate tasks/configure-pod-container/configure-pod-configmap/ into Japanese. Why is this needed This page has not been translated. Comments /language ja
/assign
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: {{< /table >}} {{< note >}} <del> <!-- Generators other than `run-pod/v1` are deprecated. -->不推荐使用 `run-pod/v1` 以外的其他生成器。 </del> <ins> <!--Generators other than `run-pod/v1` are deprecated.--> 不推荐使用 `run-pod/v1` 以外的其他生成器. </ins> {{< /note >}} <!--

there is an useless "" character on
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: [previous_patches] other = "Poprawki:" <ins> [release_date_after] other = ")" [release_date_before] other = "(wydano: " # See https://gohugo.io/functions/format/#gos-layout-string # Use a suitable format for your locale [release_date_format] other = "2006-01-02" [release_cherry_pick_deadline] other = "Termin poprawek Cherry Pick" # Deprecated. Planned for removal in a future release. # Use [release_full_details_initial_text] instead. [release_complete] other = "Pełny" [release_end_of_life_date] other = "Termin End Of Life" # Replace [release_complete] with [release_full_details_initial_text] [release_full_details_initial_text] other = "Pełny" [release_information_navbar] other = "Informacje o wydaniu" [release_minor_version] other = "Podwersja (Minor)" [release_info_next_patch] other = "Następne wydanie zawierające poprawki to **%s**." # Localization note: You can use Markdown here. # The three placeholders (in order) are: #   Kubernetes minor version #   maintenance mode date #   end of life date # # Keep this order. It is OK to use more than one sentence, and it's also OK to change the # tense of the text so long as the meaning is clear. [release_info_eol] other = "**%s** od **%s** wchodzi w tryb minimalnego utrzymania (maintenance mode), a całkowity koniec wsparcia to **%s**." [release_note] other = "Uwagi" [release_schedule] other = "Harmonogram" [release_target_date] other = "Planowany termin" [release_changelog] other = "Rejestr zmian" </ins> [seealso_heading] other = "Zobacz też"

What would you like to be This issue is a follow-up task from displays some English text: ! To replace this text with localized strings, you can update with from . Specifically, these:
/language pl
/sig docs
/triage accepted