ssh k8s1_controlPlane_1
sudo su
service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; disabled; vendor preset: enabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: inactive (dead)
Docs: https://kubernetes.io/docs/
Feb 03 18:20:40 ip-10-2-4-149 kubelet[5330]: W0203 18:20:40.973916 5330 reflector.go:539] vendor/k8s.io/client-go/informers/factory.go:159: failed to list *v1.Node: Get "htt>
Feb 03 18:20:40 ip-10-2-4-149 kubelet[5330]: E0203 18:20:40.973966 5330 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.Node: failed >
Feb 03 18:20:41 ip-10-2-4-149 kubelet[5330]: E0203 18:20:41.373569 5330 eviction_manager.go:282] "Eviction manager: failed to get summary stats" err="failed to get node info>
Feb 03 18:20:41 ip-10-2-4-149 kubelet[5330]: W0203 18:20:41.466839 5330 reflector.go:539] vendor/k8s.io/client-go/informers/factory.go:159: failed to list *v1.RuntimeClass: >
Feb 03 18:20:41 ip-10-2-4-149 kubelet[5330]: E0203 18:20:41.466881 5330 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:159: Failed to watch *v1.RuntimeClass:>
Feb 03 18:20:41 ip-10-2-4-149 kubelet[5330]: E0203 18:20:41.613869 5330 event.go:355] "Unable to write event (may retry after sleeping)" err="Post \"https://10.2.4.149:6443/>
Feb 03 18:20:42 ip-10-2-4-149 kubelet[5330]: I0203 18:20:42.243248 5330 dynamic_cafile_content.go:171] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/pki>
Feb 03 18:20:42 ip-10-2-4-149 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Feb 03 18:20:42 ip-10-2-4-149 systemd[1]: kubelet.service: Succeeded.
Feb 03 18:20:42 ip-10-2-4-149 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
status is inactive (dead)
start kubelet
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2024-02-03 18:41:35 UTC; 1s ago
Docs: https://kubernetes.io/docs/
Main PID: 5761 (kubelet)
Tasks: 11 (limit: 4597)
Memory: 25.3M
CGroup: /system.slice/kubelet.service
└─5761 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var
now status is active (running)
check connection to k8s
# k get ns
E0203 18:48:25.894982 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
E0203 18:48:25.895314 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
E0203 18:48:25.896459 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
E0203 18:48:25.896778 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
E0203 18:48:25.898099 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
The connection to the server 10.2.4.149:6443 was refused - did you specify the right host or port?
kube api don't work .
check kube-api container
# crictl | grep api
kube-api is a static pod with starts by kubelet from manifest /etc/kubernetes/manifests/kube-apiserver.yaml
check kubelet logs
journalctl -u kubelet | grep 'kube-apiserver'
Feb 03 18:20:31 ip-10-2-4-149 kubelet[4732]: E0203 18:20:31.063086 4732 file.go:108] "Unable to process watch event" err="can't process config file \"/etc/kubernetes/manifests/kube-apiserver.yaml\": /etc/kubernetes/manifests/kube-apiserver.yaml: couldn't parse as pod(no kind \"PoD\" is registered for version \"v1\" in scheme \"pkg/api/legacyscheme/scheme.go:30\"), please check config file"
error pod(no kind \"PoD\"
replace PoD to Pod in /etc/kubernetes/manifests/kube-apiserver.yaml and restart kubelet .
service kubelet restart
check connection to k8s
# k get ns
E0203 18:48:25.894982 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
E0203 18:48:25.895314 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
E0203 18:48:25.896459 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
E0203 18:48:25.896778 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
E0203 18:48:25.898099 5900 memcache.go:265] couldn't get current server API group list: Get "https://10.2.4.149:6443/api?timeout=32s": dial tcp 10.2.4.149:6443: connect: connection refused
The connection to the server 10.2.4.149:6443 was refused - did you specify the right host or port?
check kubelet logs
journalctl -u kubelet | grep 'kube-apiserver'
Feb 03 19:11:55 ip-10-2-4-149 kubelet[6661]: E0203 19:11:55.449778 6661 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-apiserver pod=kube-apiserver-ip-10-2-4-149_kube-system(b1982a51593e867c8f49a556991190ef)\"" pod="kube-system/kube-apiserver-ip-10-2-4-149" podUID="b1982a51593e867c8f49a556991190ef"
don't work it .
check container logs
rm -rf /var/log/containers/*
service kubelet restart
ls /var/log/containers/
kube-apiserver-ip-10-2-4-149_kube-system_kube-apiserver-a686706a02ecb891bd5f38eb467a231eb3ec82fc3043fca9ae292a8f4248d09a.log
# cat /var/log/containers/kube-apiserver-ip-10-2-4-149_kube-system_kube-apiserver-80029fa2ce0099c1537c155f9f9e05ad9f95bfd7b98a10fb9ab1f7afe0ad3a91.log
2024-02-03T19:15:16.153152153Z stderr F Error: unknown flag: --new-option2
Error: unknown flag:--new-option2
vim /etc/kubernetes/manifests/kube-apiserver.yaml
delete line with --new-option2
and restart kubelet
service kubelet restart
check connection to kube-api
k get ns
NAME STATUS AGE
default Active 60m
kube-node-lease Active 60m
kube-public Active 60m
kube-system Active 60m
exit to work pc
exit
exit
check connection from work pc
k get ns
NAME STATUS AGE
default Active 60m
kube-node-lease Active 60m
kube-public Active 60m
kube-system Active 60m
It is done .