Some people think the Chinese government should stick to their zero-covid policy. However, others believe abandoning the lockdown is the best choice.
Discuss both these views and give your own opinion.

While many people are justifiably resistant to the idea of abandoning the zero-covid policy, others believe that without lockdown is fundamental for recovering from the economic recession and emotional depression of people in the last three years. In my opinion, the zero-covid policy is not a sustainable way to solve the problem and should not continue after omicron becomes the main global pandemic.

Those who argue in favour of non-change point out that their vulnerable family members are in a safe environment when the lockdown continues,
This is evident as children and seniors have weaker immune systems than young adults and are more susceptible to covid virus. The same fact applies when a person infects with the covid virus, which can lead to long-coivd, such as loss of smell sense, tiresome and fatigue. One further concern of this relates to the mortality rate. It will cause millions of deaths if China reopens, which referred to Hong Kong's mortality rate after it has reopened.

However, the lockdown is not a sustainable way to deal with covid virus. This fact is illustrated in China's economic growth. The GDP data of Shanghai released by the China Bureau of Statistics shows shanghai had gone through a significant plummet in the first half year of 2022 because the whole city was in a three-month lockdown. It is more likely that the economy would not have suffered a vast strike if the government had implemented a more irrational policy. Furthermore, the lockdown caused enormous secondary disasters. One hospital refused to aid a nurse with asthma because she didn't take a nuclei test in 48 hours. Also, ambulances take long hours to respond due to roadblocks. This is also the case when it comes to other cities. Residents living in a compound in Chengdu were forbidden to go downstairs during an earthquake. By abandoning the lockdown and replacing it with a more rational procedure, the economy has a better chance of reviving and secondary disasters will diminish.

In conclusion, even though abandoning the zero-covid policy can be a catalyst for excess death, sticking to endless nuclei tests and keeping people at home should not be regarded as a long-term solution to address the problem, especially after so many unnecessary tragedies. The government should seek a sustainable method like opening the lockdown gradually and discreetly but not consistently ignoring people's voices.

Recently I read Memoirs of a Geisha by Arthur Golden. I knew a movie filmed from the book that was the same name as the book, but I hadn't watched it before I finished the book because I didn't want to be spoiled by the movie. I read the book because I had searched the book recommendations for English B1 Level, and some websites recommended it. I finished the reading after I bought the book from Amazon about two months.

The book is about a little girl named Chiyo who was sold to an Okiya and eventually became a famous geisha. The story of the background started in the 1920s and ended in the 1950s, I think. It describes the lifestyle of geisha, the scenic Gion district in Kyoto and the crucial of world war 2.

After I read it, I thought it was a suitable book for the B1 level English learner. It has limited advanced vocabulary, and the sentence is easy to read because the author wrote this book in the first-person's perspective. Also, the culture of the book is relatable to me since the book's background is based on Japanese culture. Secondly, the author wrote this book from a view of a western. It is novel.

Furthermore, it vividly depicted Gion during that period of time, and I was fascinated by the geisha culture, so I watched the movie of Memoirs of a Geisha and some Japanese TV series, such as Cooking for the Maiko house. But I was somehow disappointed with the movie since I thought it didn't describe the whole story coherently, although the scenery of the movie was really breathtaking. Also, the illustration of wartime is unforgettable. Due to the war, food was a shortage, the geisha had to work in factories, and a lot of people died. I used to think the vicious of the war was mainly borne by China which was invaded, and Japan only bombed with two warheads. But the Japanese also in a really hard time during world war 2, especially in the late war.

However, the relationship between the chairman and Sayuri (the art name of Chiyo after she became an apprentice of geisha) confused me. I can understand Sayuri has an attachment to the chairman as she didn't have any connected one since her parents died and her sister was missing; also, she was abused by senior geisha Hatasumomo, and the only kind to her was from the chairman. But it's really weird to me that the chairman, who is a middle-aged man, has a feeling for Sayuri, who was a minor. This is not even the most confusing thing. The mizuage ceremony really makes people uncomfortable, and kind of lets geisha become a prostitute. Yet, we can't use the perspective of modern society to judge history. In the 1920s, it was hard for peasants to live and maybe becoming a geisha was the best way out for a girl from a lowly background.

On the whole, this book depicted a beautiful Kyoto and unveiled the mysterious life of geisha to laypeople. I recommend this book for the English learner on the B1 level and interested in Japanese culture. After reading this book, I learnt many vocabularies, and I think I may have improved my English reading skill.

Off-topics, since China has alleviated its covid lockdown policy, maybe it's a good choice to have a trip to Kyoto.

0x00 Introduction

In part two of this series, I wrote about deploying Cilium with BGP mode in the Kubernetes Cluster, which worked fine in the internal cluster. Today, I will write about how to make the cluster pod IP routable from the outside, such as the on-premises environment.

In this article, I will use a Linux virtual machine installed BIRD as a software router that runs as a BGP peer so that the on-premises and the cluster pod IP are routable.

0x0000 The Environment

Here is the environment information of the software router:

• System Distribution: Ubuntu 22.04.1 LTS
• IP Address: 192.168.56.10
• BIRD version: 1.6.8
• kube-router: v1.5.3

0x01 Deployment

0x0101 Configure Kube-router

To use an external BGP peer, we have to change some configurations of kube-router. You can download the original YAML of kube-router from https://github.com/cloudnativelabs/kube-router/blob/v1.5.3/daemonset/generic-kuberouter-only-advertise-routes.yaml , and here is the snippet of the configurations which I modified.

...
containers:
      - name: kube-router
        image: docker.io/cloudnativelabs/kube-router
        imagePullPolicy: Always
        args:
        - "--run-router=true"
        - "--run-firewall=false"
        - "--run-service-proxy=false"
        - "--bgp-graceful-restart=true"
        - "--enable-cni=false"
        - "--enable-ibgp=true"
        - "--enable-overlay=false"
        - "--peer-router-ips=192.168.56.10"
        - "--peer-router-asns=65000"
        - "--cluster-asn=65001"
        - "--advertise-cluster-ip=true"
        - "--advertise-external-ip=true"
        - "--advertise-loadbalancer-ip=true"
...

Some options need to be noticed if you will use external BGP. The peer-router-ips are the IP addresses of external BPGs. The peer-router-asns are ASN numbers of the BGP peer to which cluster nodes will advertise cluster IP and node's pod CIDR. The advertise-cluster-ip means add Cluster IP of the service to the RIB (Routing Information Base that contains the routing information maintained by that router) so that it gets advertises to the BGP peers. Now we apply the configuration.

0x0102 Deploy External BGP Peer

Now we need to install an external BGP Peer on 192.168.56.10 .

sudo apt -y install bird-bgp

Here is the configuration of BIRD.

cat /etc/bird/bird.conf
protocol kernel {
	scan time 60;
	import none;
	export all;   # Actually insert routes into the kernel routing table
}

# The Device protocol is not a real routing protocol. It doesn't generate any
# routes and it only serves as a module for getting information about network
# interfaces from the kernel.
protocol device {
	scan time 60;

}

protocol bgp k8smaster0 {
	import all;
        local as 65000;
        neighbor 192.168.56.4 as 65001;
}

protocol bgp k8sslave0 {
	import all;
        local as 65000;
        neighbor 192.168.56.5 as 65001;
}

The above snippet means we receive the BGP propagations from Kubernetes nodes. The ASN needs to match the ASN we configured in kube-router. After the bird.conf has been configured, we can use the following commands to start the BIRD process and check the routes:

sysadmin@ubuntu:~$ sudo invoke-rc.d bird start

sysadmin@ubuntu:~$ sudo birdc show route
BIRD 1.6.8 ready.
10.98.28.103/32    via 192.168.56.4 on enp0s8 [k8smaster0 14:40:34] * (100) [AS65001i]
                   via 192.168.56.5 on enp0s8 [k8sslave0 14:40:33] (100) [AS65001i]
10.96.229.67/32    via 192.168.56.4 on enp0s8 [k8smaster0 14:40:34] * (100) [AS65001i]
                   via 192.168.56.5 on enp0s8 [k8sslave0 14:40:33] (100) [AS65001i]
10.96.0.1/32       via 192.168.56.4 on enp0s8 [k8smaster0 14:40:34] * (100) [AS65001i]
                   via 192.168.56.5 on enp0s8 [k8sslave0 14:40:33] (100) [AS65001i]
10.96.0.10/32      via 192.168.56.4 on enp0s8 [k8smaster0 14:40:34] * (100) [AS65001i]
                   via 192.168.56.5 on enp0s8 [k8sslave0 14:40:33] (100) [AS65001i]
10.112.0.0/24      via 192.168.56.4 on enp0s8 [k8smaster0 14:40:34] * (100) [AS65001i]
10.112.1.0/24      via 192.168.56.5 on enp0s8 [k8sslave0 14:40:33] * (100) [AS65001i]

The result shows the cluster service IP and Cluster pod IP have been propagated to the external BGP peer. Now we can access the pod IP from 192.168.56.10 directly.

## Get the Nginx Pod IP 
root@k8smaster0:~/kube-router# kubectl get pods -A -l app=nginx -o wide
NAMESPACE   NAME                                READY   STATUS    RESTARTS      AGE   IP             NODE        NOMINATED NODE   READINESS GATES
default     nginx-deployment-7fb96c846b-7jhtt   1/1     Running   2 (67m ago)   47h   10.112.1.194   k8sslave0   <none>           <none>
default     nginx-deployment-7fb96c846b-lqpgf   1/1     Running   2 (67m ago)   47h   10.112.1.90    k8sslave0   <none>           <none>

## Curl from 192.168.56.10
sysadmin@ubuntu:~$ curl -I 10.112.1.194
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Wed, 11 Jan 2023 06:49:32 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 04 Dec 2018 14:44:49 GMT
Connection: keep-alive
ETag: "5c0692e1-264"
Accept-Ranges: bytes

Then we tried to access the cluster service IP.

## Get service IP of nginx
root@k8smaster0:~/kube-router# kubectl get service -A -l app=nginx -o wide
NAMESPACE   NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE   SELECTOR
default     nginx   ClusterIP   10.96.229.67   <none>        80/TCP    47h   app=nginx

## Curl from 192.168.56.10
sysadmin@ubuntu:~$ curl 10.96.229.67 -v
*   Trying 10.96.229.67:80...

But the result shows we still can’t access cluster service IP directly. It’s strange; let's check the monitor metrics of Cilium to see if or not the traffic from external BGP peer VM are normal.

## Step 1. Find the Nginx service managed by which daemonset of Cilum

root@k8smaster0:~/kube-router# kubectl -n kube-system exec cilium-2xlxh -- cilium service list
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)
ID   Frontend           Service Type   Backend
1    10.96.229.67:80    ClusterIP      1 => 10.112.1.194:80 (active)
                                       2 => 10.112.1.90:80 (active)

## Step 2. Use Cilium daemonset located on the node of CIDR 10.112.1.0/24
## to check traffic

root@k8smaster0:~/kube-router# kubectl -n kube-system exec -ti cilium-hcr6r -- cilium monitor --type drop
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)
Listening for events on 4 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit

## Step 3. Curl from 192.168.56.10

sysadmin@ubuntu:~$ curl 10.96.229.67 -v
*   Trying 10.96.229.67:80...

## Step 4. Check the stdout of cilium monitor and we can see the TCP packet has been dropped

root@k8smaster0:~/kube-router# kubectl -n kube-system exec -ti cilium-hcr6r -- cilium monitor --type drop
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)
Listening for events on 4 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
xx drop (Is a ClusterIP) flow 0x0 to endpoint 0, file bpf_host.c line 665, , identity world->unknown: 192.168.56.10:33802 -> 10.112.1.90:80 tcp SYN
xx drop (Is a ClusterIP) flow 0x0 to endpoint 0, file bpf_host.c line 665, , identity world->unknown: 192.168.56.10:33802 -> 10.112.1.90:80 tcp SYN

I read the official document, and it seems to be because of the option of bpf-lb-external-clusterip. The official document shows this option enables external access to ClusterIP services, and by default, it is false. Now we know why the curl failed, so we can modify it to true:

root@k8smaster0:~/kube-router# cilium config view | grep -i bpf-lb-external-clusterip
bpf-lb-external-clusterip                      false

root@k8smaster0:~/kube-router# cilium config set bpf-lb-external-clusterip true
✨ Patching ConfigMap cilium-config with bpf-lb-external-clusterip=true...
♻️  Restarted Cilium pods

## curl from 192.168.56.10
sysadmin@ubuntu:~$ curl 10.96.229.67 -I
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Wed, 11 Jan 2023 07:26:35 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 04 Dec 2018 14:44:49 GMT
Connection: keep-alive
ETag: "5c0692e1-264"
Accept-Ranges: bytes

Finally, we can access the cluster service from outside. We need to do the last step to test if external IP is routable from pods.

## Create a busybox deployment 
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

## Ping from pod of busybox
root@k8smaster0:~/app# kubectl exec -it busybox -- ping -c 3 192.168.56.10
PING 192.168.56.10 (192.168.56.10): 56 data bytes
64 bytes from 192.168.56.10: seq=0 ttl=62 time=3.074 ms
64 bytes from 192.168.56.10: seq=1 ttl=62 time=0.566 ms
64 bytes from 192.168.56.10: seq=2 ttl=62 time=0.927 ms

--- 192.168.56.10 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.566/1.522/3.074 ms

0x02 Summary

By now, the Kubernetes cluster and external BGP router are routable from each other. You only need to configure some static routes on your local environment and external BGP router if you want the K8s cluster and the local environment to be routable to each other.

0x00 Introduction

In my previous blog of this series, I wrote about how to prepare the environment for deploying Kubernetes with BGP Network. Today, I am going to write about the process of how I deployed.

0x0000 The Environment

Before We start, here are the tools and versions I used to deploy Kubernetes.

• kubeadm: v1.25.4
• cilium: v1.12.2
• helm: v3.10.2
• kube-router: v1.5.3

Here is my configuration for kubeadm, and I will explain the usage of the configurations I used.

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd        # Specify cgroup driver of kubelet,recommend systemd
---
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.56.4   # the address for apiserver listening
  bindPort: 6443
    #nodeRegistration:
    #  criSocket: unix:///var/run/containerd/containerd.sock
    #  imagePullPolicy: IfNotPresent
    #  name: k8smaster0
    #  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.25.4
# Recommand to set up controlPlane if you want to add apiserver for HA,
# and the cluster-endpoint is to use for the DNS parsed to the apiservers.
controlPlaneEndpoint: "cluster-endpoint:6443"  
networking:
  dnsDomain: cluster.local
# Specify the service CIDR range
  serviceSubnet: 10.96.0.0/12
# Specify the pod CIDR range
  podSubnet: 10.112.0.0/12
scheduler: {}

Usually, We can start to deploy Kubernetes using kubeadm, but unfortunately, as a Chinese developer behind the GFW, there are some obstacle to pulling the images. So I wrote the scripts to pull images from the domestic image mirror sources, and the scripts will rename images to the name that kubeadm used.

After we install kubeadm, we can use kubeadm config images list to show the images require for kubeadm. Here is my command execution result:

root@k8smaster0:~# kubeadm config images list
I1221 07:38:49.772660    1024 version.go:256] remote version is much newer: v1.26.0; falling back to: stable-1.25
registry.k8s.io/kube-apiserver:v1.25.5
registry.k8s.io/kube-controller-manager:v1.25.5
registry.k8s.io/kube-scheduler:v1.25.5
registry.k8s.io/kube-proxy:v1.25.5
registry.k8s.io/pause:3.8
registry.k8s.io/etcd:3.5.5-0
registry.k8s.io/coredns/coredns:v1.9.3

As you know, we have already installed containerd as Kubernetes runtime in the previous article. However, the usage of containerd is somewhat different from docker because containerd uses ctr as a command line tool, and also, containerd has a namespace concept. If we use ctr command without specifying namespaces, it will use the default namespace. However, the Kubernetes CRI use k8s.io namespace. So if we want to pre-pull images, we need to put them into the correct namespace; otherwise, Kubernetes can not find the resources. Here are my commands to pre-pull images:

#Download the scripts for pulling images from China.
wget https://raw.githubusercontent.com/N0mansky/docker_wrapper/master/crt_wrapper.py
chmod +x crt_wrapper.py

# Pullings images to k8s.io namespace by using scripts
./crt_wrapper.py pull registry.k8s.io/xxxxxx

# After we have pulled the images, we can check by using the command
ctr -n k8s.io image ls -q

# Or we can use the crictl
root@k8sslave0:~# crictl image ls
IMAGE                                                         TAG                 IMAGE ID            SIZE
docker.io/cloudnativelabs/kube-router                         latest              a5e6dc4b76a3f       45MB
docker.io/library/busybox                                     1.28                8c811b4aec35f       728kB
docker.io/library/nginx                                       1.14.2              295c7be079025       44.7MB
quay.io/cilium/cilium                                         v1.12.2             743cf6b60787d       167MB
quay.io/cilium/cilium                                         v1.12.4             b7257a8403c50       167MB
quay.io/cilium/operator-generic                               v1.12.2             1f3c9d6876457       18.9MB
quay.io/cilium/operator-generic                               v1.12.4             ca5b3c9580cb3       18.9MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns   v1.9.3              5185b96f0becf       14.8MB
registry.k8s.io/coredns/coredns                               v1.9.3              5185b96f0becf       14.8MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause     3.8                 4873874c08efc       311kB
registry.k8s.io/pause                                         3.8                 4873874c08efc       311kB
registry.k8s.io/kube-proxy                                    v1.25.4             2c2bc18642790       20.3MB

0x01 Deployment

0x0100 Create Cluster

After all the pre-work has been done, it is time to create a cluster. First, I initiated the master with the command:

# I specified to skip kube-proxy, because I will use cilium to replace it.
kubeadm init --config kubeadm-config.yml --skip-phases=addon/kube-proxy

# After the above command had finished, the following results were printed.
...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:7337839717eb93c80bad2157ecbed814c389f8fa843c2d6b41e305e763751107 \
	--control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:7337839717eb93c80bad2157ecbed814c389f8fa843c2d6b41e305e763751107

After the master node had initialized successfully, I used the following command to add a worker node:

# Executing the join command on worker node
kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:7337839717eb93c80bad2157ecbed814c389f8fa843c2d6b41e305e763751107

# After the above command had finished, the following result were printed.
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

# And the nodes all were in the ready status
root@k8smaster0:~# kubectl get nodes -A
NAME         STATUS   ROLES           AGE     VERSION
k8smaster0   Ready    control-plane   5m42s   v1.25.4
k8sslave0    Ready    <none>          66s     v1.25.4

Now we can use kubectl get pods -A to check the state of pods. We can see the result didn’t show the pods of kube-proxy, and the Coredns are in ContainerCreating status. That is because we didn’t install kube-proxy and a network add-on, and the Coredns containers need a pod network add-on for creation.

root@k8smaster0:~# kubectl get pods -A
NAMESPACE     NAME                                 READY   STATUS              RESTARTS           AGE
kube-system   coredns-565d847f94-l7pz9             0/1     ContainerCreating   0                  18d
kube-system   coredns-565d847f94-pfxbj             0/1     ContainerCreating   0                  18d
kube-system   etcd-k8smaster0                      1/1     Running             1154 (5m50s ago)   18d
kube-system   kube-apiserver-k8smaster0            1/1     Running             1327 (5m50s ago)   18d
kube-system   kube-controller-manager-k8smaster0   1/1     Running             1125 (5m50s ago)   18d
kube-system   kube-scheduler-k8smaster0            1/1     Running             1186 (5m50s ago)   18d

0x0101 Install Cilium

After we create the cluster, it is time to install the Network add-on. At this time, I am using cilium. There are many methods to install Cilium. I used helm to install it. Here is the command I used to install Cilium.

root@k8smaster0:~# cat install_cilium.sh
API_SERVER_IP=192.168.56.4
# Kubeadm default is 6443
API_SERVER_PORT=6443
helm install cilium cilium/cilium --version 1.12.4 \
    --namespace kube-system \
    --set kubeProxyReplacement=strict \
    --set k8sServiceHost=${API_SERVER_IP} \
    --set k8sServicePort=${API_SERVER_PORT} \
    --set ipv4NativeRoutingCIDR=192.168.56.0/24 \
    --set tunnel="disabled" \
    --set ipam.mode=kubernetes

Let me explain those options. The kubeProxyReplacement means we are using Cilium to replace the kube-proxy component. The tunnel and ipv4NativeRoutingCIDR indicate we are using Native Route mode instead of the overlay network and the ipam.mode means we delegate each node in the cluster to allocate IP addresses for the Pods.

The CoreDNS has been running since we installed Cilium.

root@k8smaster0:~# kubectl get pods -A
NAMESPACE     NAME                                 READY   STATUS    RESTARTS           AGE
kube-system   cilium-2xlxh                         1/1     Running   0                  6m53s
kube-system   cilium-hcr6r                         1/1     Running   0                  6m53s
kube-system   cilium-operator-675567f547-8jz7l     1/1     Running   0                  6m53s
kube-system   cilium-operator-675567f547-l68zt     1/1     Running   0                  6m53s
kube-system   coredns-565d847f94-l7pz9             1/1     Running   0                  18d
kube-system   coredns-565d847f94-pfxbj             1/1     Running   0                  18d
kube-system   etcd-k8smaster0                      1/1     Running   1154 (4h19m ago)   18d
kube-system   kube-apiserver-k8smaster0            1/1     Running   1327 (4h19m ago)   18d
kube-system   kube-controller-manager-k8smaster0   1/1     Running   1125 (4h19m ago)   18d
kube-system   kube-scheduler-k8smaster0            1/1     Running   1186 (4h19m ago)   18d

0x0102 Install kube-router

At the moment, the setting up of the network still has some work to do. If we want to use BGP mode, we must install BGP Daemonset on each node for BPG peering and route propagation. There have many options, such as kube-router, BIRD and Cilium Native BGP. I chose kube-router instead of others because the kube-router is easy to use.

You can download the YAML file which I used to install kube-router from here https://github.com/cloudnativelabs/kube-router/blob/v1.5.3/daemonset/generic-kuberouter-only-advertise-routes.yaml ,and I used following options:

...
containers:
      - name: kube-router
        image: docker.io/cloudnativelabs/kube-router
        imagePullPolicy: Always
        args:
        - "--run-router=true"
        - "--run-firewall=false"
        - "--run-service-proxy=false"
        - "--bgp-graceful-restart=true"
        - "--enable-cni=false"
        - "--enable-ibgp=true"
        - "--enable-overlay=false"
        - "--cluster-asn=65001"
        - "--advertise-cluster-ip=true"
        - "--advertise-external-ip=true"
        - "--advertise-loadbalancer-ip=true"
...

Now, the internal cluster is already using BGP. We can create a Deployment and a service to test it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx

Check the Nginx pods and services IP addresses by the following command. It indicated Cilium works fine.

root@k8smaster0:~/app# kubectl get service nginx
NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.96.229.67   <none>        80/TCP    12m
root@k8smaster0:~/app# kubectl get pods -o wide -l app=nginx
NAME                                READY   STATUS    RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-7fb96c846b-7jhtt   1/1     Running   0          12m   10.112.1.174   k8sslave0   <none>           <none>
nginx-deployment-7fb96c846b-lqpgf   1/1     Running   0          12m   10.112.1.125   k8sslave0   <none>           <none>
root@k8smaster0:~/app# curl 10.96.229.67
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

And we can use the Cilium command to get the services managed by Cilium.

root@k8smaster0:~/app# kubectl get pods -A -l k8s-app=cilium
NAMESPACE     NAME           READY   STATUS    RESTARTS   AGE
kube-system   cilium-2xlxh   1/1     Running   0          60m
kube-system   cilium-hcr6r   1/1     Running   0          60m
root@k8smaster0:~/app# kubectl -n kube-system exec cilium-2xlxh -- cilium service list
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)
ID   Frontend           Service Type   Backend
1    10.96.0.1:443      ClusterIP      1 => 192.168.56.4:6443 (active)
2    10.96.0.10:53      ClusterIP      1 => 10.112.0.59:53 (active)
                                       2 => 10.112.0.84:53 (active)
3    10.96.0.10:9153    ClusterIP      1 => 10.112.0.59:9153 (active)
                                       2 => 10.112.0.84:9153 (active)
4    10.98.28.103:443   ClusterIP      1 => 192.168.56.4:4244 (active)
                                       2 => 192.168.56.5:4244 (active)
5    10.96.229.67:80    ClusterIP      1 => 10.112.1.125:80 (active)
                                       2 => 10.112.1.174:80 (active)
root@k8smaster0:~/app#

We can see the Nginx Cluster IP 10.96.229.67 is active.

By now, we have finished all the work, the internal cluster is using BPG to communicate with each other, but we still have some work to do to let the external can route to the cluster. I’ll write it in my next blog.

0x00 Introduction

0x0000 Background

Recently, my company abandoned the local data centre and migrated all services to the k8s cluster in a public cloud provider. But after that, some problems emerged, and I was dealing with them. One of the problems is that we also migrated the development environment to the k8s cluster, but the developers of the R&D team still need to access the development environment directly. Let me explain why.

We used a microservice model to design our app architecture. Because my colleague needed access to the other modules and middleware, such as Zookeeper, Kafka, etc., while debugging in IDE, so you may ask, why don't my colleagues access services in k8s via the ingress? Well, usually, it can solve the problem. But in this case, because our services used Nacos as the service discovery/registry centre and Apache Dubbo as the RPC Framework, the services invoke the other services directly by IP address and port, so it's hard to let modules communicate by k8s service or ingress.

After that, I first needed to get through the network between the on-premises and cloud VPC, and I used the IPsec VPN protocol to deal with that. That's written in another post. Ok, Now you may know the background of my work. Still, I wasn't going to implement this solution in the on-used environment—the entire solution was deployed in an experimental environment because some work processes had to finish. Anyway, the result is not my point. The whole process is much more worthwhile.

0x0001 The Experimental Environment

In the previous, I said I deployed this in an experimental environment, which I set up with some virtual machines in the VirtualBox. So I have to introduce my network structure and System in my environment.

Here’s my IP assignment:

• k8s node CIDR: 192.168.56.0/24
• k8s master0 IP: 192.168.56.4
• k8s slave0 IP: 192.168.56.5
• k8s cluster-IP CIDR: 10.96.0.0/12
• k8s pod-IP CIDR: 10.112.0.0/12
• On-Premises CIDR: 192.168.57.0/24

Here’s my OS Configuration:

• System Version: Ubuntu 22.04.1 LTS
• Linux Kernel Version: 5.15.0-56-generic

Besides what I mentioned above, I also installed eBPF tools on my ubuntu. Here’s the command I used for the installation.

sudo apt-get install -y  make clang llvm libelf-dev libbpf-dev bpfcc-tools libbpfcc-dev linux-tools-$(uname -r) linux-headers-$(uname -r)

And I installed two NICs on each VM. One of the NICs was using a NAT network, and the other was using a Host-Only network, configuring IP within the k8s Nodes CIDR range.

Here’s a network configuration example from one of the VMs.

root@k8smaster0:~# cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp0s3:
      dhcp4: true
    enp0s8:
      dhcp4: false
      addresses: [192.168.56.4/24]
      nameservers:
        addresses: [8.8.8.8,114.114.114.114]
  version: 2
root@k8smaster0:~#

0x01 Deployment

0x0100 Pre-requisite

After I installed ubuntu and configured the network in the virtual machines, its’ time to install Kubernetes.

First, I specified the hostname with IP addresses in the /etc/hosts on each node, which is unnecessary. Still, I strongly recommend doing it because the kubelet will use the IP parsed by hostname as the node IP for the registry by default. Otherwise, you need to specify the parameter with --node-ip=x.x.x.x when starting kubelet. For example, If I want to register a k8s worker node with the hostname of k8sslave0 and two NICs, the first NIC with IP10.0.2.15 configured with the default route and the second with IP 192.168.56.5. If I want to use 192.168.56.5 as my registry IP, I need to configure /etc/hosts as follows:

127.0.0.1 localhost
127.0.1.1 ubuntu
192.168.56.5 k8sslave0

After configuring the host for each node, we need to install container runtime on each node so pods can run there. As you may know, many choices exist, but many people prefer to use the Docker Engine as runtime directly. Still, I used containerd as my runtime because Kubernetes no longer support Docker Engine directly after v1.23. You can access Kubernetes’s official documents to get more details.

Here are the commands to install containerd on ubuntu. You can find all of it on Docker’s official site.

# Uninstall old versions
sudo apt-get remove docker docker-engine docker.io containerd runc

# Set up the repository
sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# Add Docker's official GPG key:
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Use the following command to set up the repository:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install containerd.io

Then we need to configure systemd as the cgroup driver, this is optional, but systemd is recommended if you use cgroupv2. Here’re the commands to enable systemd.

# If we use package management tools like apt or yum, 
# we need to generate the default configuration first.
containerd config default > /etc/containerd/config.toml

# Then, modify the config.toml to enable systemd
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

# We will use kubeadm 1.25.4 to install k8s later,
# so, we need to change the sandbox image configured in the config.toml
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.8"

sudo systemctl restart containerd

We also need to check if our CNI is installed. Usually, after the container runtime is installed, the CNI will also be installed with the runtime. As default, the CNI binary files are located in the directory /opt/cni/bin, and the configuration directory is /etc/cni/net.d. Of course, the directory is configurable, which can find in the containerd configuration file.

cat /etc/containerd/config.toml

## Here's the configuration related to the cni
...
[plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      ip_pref = ""
      max_conf_num = 1

At this point, all of our preparation work for installing Kubernetes has been done. In my next blog, I will continue introducing how to install Kubernetes and configure cilium.

Last week, I had an error when installing cryptography by pip. Then I did a lot of searches, and most of the answers told me to install python-devel, but it still didn't work after I installed that, which drove me crazy. By the way, my process was running on Centos 7.4, and my python version is 2.7.5.

While installing cryptography by pip, GCC needs to compile a bunch of C files, and the C files reference header files, such as <pyconfig.h>. However, the GCC did not search pyconfig.h in /usr/include/python2.7/, because the GCC default search path does not include that path. We can print the current search pathes of GCC via command: gcc -xc -E -v -.

Finally, after I found the radical problem of this error, it was easy to solve by including a new search path for the GCC. Here is the command: export C_INCLUDE_PATH="$C_INCLUDE_PATH:/usr/include/python2.7/"

Why do we terrible at English

English as a global language is essential, especially for someone who wants to obtain first-hand information or live oversea. Chinese students spend much time learning English in their K-12 educational experience, but many still stay low. The reason I think is because of the wrong way of learning english.

In Chinese english Education, the most crucial purpose of learning english is not to communicate with people but to pass exams. That is why Chinese students are constantly reading and writing much better than listening and speaking because they spend much time on grammar, vocabulary, and reading exercises.

The origin of language

At first, primitive humans did not have a literal. The way of conveying information is speaking, the first edition of language. So we can see the basis of language is not about reading and writing but listening and speaking. After years of human revolution, primitive humans became cultured. They lived far away than they used to be, which is why we need the paper word to write letters, announcements, and something else. That is why language can be split into two parts: listening and speaking, which is based on audio, and reading and writing, which is based on vision.
The right way to learn English

As mentioned above, We know that language learning is divided into two parts, and each part can be divided into the input and output parts. If we want to improve our whole English level, the input is critical because the input is the ceiling of the output. That is why we need to set listening and reading as our priority.

The Part of the Input

Listening

As a Chinese IT industry practitioner, using VPN is a fundamental skill, which is a great advantage to learning English. Because the video resources on chinese websites such as bilibili can not compare with Youtube, and the most important is that the resources on Youtube are free. Besides Youtube, podcasts are also a helpful tool for learning English. Listening to podcasts is way better than watching TV series or movies because audio has more intensive information than video.

Reading

Besides listening, reading also is a part of the input. This part is relative to writing. To write concisely and correctly, we must read continuously and read everything in english, not just fiction, reports, novels, etc. For example, as an IT developers, we can read all software documents in English. Software documents are often written in simple sentences, which is a good beginning for us to start reading in English. If we can skim the documents without obstacles, we can change to reading novels.

The part of the output

Speaking

This part is the most difficult for me. As a non-native speaker, I do not think we have too much opportunity to speak, so I do not focus too much on that. I only spent a little time shadowing English Youtubers or reading books aloud. Maybe I will pay some English speaking coach to improve my speaking.

Writing

Writing is also tricky, but it is easier to solve than speaking. Speaking needs a partner to practice, but writing can practice by ourselves. We can write diaries, blogs, articles, etc. Another trick of writing is to use grammar-checking tools. Writing does not have to be too much; we can start with a tiny step, with no need to feel too much pressure. This is why I decide writing in English from now on. Maybe we will write with flaws, incorrect grammar usage, or something else, but a least it has started.

The last

Anyway, after learning english for two years, I feel kind of stuck, which is why I wrote this blog to recap. I hope I will be better.

0x00 背景

需要打通阿里云VPC 网络与本地IDC机房的私有网络，遇到不少问题都没有找到解决方案，在看了不少strongswan的wiki文章后才解决，特此记录下。该网络拓扑伪参数如下：

ali_vpc_network(172.16.0.0/16) <--> ECS(IPSEC Gateway)<--> IDC Firewall(IPSEC Gateway)<--> Local Network(192.168.22.0/24,192.168.24.0/23)

## ECS Internal IP: 172.16.10.1 Elastic IP: 1.1.1.1
## IDC Firewall IP: 192.168.10.1 Public IP: 2.2.2.2

0x01 ECS 部署命令

安装strongswan

yum install epel-release -y
yum install strongswan -y
## 开启ipforward 修改 /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
### 应用配置
sysctl -p

配置 strongswan /etc/strongswan/ipsec.conf

# ipsec.conf - strongSwan IPsec configuration file
# basic configuration
config setup
        # strictcrlpolicy=yes
        # uniqueids = no
        charondebug="all"
        uniqueids=never
conn %default
        authby=psk
        type=tunnel
conn office_wire
    dpdaction=hold # 意外断开后尝试重连时长
    dpddelay=30s # 意外断开后超时时长，只对 IKEv1 起作用
    dpdtimeout=60s
    fragmentation=yes
    keyexchange=ikev1           #IPsec连接使用的IKE协议的版本
    left=172.16.10.1
    leftsubnet=172.16.0.0/16    #VPC的网段
    leftid=1.1.1.1              # Elastic IP
    right=2.2.2.2               # IDC public IP
    rightsubnet=192.168.22.0/24 #本地的网段
    auto=start
    ike=3des-md5-modp1024       #IPsec连接中IKE协议的加密算法-认证算法-DH分组
    ikelifetime=7200s          #IKE协议的SA生命周期
    esp=3des-sha1       #IPsec连接中IPsec协议的加密算法-认证算法-DH分组
    lifetime=7200s             #IPsec协议的SA生命周期
    type=tunnel
conn office_wireless
    also=office_wire
    rightsubnet=192.168.24.0/23 #本地的网段

配置strongswan的psk /etc/strongswan/ipsec.secrets

# ipsec.secrets - strongSwan IPsec secrets file
1.1.1.1 2.2.2.2 : PSK password_placeholder  #123456为IPsec连接的预共享密钥，本地IDC侧和VPN网关侧的预共享密钥需一致

启动strongswan

systemctl start strongswan
systemctl enable strongswan
### 查看连接状态

strongswan statusall
## 如果有sa 为up 则建立成功

## 这时候可以ping通
ping 192.168.22.x

如果还需要VPC 网络中 172.16.0.0/16 中的其他ECS也能访问本地IDC的机器，则需要在VPC网络中的路由表中配置 172.16.10.1 为下一跳路由。同时还要还安全组中开启允许local network 的 inbound访问。

0x02 IDC Firewall 配置

这步不多说了，主要的问题在于感兴趣流的配置。一般 访问外网是NAT出去，假设ACL 3010 应用于该NAT outbound，则需要在该3010 中deny所有到 aliyun vpc的流量，同时需要在ipsec policy 中对应的ACL，这里假设是3001中 permit 所有到aliyun vpc的流量。这样才会触发感兴趣流，才能成功建立ike sa & ipsec sa。

0x03 Troubleshooting

1. 本地可以ping 通 ECS VPC Gateway，但是无法ping通其他机器
   1. 可能是ip_forward没开，或者iptables forward policy 为drop，最好改为accept
2. 可以ping通，也可建立tcp连接，但是http服务访问不行，或者mysql访问卡住了。一般是mtu的问题

   1. 可以执行：iptables -t mangle -A FORWARD -o eth0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360

   2. 或修改/etc/strongswan/strongswan.d/charon/kernel-netlink.conf 中的 mtu为 1360

   3. 重启strongswan

3. strongswan无法和本地成功建立连接：
   1. 因为ECS 是在弹性公网IP下面，ifconfig 是看不到公网IP的，所以配置文件中的left要使用private IP
4. strongswan无法建立设置的多个private network cidr
   1. 这个应该是strongswan的一个bug，在配置文件中使用also参数建立多个private network
5. 在Linux上怎么查看已建立的Ipsec policy?
   1. ip xfrm policy show

方案一：重建 Replicas

MySQL 5.6及以上版在复制中引入了新的全局事务ID（GTID）支持。 在启用了GTID模式的情况下执行MySQL和MySQL 5.7的备份时，Percona XtraBackup会自动将GTID值存储在xtrabackup_binlog_info中。 该信息可用于创建新的（或修复损坏的）基于GTID的副本。

前提条件

1. MySQL 机器上需要安装 percona xtrabackup

优点

1. 比较安全，操作简单

缺点

1. 数据量较大的时候备份所需的时间比较久
2. 当数据库有做读写分离的时候，Slave 承担的读请求需要转移到 Master

操作步骤

Master

1. 在 Master 上使用 xtrabackup 工具对当前的数据库进行备份，执行该命令的用户需要有读取 MySQL data 目录的权限

innobackupex --default-file=/etc/my.cnf --user=root -H 127.0.0.1 --password=[PASSWORD]  /tmp

2. 将该备份文件拷贝到 Slave 机器上

Slave

1. 在 Slave 机器上执行该命令，准备备份文件

innobackupex --default-file=/etc/my.cnf --user=root -H 127.0.0.1 --password=[PASSWORD] --apply-log /tmp/[TIMESTAMP]

2. 备份并删除 Slave data目录

systemctl stop mysqld
mv /data/mysql{,.bak}

3. 将备份拷贝到目标目录，并赋予相应的权限，然后重启 Slave

innobackupex --default-file=/etc/my.cnf --user=root -H 127.0.0.1 --password=[PASSWORD] --copy-back /tmp/[TIMESTAMP]
chmod 750 /data/mysql
chown mysql.mysql -R /data/mysql
systemctl start mysqld

4. 查看当前备份已经执行过的最后一个的GTID，如下示例

$ cat /tmp/[TIMESTAMP]/xtrabackup_binlog_info
mysql-bin.000002    1232        c777888a-b6df-11e2-a604-080027635ef5:1-4

这个GTID也会在 innobackupex 备份完成后打印出来

innobackupex: MySQL binlog position: filename 'mysql-bin.000002', position 1232, GTID of the last change 'c777888a-b6df-11e2-a604-080027635ef5:1-4'

5. 使用 root 登录 MySQL，进行如下配置

NewSlave > RESET MASTER;
NewSlave > SET GLOBAL gtid_purged='c777888a-b6df-11e2-a604-080027635ef5:1-4';
NewSlave > CHANGE MASTER TO
             MASTER_HOST="$masterip",
             MASTER_USER="repl",
             MASTER_PASSWORD="$slavepass",
             MASTER_AUTO_POSITION = 1;
NewSlave > START SLAVE;

6. 查看 Slave 的复制状态是否正常

NewSlave > SHOW SLAVE STATUS\G
         [..]
         Slave_IO_Running: Yes
         Slave_SQL_Running: Yes
         [...]
         Retrieved_Gtid_Set: c777888a-b6df-11e2-a604-080027635ef5:5
         Executed_Gtid_Set: c777888a-b6df-11e2-a604-080027635ef5:1-5

我们可以看到副本已检索到编号为5的新事务，因此从1到5的事务已在此副本上了。这样我们就完成了一个新 replicas 的搭建。

方案二：使用percona-toolkit进行数据修复

PT工具包中包含pt-table-checksum和pt-table-sync两个工具，主要用于检测主从是否一致以及修复数据不一致情况。

前提条件

1. MySQL 机器上需要安装 percona-toolkit 工具

优点

1. 修复速度快，不需要停止从库

缺点

1. 操作复杂，操作前最后先备份数据库
2. 待修复的表需要具有 unique constraint

操作步骤

背景示例

IP 关系对应

|  IP  | Role  |
|  ----  | ----  |
| 192.168.100.132  | Master |
| 192.168.100.131  | Slave |

假设待恢复的表结构如下所示

mysql> show create table test.t;
+-------+-------------------------------------
| Table | Create Table                                                                                                                                  |
+-------+-------------------------------------
| t     | CREATE TABLE `t` (
  `id` int(11) NOT NULL,
  `content` varchar(20) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-------+-------------------------------------

正常主备一致的情况下，Master 和 Slave 的数据均为如下所示

mysql> select * from test.t;
+----+---------+
| id | content |
+----+---------+
|  1 | a       |
|  2 | b       |
+----+---------+
2 rows in set (0.00 sec)

在极端情况下，假如出现了如下主备不一致的情况，情形如下：

1. Master 新增了一条 id 为 3 的记录，如下所示，但并没有同步到 Slave，同时自动 failover 到了 Slave。
2. Old Slave 作为 New Master 在服务了一段时间后，表中增加了新的记录。

重新启动 Old Master 后，Old Master的数据如下所示:

old_master> select * from test.t;
+----+---------+
| id | content |
+----+---------+
|  1 | a       |
|  2 | b       |
|  3 | c       |
+----+---------+
3 rows in set (0.00 sec)

New Master 的数据如下所示:

new_master> select * from test.t;
+----+---------+
| id | content |
+----+---------+
|  1 | a       |
|  2 | b       |
|  3 | cc      |
|  4 | dd      |
+----+---------+
4 rows in set (0.00 sec)

此时如果将 old master 配置为 new master 的slave，则会报错，比如出现如下报错

...Last_IO_Error: binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID.

可以看到 Old Master 的 GTID 已到 255

Executed_Gtid_Set: 5b750c75-86c2-11eb-af71-000c2973a2d5:1-10,
60d082ee-86c2-11eb-a9df-000c2988edab:1-255

而 New Master 的 GTID才到254

mysql> show master status\G
*************************** 1. row ***************************
             File: mysql-bin.000001
         Position: 4062
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: 5b750c75-86c2-11eb-af71-000c2973a2d5:1-2,
60d082ee-86c2-11eb-a9df-000c2988edab:1-254
1 row in set (0.00 sec)

此时我们配置 Old Master 跳过错误，将 Old Master 恢复成可以正常从 New Master 复制的状态

old_master> stop slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)

old_master> set gtid_next='60d082ee-86c2-11eb-a9df-000c2988edab:254';  --Specify the version of the next transaction,the GTID you want to skip
Query OK, 0 rows affected (0.00 sec)

old_master> begin;
Query OK, 0 rows affected (0.00 sec)

old_master> commit;                                                   -- Inject an empty transaction
Query OK, 0 rows affected (0.00 sec)

old_master> set gtid_next='AUTOMATIC'；   -- Restore to automaic GTID
Query OK, 0 rows affected (0.00 sec)

old_master> start slave;
Query OK, 0 rows affected (0.13 sec)

然后我们在 Old Master 上可以看到复制在正常进行

mysql> show slave status\G
            ...
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
            ...
            Executed_Gtid_Set: 5b750c75-86c2-11eb-af71-000c2973a2d5:1-10,
60d082ee-86c2-11eb-a9df-000c2988edab:1-255
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:

最后我们在 New Master 上清除 slave_master_info

new_master> reset slave all for channel '';
Query OK, 0 rows affected (0.00 sec)

new_master> show slave status\G;
Empty set (0.01 sec)

校验一致性

接下来我们要校验主从一致性，在 New Master上执行 pt-table-checksum，ROWS为4，存在一条DIFFS

[root@localhost ~]# pt-table-checksum h='127.0.0.1',u='mha',p='[PASSWORD]',P=3306 --no-check-binlog-format --databases test
Checking if all tables can be checksummed ...
Starting checksum ...
            TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
03-29T19:24:18      0      1        4          1       1       0   0.322 test.t

双向同步（同步操作会修改数据，操作前进行数据备份）

在同步过程中，pt-table-sync 会在 Master 上进行数据修改，pt-table-sync的参数作用如下

pt-table-sync --databases test --bidirectional --conflict-column='*' --conflict-comparison 'newest' h='192.168.100.132',u='mha',p='[PASSWORD]',P=3306 h='192.168.100.131' --print
--database                指定待执行的数据库
--bidirectional           为双向同步
--conflict-column         对比该列当冲突发生时
--conflict-comparison     冲突对比策略
--print                   输出对比结果
--dry-run                 测试运行
--execute                 执行测试

# 左边的DSN为 Slave
# 右边的DSN为 Master

这里我们指定—conflict-name='content'作为对比列，一般使用业务主键作为该列。可以看到打印出了待执行的语句

[root@localhost ~]# pt-table-sync --databases test --bidirectional  --conflict-column='content' --conflict-comparison 'newest' h='192.168.100.132',u='mha',p='[PASSWORD]',P=3306 h='192.168.100.131' --print
/*192.168.100.132:3306*/ UPDATE `test`.`t` SET `content`='cc' WHERE `id`='3' LIMIT 1;
/*192.168.100.132:3306*/ INSERT INTO `test`.`t`(`id`, `content`) VALUES ('4', 'dd');

接下来执行语句

[root@localhost ~]# pt-table-sync --databases test --bidirectional  --conflict-column='content' --conflict-comparison 'newest' h='192.168.100.132',u='mha',p='[PASSWORD]',P=3306 h='192.168.100.131' --execute

然后在 Master 上再次执行数据对比，可以看到数据正常了

[root@localhost ~]# pt-table-checksum h='127.0.0.1',u='mha',p='[PASSWORD]',P=3306 --no-check-binlog-format --databases test
Checking if all tables can be checksummed ...
Starting checksum ...
            TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
03-30T12:09:57      0      0        4          0       1       0   0.330 test.t

参考文档

https://www.percona.com/doc/percona-xtrabackup/2.4/howtos/recipes_ibkx_gtid.html

https://www.percona.com/doc/percona-toolkit/3.0/pt-table-checksum.html

https://www.percona.com/doc/percona-toolkit/3.0/pt-table-sync.html

https://developer.aliyun.com/article/708726

https://tech.meituan.com/2017/11/17/mysql-flashback.html

0x00 前言

在计算机世界中，一个东西习惯分成不同的层级，来模块化和解耦系统，这也就意味着一个东西有不同的层，比如计算机网络它有 OSI 七层模型和 TCP/IP 五层模型等，计算机体系结构从上到下可大致分为应用层，操作系统层，微指令架构层，硬件层等等，像其他的如操作系统和编程语言也是一样的道理，可见抽象和封装就是计算机世界的基本原则。

同样，我们在谈信息安全的时候，信息安全也是有不同的层级的。上一篇文章中我们说过了，安全的本质是数据，而随着数据的流动，就分层了不同的层级。当数据在网络中流动的时候，它是网络安全；当数据流经到了 Web 容器和 Web 应用的时候，它是 Web 安全；当数据流经到操作系统是，它是主机安全。

在了解了这些后，我们针对不同的层有不同的学习方向，而今天我记录的学习笔记是 Web 安全的学习笔记。在我们了解具体的漏洞前，需要抱着一切用户输入皆不可信的思想，很多漏洞利用本质就是把用户的输入的数据当做代码执行。

0x10 常见 Web 漏洞

0x11 XSS

原理：用户的输入被当做 js 代码执行了
分类：

• 反射型: 恶意代码输入直接被输出执行，一般在前后端一体的架构中，服务端改变了返回的页面代码
• 存储型: 恶意代码被提交存储在数据库中，可以被加载执行
• DOM型: 恶意代码提交到后端服务器后，前端代码根据后端服务器返回的内容通过 js 修改了 htm 页面，达到了插入脚本的目的，常见于前后端分离的项目。
  危害：
• 窃取 cookie
• 钓鱼操作
• 记录用户行为
• 未授权操作
  防护：
• 验证输入输出
• 编码
• 白名单过滤
• CSP(Content Security Policy)

0x12 SQL注入

原理：构造恶意 SQL 语句参数，导致 SQL 语句被篡改
分类：参数类型可分为数字型和字符型，而根据回显结果可以分为回显注入，报错注入和盲注
危害：篡改窃取数据
防护：预编译语句，验证输入，使用存储过程

0x13 CSRF

原理：伪造用户操作
防护：csrf token，二次验证，防止参数被攻击者猜解

0x14 SSRF

原理：通过 url 控制服务器去访问其他服务器
防护：白名单限制，协议和资源限制，请求端限制

0x15 反序列化漏洞

原理：通过在数据中嵌入自定义代码提交到服务器，服务器反序列化过程中执行了代码，来控制整台服务器
防护：限制序列化和反序列化的类，RASP 检测

0x16 信息泄露

原理：注释、异常、返回信息或者代码泄露
防护：模糊处理，关闭异常打印和错误，监控 github 等网站

0x17 插件漏洞

原理：第三方插件存在漏洞
防护：第三方插件监控和检测

0x18 上传漏洞

原理：上传图片或者 webshell
防护：文件校验

0x20 小结

web 安全的漏洞有很多，这里也只记了个大概，总之漏洞的原理就是程序把数据当做代码执行，而我们的应对防止无非是验证数据和过滤等等，还是那句话，一切输入皆不可信。

0x00 前言

作为一名在安全和运维岗位都有过从业经验的人来说，我一直觉得运维和安全是密不可分的，运维和安全团队的特点就是时常要面对非常棘手的事故，一没处理好造成公司的损失那年终奖就鸡飞蛋打了，并且这两个岗位都是属于平时没出事就一切安好，一出事就是背锅的角色，在没出事时别人觉得养着你有何用，出了事后别人觉得要你有何用。。。所以我觉得对运维和安全来说，有防范于未然和主动响应的意识真的很重要。

现在回想起在上家公司的时候，公司是一家 to B的安全乙方厂商，我当时是在做安全实施的工作，经常需要到客户那里拿安全产品进行巡检和部署安全产品等等，我就发现安全真是马太效应，大公司都有一定规模的专业安全团队，并且像一些 WAF、IDS/IPS 等基础的安全产品很完备，甚至还有专门的安全研发团队，src和安全运营，他们可能暂时缺少的是一些比较上层建筑的安全产品，比如 RASP 之类的微服务应用防护，DevSecOps 安全开发的产品等等，而中小公司则基本就是字典里没有 “安全” 这两个字，可能好点的会有一人安全，就只有一个安全工程师，这个人要负责的事也挺杂挺重的，什么等保合规啦，渗透测试，基线检查，内部安全准则文档编写，安全开发培训等等，更有一些公司则是安全的活全由运维哥来兼职，所以我现在小公司的什么 app 产品我都不太敢用，指不定哪天就被脱裤了。。

当然，小公司不在业务上投入太多的资源也是可以理解的，毕竟安全是末位需求，小公司资源有限，但是我认为随着等保2.0的正式实施，几乎所有的企业都要参加等保，国家也越来越重视安全，安全问题不是那么好规避了，所以需要运维和开发等技术人员有基本的安全的意识。由于我现在的公司之前也出过安全事故，所以我面试也会注重求职者的安全知识，但从我之前面试过的运维求职者来看，基本上一点都不懂安全。目前来说，在小公司运维要兼职做安全，在大公司则运维团队需要更好的和安全团队协作来应急响应线上问题进行处理，所以我觉得不管是在大公司和小公司，运维都要学习一定的安全知识，做好运维安全，另外现在云计算基本都是标配了，在云上又会面临新的安全问题，可以说运维安全的门槛是越来越高了。

虽然我也之前在乙方安全厂商待过一段时间，安全技术书也看了不少，但之前是走的野路子，且我现在觉得从运维的角度来看安全和在乙方差异还是蛮大的，这也是我为什么重拾安全，系统的学习一波安全，把学到的安全知识和自己的实际经历记录成学习笔记。

0x01 安全基础

0x010 本质是数据

谈安全自然是要有要保护安全的对象，比如银行要保证钱的安全，钱就是被保护的对象，所以银行的金库需要安保措施。同样，对于互联网公司来说，什么是要被保护的？那就是业务，业务最终产生的是什么？业务最终产生的则是用户数据，比如互金行业，他的数据就是借贷人的数据，如果这些数据被丢失了或者损坏了，那后果有多严重可想而知。所以，对于数据这个 objective（客体） 的保护就是我们的安全。
数据的保护主要有 CIA 3 个原则：

• 机密性（Confidentiality）:
  • 确保只有该看的人才能看见，如我们不会允许陌生人查看我们的隐私，我觉得把数据比喻成文件的话，这就是数据的读权限
  • 针对机密性则产生了各种加密，混淆等技术
• 完整性（Integrity）:
  • 保证数据不能随便被篡改，比如我们不可能让别人把我们的钱改少吧
  • 数据写权限
  • 如签名技术
• 可用性（Availability）
  • 确保数据可以被访问到，才能保证业务可用
  • 比如某些 P2P 公司恶意竞争，针对我们公司 DDoS 攻击就是针对可用性的攻击

0x011 3A原则

那么我们需要如何去保证 CIA 3 个原则呢？那就是授权认证，也就是我们平时说的 3A认证:

• 认证(Authentication)：你是谁，事前防御
• 授权(Authorization)：你能做什么，事中防御
• 审计(Audit)：你做过什么，事后审计

0x012 安全实施基础

如何实施 3A 认证？安全的原则建立在加密的机制上，密码学就是安全的基础设施，所以先需要了解密码学，常用的加密算法有：

• 对称加密：加密解密用一把钥匙：
  • DES(Data Encryption Standard): 密钥长度 56，不太安全
  • IDES(International Data Encryption Algorithm): 密钥长度 128，比 DES 加密强，比 AES 慢
  • AES(Advanced Encryption Standard): 密钥有 128，192，256 等长度，目前安全，用 AES-CTR
  • 国密SM1和SM4(SM4 Cryptographic Algorithm): 国产算法，一般国企要求
• 非对称加密：加密和解密密钥不同：私钥加密，公钥解密是签名，确认身份的；公钥加密，私钥解密是数据加密
  • RSA: 质数计算
  • ECC(Elliptic Curve Cryptography): 椭圆曲线
  • 国密SM2: 也是椭圆曲线
• hash 算法，把消息 Mapping 到一个有限集计算 hash 值，不可逆，一般用作校验
  • MD5(Message-Digest Algorithm 5): 已被破解
  • SHA(Secure Hash Algorithm): 推荐用 SHA-256及以上
  • 国密SM3(SM3 Cryptographic Algorithm): 和 SHA-256强度差不多

除了密码学外，在实际的落地过程中，认证问题存在的问题有哪些？ 认证的风险有比如弱口令，密码泄露等等。而在实际中认证通常用的系统是单点登录系统，特别是微服务架构和多应用平台的系统。而针对内部认证一般是 LDAP 用的比较多。典型的 SSO 有：

• CAS
• JWT
• OAuth
• OpenID

除了认证要做好之外，授权也要做好。授权无非是对 subjective（主体） 对 objective（客体） 的 verb（动词） 允许或拒绝，而针对这两者之间，不同的场景就有不同的模型了，常见的有：

• DAC(Discretionary Access Control) 自主访问控制: objective 的所有者决定访问的权限，比如 linux 文件系统
• role-BAC: 基于角色的访问控制，比如 kubernetes 中，serviceaccount 的权限由他 rolebinding 到的 role 决定
• rule-BAC: 基于规则的访问控制，比如 iptables,不符合规则就 deny
• MAC(Mandatory Access Control): 强制访问控制，对 subject 和 object 都打标签，根据标签制定访问控制策略，比如 selinux 使用的就是这个

3A 原则的最后一点则是 Audit，需要记录下我们到底做了什么，比如运维中必备的 ELK一套日志基础设施就是起到审计的作用。另外现在运维常提的 AIOps，也会用到机器学习的一些手段对日志进行异常分析和做业务风控，这些都是属于 Audit。

0x013 威胁评估

在我们了解了安全的本质是数据后，我们要如何保护呢？这些数据又面临着哪些风险？这就需要威胁风险评估的必要。威胁评估有三个步骤：

• 识别数据：我们首先要知道数据在哪里，比如我们的 Web 服务器，这台机器上放着着我们的业务，那么这台服务器就是我们的资产，同时我们也根据资产的不同等级来实施不同的保护措施
• 识别攻击：黑客会通过哪些手段来进行攻击，暴力破解，注入还是社工？我们这台服务器的暴露面在哪里？上面开启的哪些端口，黑客会怎么攻进来？
• 识别漏洞：这台服务器存在哪些漏洞，我们用到了什么组件？struts2 还是 spring？有没有字符串拼接SQL语句？这些都是可能存在的漏洞

在对面临的威胁有了一个大概的评估后，我们才好推动安全和解决风险，另外如果说的直接一点，我们必须有对整个资产面临的风险的把控能力，才好拿到预算并做好公司的安全预案，毕竟钱多好办事呐。

0x00 前言

在上篇学习笔记中我总结了 I/O 学习笔记，本篇会总结 Linux 网络的性能优化学习笔记。在上篇文章中我们提到了说磁盘 I/O 就是计算机的最终数据持久化的地方，如下图在冯诺依曼架构中也就是对应输入输出的设备，而存储器则就是内存。

在单台计算机的时候，可以说计算过程就是先从输入设备获取指令和数据，在经过运算器的处理后，最终将运算结果输出到输出设备上，而 I/O 的产生则是在和输入和输出设备打交道了。如果是多台计算机，计算机之间如何互相沟通呢？那么就需要通过网络来连接各个计算机了。

在计算机之间的沟通，无非也就是指令和数据就是这两种。比如我访问一个 Web 网站，本质上不就是从网站服务器上获取数据吗？又比如在分布式的计算集群中，master 节点向 slave 节点下发计算任务，这就输出指令呀。是不是感觉网络和磁盘 I/O 很相似？没错，他们在冯诺依曼结果中都是作为输入输出设备，所以他们有很多相似之处，比如在 Linux 系统中，网络 I/O 的操作和文件操作很相似，但网络的层次比文件系统 I/O 更多，所以会更加复杂，一篇文章无法概述，本笔记只是记录一些要点。

0x01 网络基础

0x0100 收发流程

网络和文件系统一样，为了解耦异构设备，都是分层的，分层在网络中主要分为 OSI 和 TCP/IP 模型。如下所示

而自然和磁盘一样，Linux 内核在处理网络包的时候对不同的层会进行不同的处理。

• 应用层： 内核态系统调用，通过 socket 来发包
• 传输层： 对应用层的包进行封装，是五元组的抽象
• IP层： IP 报文发送，路由
• 链路层： 封帧，mac 寻址
• 网卡驱动： 收发包，网卡驱动收发包会通过硬中断和内核交互

0x0101 收包

1. 网卡 DMA（可以理解为直接读 I/O 到内存，减少 cpu 的介入） 把收到的网络帧放到收包队列（在内存中），硬中断通知中断程序处理
2. 中断程序分配一个叫 sk_buff 的 socket buffer数据结构，拷贝网络帧到这个结构，然后软中断通知内核来处理网络帧了
3. 内核获得网络帧后，就跟剥洋葱一样，一层层的剥开
   1. 数据链路层：检查报文，去掉帧头帧尾，看 ip 层是什么报文，如果是 6 就交给上层 ipv6 对应的协议处理，如果是 4 就交给 4
   2. IP层： 判断网络包走向，是转发还是交给上层处理。比如 iptable 中的 forward 链就是转发链，如果要交给上层则确定上层协议是 tcp/udp 等，并去掉头尾
   3. 传输层：根据 tcp/udp 头，根据四元组找到对应的 socket，然后把数据拷贝到 socket 里
   4. 应用层：操作 socket 读取数据

0x0102 发包

1. 应用层：通过系统调用，把数据包写到 socket 里
2. 传输层：增加 tcp 头
3. ip 层：增加 ip 头，路由寻址，找到下一跳，根据 MTU 分片
4. 数据链路层：根据 mac 地址寻址，找到下一跳的 mac，增加头尾，放到发包队列，软中断通知中断程序驱动
5. 物理层：驱动程序 DMA 读取网络帧，通过物理网卡通过脉冲信号或光信号发包

0x0110 性能指标

1. 带宽： 最大传输速率
2. 吞吐量： 单位时间内传输数据量
3. 延时： 发出请求到收到响应的时间
4. PPS： Packet Per Second，每秒发包量
5. 网络可用性
6. 并发数
7. 丢包率
8. 重传率

0x02 网络 I/O 原理

I/O通知方式：

1. 水平触发： 应用程序 可以随时读fd
2. 边缘触发： 只能在 fd 发生变化是才可以读

I/O 多路复用：

1. 非阻塞 I/O 水平触发：
   • select： fd 数量有限制，轮询检查 fd。O(n)
   • poll： 无 fd 限制，轮询检查fd。O(n)
2. 非阻塞 I/O边缘触发：
   • epoll： 内核红黑树管理 fd，事件驱动
3. 异步 I/O：异步通知，通过信号或者回调

工作模型：

• 主进程 + 多个 workder 子进程：nginx
  • 主 bind + listen 初始化 socket
  • 子 accept + epoll_wait 都绑定这个 socket
  • 有惊群问题，需要锁机制来确保只有一个 worker 节点唤醒
• 多进程监听同个端口，需要开启端口重用，等于是内核将该端口的请求分发到不同的 worker 进程上

高性能网络模型：

• DPDK：跳过内核网络协议栈，直接用户态轮询
• XDP：在内核网络协议栈前处理网络包

0x03 网络优化方式

网络也是分层的，不同的层有不同的优化手段

1. 应用层：
   • 优化 I/O 模型
   • 长连接
   • DNS 缓存
2. socket： 调整内核参数
3. 传输层： 优化传输层协议
4. 网络层：路由等
5. 链路层：收发包

基本上就是：

1. 先查看一波网络性能指标： sar，ip,ss 等工具
2. tcpdump,wireshark抓包，分析异常网络流量情况
3. 调整对应内核参数，基本上可以解决 80% 的问题

0x04 总结

虽然我是计算机网络专业毕业，但是想一篇文章写完所有的优化点还是太难了，毕竟计算机网络是一门单独的学科，只能把简单地把网络原理和优化思路记录下，在实际使用的时候还是要根据具体的情况来排查问题，另外有很多细节我比较熟悉，觉得太简单了就不一一记录了。

0x00 前言

在上一篇笔记中我们了解到了内存本质上它的作用就是作为外部存储的缓存，而数据的持久化和未命中的数据访问等最终还是需要外部存储如磁盘，那么自然就需要频繁的 I/O 操作，这篇笔记就是介绍磁盘的工作原理和 I/O 的优化手段。

0x10 原理

老惯例，在要学习 I/O 的性能分析和优化手段之前，我们要先学习外部存储的原理。从上篇内存篇笔记中我们知道内存主要有 buff/cache 分别为磁盘 I/O 和文件系统 I/O 进行缓存，磁盘读写即为直接访问磁盘块设备，又称为裸 I/O，文件系统则是建立在磁盘上的一种文件的管理和组织方式，它是处于在块设备之上的。我们的应用程序有的是通过文件系统 I/O 进行访问，有的则是直接裸 I/O 访问的磁盘，针对不同的访问方式，我们有不同的优化手段。另外通常情况下，从上到下 I/O 栈主要分为三层：

• 文件系统层： 包括虚拟文件系统层，为应用程序提供文件访问接口和管理磁盘数据
• 通用块层：包括设备 I/O 队列和 I/O 调度器，对 I/O 进行管理和调度，再发给下一级设备
• 设备层：包括存储设备和存储驱动，负责最终的 I/O

0x11 文件系统

文件系统是一种文件的组织和管理机制，而不同的管理机制就是不同文件系统，如 ext4,xfs, ntfs 等等。在文件系统中，一个文件不仅仅只是存储在磁盘上的数据，这些数据会有附带的元信息。就好像一个“人”除了本身的生理躯体外，他还有名字、出生日期、身份证号码和家庭住址等很多信息才能被称为一个人，他才能被标识和寻址。在 Linux 上，文件系统为了管理会为每个文件都分配两个数据结构来记录文件的元信息和目录结构：

• 索引节点：inode(index node)
  1. 记录文件元数据，inode 编号，文件大小，权限，修改日志等
  2. inode 和文件一一对应，就像每个人对应一个 id card
  3. 存储在磁盘，需要占用磁盘空间
• 目录项： dentry(directory entry)
  1. 记录文件的名字，inode
  2. 记录文件和其他文件的关联关系，就像家谱一样
  3. 由内核维护并存在内存中，也叫目录项缓缓存
  4. 和inode 是多对一的关系，可以理解为一个文件有多个别名，就好像一个人可以有多个身份，既可以是一个人的爸爸，也可以是另一个人的儿子

索引节点和目录项的关系如下图所示：

从这个图我们可以得知以下几点：

1. 每个 dentry 就是一个数据结构，这个数据结构里面包含了 inode，文件名和其他关联的 dentry 地址等
2. 其中 inode 是存储在磁盘的，dentry 是内存缓存
3. 由上篇内存笔记可以推测，inode 也会缓存到内存中
4. dentry 之间类似树形结构
5. inode 和文件数据是存储在不同的磁盘 block 的

根据第六点，我们又可以引申出另外一个知识点，那就是给磁盘格式化文件系统的时候，会把文件系统分为三个 block：

• 超级块： 存储整个文件系统的状态
• inode block： 存储 inode 节点
• 数据块：存储文件数据

而机械硬盘的最小扇区是512B，而每次读这么小效率低，所以文件系统还会把扇区组织成4KB的逻辑块，如上图左边所示，数据块被分为了一个个逻辑块

0x1100 虚拟文件系统

所有的文件系统的四大基本要素：

• dentry
• inode
• 逻辑块
• 超级块

而不同的文件系统有不同的组织方式，Linux 内核在用户进程和文件系统之间引入抽象层 VFS（Virtual File System），就好像 Java 里面的实现抽象方法一样， VFS 是抽象类，具体的文件系统则实现抽象类定义的接口。

如下图所示：

我们可以看到 Linux 支持的文件系统可分为：

1. 磁盘文件系统
2. 网络文件系统
3. 内存文件系统

文件系统先要在 VFS 中找个挂载点挂载，然后用户进程就可以通过 VFS 的文件接口访问其中的文件了

0x1101 文件系统 I/O

通过文件接口可以对文件进行读写等操作，读写方式的差异导致 I/O 有很多分类。主要有四种：

• 是否标准库缓冲（缓冲由标准库内部实现）：
  1. 缓冲 I/O：使用标准库缓存加速文件访问
  2. 非缓冲 I/O：直接通过系统调用访问
• 直接/非直接 I/O:
  1. 直接 I/O 跳过操作系统的页缓存（cache），直接访问文件系统。使用 O_DIRECT标志
  2. 非直接 I/O 使用 cache
• 阻塞/非阻塞 I/O（调用者视角，关注是否等待调用结果）：
  1. 调用者执行 I/O 操作，如果被调用者没有响应，则调用者线程阻塞，不能干别的事了。
  2. 调用者执行 I/O 操作，如果被调用者没有响应，调用者线程也不会被阻塞，可以去干别的事。后续调用者可以通过轮询（调用者过段时间就去看下好了没）或则事件通知（被调用者好了后主动通知调用者）的方式获取到响应。
• 同步/异步 I/O（被调用者视角，关注的是消息的通知机制）：
  1. 同步是被调用者直到 I/O 操作执行完才会返回结果给调用者。如果调用者是阻塞的，那么调用者就只能一直等着被调用者执行完了；如果调用者是非阻塞的，调用者就先跑了，他可能过段时间再来看下是不是好了（轮询）或者打电话给他（事件通知）
  2. 异步是被调用者马上返回 I/O 操作给调用者，让调用者先去干别的事，好了再通知他。

0x1102 文件系统性能观察

• 容量： df 命令
• 缓存：
  • 查看/proc/memeinfoSlab,/proc/slabinfo文件查看 dentry 和 inode 缓存
  • slabtop 工具

0x12 磁盘 I/O

磁盘主要分为机械磁盘和固态磁盘，磁盘的连续读写速度比随机读写速度更快，在 Linux 中，磁盘是作为块设备使用的。为了减小不同磁盘设备带来的差异，Linux 通过通用块层来管理不同的块设备。通用块层的主要功能如下：

• 承上启下，为文件系统提供统一的标准接口和提供统一框架管理块设备的驱动程序
• 对上层的 I/O 请求排队并进行调度，提高磁盘读写效率，Linux 支持 4 种调度算法
  • NONE: 不使用任何调度，常用于 vm
  • NOOP: 先入先出的队列，常用于 ssd
  • CFQ（Completely Fair Scheduler）: 完全公平调度器，基于时间片段来分布每个进程的请求，支持调度优先级，适合大量进程的系统，如 desktop，media
  • DeadLine: 读写使用不同的 I/O 队列，多用于 I/O 压力比较重的场景，如数据库

0x1200 磁盘性能指标

• 使用率：处理 I/O 的时间百分比（只考虑 I/O 数量，不考虑 I/O 大小）
• 饱和度：处理 I/O 的繁忙程度，100%时无法接受 I/O 请求
• IOPS(Input/Output Per Second)： 每秒接受的 I/O 请求
• 吞吐量：每秒的 I/O 请求大小
• 响应时间：I/O 请求发出到响应的时间

0X1201 性能观测

• 整体：iostat
• 单个进程：pidstat,iostop
• 磁盘性能度量：fio

0x20 性能调优

在了解了 I/O 栈的原理后，我们分析性能瓶颈的路数大概都是

1. iostat 观察整体的磁盘情况，分析各项指标
2. pidstat 观察具体进程的情况，分析各项指标
3. strace分析进程的 I/O 行为
4. 结合应用程序，分析I/O 来自哪里，是否正常

其中文件系统和 I/O 指标和磁盘的 I/O 指标需要分开来分析，另外在分析这些指标的时候，需要根据具体的 I/O 场景来进行分析，除了分析 I/O 指标，内存缓存也是需要分析的。

如果要对 I/O 进行调优，则每层都有不同的手段：

• 应用层

  • 连续读写代替随机读写
  • 多使用系统缓存
  • 使用外部缓存，如 MySQL 前放 Redis
  • 读写同一块磁盘空间的时候，使用 mmap（内存文件映射）
  • 同步写的时候，请求合并，减少访问 I/O 次数
  • 如果有多个进程时，可以用 cgroup 限制 I/O
  • 在 CFQ 中使用 ionice 调整进程的 I/O 优先级

• 文件系统层

  • 使用合适的文件系统
  • 优化文件系统配置选项
  • 优化文件系统缓存
  • 不需要持久化可使用 tmpfs

• 磁盘

  • 用 SSD
  • 组 RAID
  • 选择合适的 I/O 调度算法
  • 不同进程使用不同的磁盘
  • 针对顺序读场景增大预读数据
  • 优化内核，如调整 I/O 队列长度

0x00 前言

此篇文章是上篇理解CNI（容器网络接口）的后续

原作者：Jon Langemak
原文地址：Using CNI with Docker
译者: n0mansky

0x11 正文

在理解CNI（容器网络接口）（如果没读建议先读下）中，我们通过一个示例介绍了CNI如何将网络命名空间连接到 bridge 接口，CNI负责创建 bridge 网卡并使用 VETH pair 连接 bridge 和命名空间。在本文中，我们将探讨如何为 Docker 创建的容器连接到 bridge，具体步骤和上篇文章讨论的差不多，让我们开始吧。

本文假定您已按照第一篇文章理解CNI（容器网络接口）中的步骤动手操作过了，并已经创建了一个包含CNI二进制文件的'cni'目录（~/cni）。如果没有，请返回上一篇文章，并按照步骤下载CNI二进制文件。同时您需要安装Docker，我使用的是Docker 1.12版。（译者注：Docker 版本大于 1.12 也是可以的，我的是 18.09 ）

首先我们需要执行下面的命令来创建一个 Docker 容器

user@ubuntu-2:~/cni$ sudo docker run --name cnitest --net=none -d jonlangemak/web_server_1
835583cdf382520283c709b5a5ee866b9dccf4861672b95eccbc7b7688109b56
user@ubuntu-2:~/cni$

我们注意到当执行上述命令时，设置的网络为 none。当使用none时，Docker 将为该容器创建一个不会连接任何网络的命名空间。如果我们查看容器，我们应该看到它只会有一个环回接口…

user@ubuntu-2:~/cni$ sudo docker exec cnitest ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
user@ubuntu-2:~/cni$

接下来我们要使用CNI将该容器连接到网络。在此之前，我们需要一个给CNI使用的定义和给容器网络本身的一些定义。对于CNI定义，我们将创建一个新定义配置文件，并指定一些选项观察其工作方式。我们使用下面命令创建配置（假设您是在~/cni中创建该文件）...

cat > mybridge2.conf <<"EOF"
{
    "cniVersion": "0.2.0",
    "name": "mybridge",
    "type": "bridge",
    "bridge": "cni_bridge1",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.15.30.0/24",
        "routes": [
            { "dst": "0.0.0.0/0" },
            { "dst": "1.1.1.1/32", "gw":"10.15.30.1"}
        ],
        "rangeStart": "10.15.30.100",
        "rangeEnd": "10.15.30.200",
        "gateway": "10.15.30.99"
    }
}
EOF

除了上一篇文章中看到的参数外，我们还添加了以下内容……

• rangeStart：定义CNI给容器分配子网 IP 的起始地址
• rangeEnd：定义CNI给容器分配子网 IP 的结束地址
• gateway：定义网关地址。在上篇文章中我们没有定义，因此CNI在 bridge 接口上用的第一个IP作为网关。

您可能会注意到，此配置中缺少关于 DNS 的配置，这个我们先不提，下篇文章会说。

目前我们已经定义好了网络，我们还需要容器网络命名空间的路径和容器ID。要获取该信息，我们可以用docker inspect命令。

user@ubuntu-1:~/cni$ sudo docker inspect cnitest | grep -E 'SandboxKey|Id'
        "Id": "1018026ebc02fa0cbf2be35325f4833ec1086cf6364c7b2cf17d80255d7d4a27",
            "SandboxKey": "/var/run/docker/netns/2e4813b1a912",
user@ubuntu-1:~/cni$

在此示例中，我用grep -E正则模式来匹配查找容器ID 和 SandboxKey。在 Docker 中，网络命名空间文件位置称为“ SandboxKey”，而“ Id”是 Docker 为容器分配的ID。有了这些信息我们就可以构建调用CNI插件的环境变量了，如下所示：

• CNI_COMMAND= ADD
• CNI_CONTAINERID= 1018026ebc02fa0cbf2be35325f4833ec1086cf6364c7b2cf17d80255d7d4a27
• CNI_NETNS= /var/run/docker/netns/2e4813b1a912
• CNI_IFNAME= eth0
• CNI_PATH=`pwd`

我们将所有内容放到一条命令中，如下所示：

sudo CNI_COMMAND=ADD CNI_CONTAINERID=1018026ebc02fa0cbf2be35325f4833ec1086cf6364c7b2cf17d80255d7d4a27 CNI_NETNS=/var/run/docker/netns/2e4813b1a912 CNI_IFNAME=eth0 CNI_PATH=`pwd` ./bridge < mybridge2.conf

然后运行插件...和我们在上篇文章中看到的一样，插件执行后会将操作的结果以JSON返回

user@ubuntu-1:~/cni$ sudo CNI_COMMAND=ADD CNI_CONTAINERID=1018026ebc02fa0cbf2be35325f4833ec1086cf6364c7b2cf17d80255d7d4a27 CNI_NETNS=/var/run/docker/netns/2e4813b1a912 CNI_IFNAME=eth0 CNI_PATH=`pwd` ./bridge < mybridge2.conf
{
    "ip4": {
        "ip": "10.15.30.100/24",
        "gateway": "10.15.30.99",
        "routes": [
            {
                "dst": "0.0.0.0/0"
            },
            {
                "dst": "1.1.1.1/32",
                "gw": "10.15.30.1"
            }
        ]
    },
    "dns": {}
}user@ubuntu-1:~/cni$

让我们再次查看宿主机和容器网络，看看有什么变化...

user@ubuntu-1:~/cni$ ifconfig
cni_bridge0 Link encap:Ethernet  HWaddr 0a:58:0a:0f:14:01
          inet addr:10.15.20.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::a464:72ff:fe98:2652/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:536 (536.0 B)  TX bytes:648 (648.0 B)

cni_bridge1 Link encap:Ethernet  HWaddr 0a:58:0a:0f:1e:63
          inet addr:10.15.30.99  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::88f:bbff:fed9:118f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:536 (536.0 B)  TX bytes:648 (648.0 B)

docker0   Link encap:Ethernet  HWaddr 02:42:65:43:f5:a7
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ens32     Link encap:Ethernet  HWaddr 00:0c:29:3e:49:51
          inet addr:10.20.30.71  Bcast:10.20.30.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe3e:4951/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2568909 errors:0 dropped:67 overruns:0 frame:0
          TX packets:2057136 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:478331698 (478.3 MB)  TX bytes:1336636840 (1.3 GB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:5519471 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5519471 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:2796275357 (2.7 GB)  TX bytes:2796275357 (2.7 GB)

veth719c8174 Link encap:Ethernet  HWaddr aa:bb:6e:c7:cc:d8
          inet6 addr: fe80::a8bb:6eff:fec7:ccd8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1206 (1.2 KB)

vethb125661a Link encap:Ethernet  HWaddr fa:54:99:46:65:08
          inet6 addr: fe80::f854:99ff:fe46:6508/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1206 (1.2 KB)

user@ubuntu-1:~/cni$

从宿主机角度来看，我们现在有很多网络接口。cni_bridge0接口和它关联的 VETH pair是上篇文章操作留下的，而cni_bridge1及其关联的VETH pair接口则是我们刚刚创建的。您可以看到cni_bridge1接口的 IP 是我们在CNI网络配置中“gateway”部分的IP地址。您还会注意到有一个docker0接口，它是在安装 Docker 时默认创建的。

我们的容器有何变化呢？让我们看看吧...

user@ubuntu-1:~/cni$ sudo docker exec cnitest ifconfig
eth0      Link encap:Ethernet  HWaddr 0a:58:0a:0f:1e:64
          inet addr:10.15.30.100  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::f09e:73ff:fe3e:838c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1206 (1.2 KB)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

user@ubuntu-1:~/cni$ sudo docker exec cnitest ip route
default via 10.15.30.99 dev eth0
1.1.1.1 via 10.15.30.1 dev eth0
10.15.30.0/24 dev eth0  proto kernel  scope link  src 10.15.30.100
user@ubuntu-1:~/cni$

如您所见，容器的网络配置和我们的预期一致...

• IP地址在定义的范围内（10.15.30.100）
• 其接口名为“ eth0”
• 默认路由指向网关IP地址10.15.30.99
• 我们额外加的1.1.1.1/32的下一跳路由为10.15.30.1

最后，我们可以尝试从宿主机访问容器中的服务...

user@ubuntu-1:~/cni$ curl http://10.15.30.100
<body>
<html>
<h1><span >Web Server #1 - Running on port 80</span></h1>
</body>
</html>
user@ubuntu-1:~/cni$

正如我们所示，连接Docker容器与上篇文章中直接连接命名空间没有太大不同，实际上可以认为过程是相同的，我们只需要知道Docker把容器的网络命名空间的定义存在哪里就行了。在我们的下一篇文章中，我们将讨论如何通过 CNI 为容器进行DNS相关设置。

0x00 前言

随着微服务架构和云计算的普及，越来越多企业的应用都上了云，不仅是云基础设施 IaaS ，如 kubernetes 等 PaaS 项目也是越来越热门。但新的技术会带来新的架构复杂，同时也会使排查问题更加困难，因此很多运维和开发同学都觉得用平时用的顺手的工具和手段在容器里排查问题不好使了。工欲善其事必先利其器，正是由于这样的情况，所以我们排查容器问题的时候，需要引入新的工具和手段。

0x10 容器基础

在学习工具的使用前，我们首先需要简单的了解下容器的原理。假如一台机器是一间房子，那么进程就是住在里面的一个个的人，在单体应用的时代，所有人都住在一间房子里，而容器技术就是通过一些手段把这些人都隔开，让每个人都以为自己住上有独卫（网络，IPC，namespace 等资源）的单间。而
这些隔离和限制的主要使用的如下技术：

• cgroups 资源限制
• namespace 资源隔离
• rootfs 文件系统隔离

在单体应用的时代，所有的进程都在同一个命名空间里，且启动的进程都没有隔离命名空间，那么自然调试工具进程也在同一个命名空间，也就可以 debug 其他进程。而容器技术由于分割成一个个的小房间，如果想要查看单个房间的情况，虽然在大管家（宿主机）的上帝视角一样可以看到， 但为了减少干扰并更加符合我们平时的使用习惯，我们就需要进入到房间（命名空间）里面查看。

例如我们可以通过 docker inspect CONTAINER_ID 获取到某个容器资源隔离的文件的地址，如下"SandboxKey": "/var/run/docker/netns/50def85bf6e2"就是。

@ubuntu ➜ k8s-debug  docker inspect 0dde03166e02
...
            "SandboxKey": "/var/run/docker/netns/50def85bf6e2",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "7faa1e764317cdfadf7f31b7ed2fecff62b458211f79fadb3362c8e22755f326",
            "Gateway": "172.17.0.1",                    
...

而容器的诊断工具就是自带了部分调试工具的镜像，并能根据容器 ID 帮我们自动地进入到房间（网络，IPC 命名空间等）。
在初步了解了容器的原理后，我们便可进入工具的介绍了。

0x20 netshoot

首先介绍的第一个工具是netshoot，netshoot 的自我定位就是容器网络诊断的瑞士军刀，简单来说，netshoot 其实就是一个装满了各种工具的镜像，他用起来也很简单。

• 执行docker run -it --net container:<container_name|container_id> nicolaka/netshoot 就行，这里的--net是 docker 命令指定该容器要联结到哪个容器的网络命名空间
• 如果要进入宿主机的命名空间则指定--net host就行了
• 如果要诊断 docker NIC设备的网络 情况，则可以用工具nsenter进入NIC 设备的命名空间排查，后面我会介绍这个工具

另外，如果是在 kubernetes 里面，我们可以通过执行kubectl run test-lab --generator=run-pod/v1 --rm -i --tty --overrides='{"spec": {"hostNetwork": true}}' --image nicolaka/netshoot -- /bin/bash 这个命令进入宿主机的网络。别怕这个命令长，我来一一解释下这条命令的各个选项的作用

• kubectl run test_lab --generator=pod/v1 --rm -i --tty意思是创建一个一次性的名叫test_lab的 Pod 资源并且使用标准输入输出交互
• --overrides='{"spec": {"hostNetwork": true}}'意思是使用宿主机网络，具体哪台宿主机要看这个 Pod 调度到哪个节点。
• --image nicolaka/netshoot指定 Pod 的镜像
• -- 这是 bash 的内置命令选项，是标志命令的结束的意思，举个例子：如果我想要在文件里用grep搜索-v字符串，grep -v filename中-v会被视为选项，但我如果使用grep -- -v filename那么就可以正常搜索了

0x21 演示

下面我来演示几个例子:

1. 使用 tcpdump 抓容器的包并拷贝 pcap 文件出来，便于用 wireshark 分析

   @ubuntu ➜ k8s-debug  mkdir -p /tmp/netshoot
   @ubuntu ➜ k8s-debug  docker ps
   CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS              PORTS                                          NAMES
   0dde03166e02        jumpserver/jms_all:1.5.4   "entrypoint.sh"     3 weeks ago         Up 3weeks          ...   mystifying_williamson
   @ubuntu ➜ k8s-debug  docker run -it -v /tmp/netshoot:/tmp --net container:0dde03166e02  nicolaka/netshoot
   Welcome to Netshoot! (github.com/nicolaka/netshoot)               
   root @ / 
   [1]   → tcpdump -nn -i any -w /tmp/pkg.pcap
   [2]   → exit
   

   具体命令使用与之前的差不多，只不过把宿主机上的/tmp/netshoot目录 bind-mount 到了容器的/tmp目录
2. 有时候我们还需要调试 bridge 或者 overlay 网络，可以使用 nsenter，nsenter 可以进入任何命名空间

   @ubuntu ➜ ~  docker network ls
   NETWORK ID          NAME                DRIVER              SCOPE
   ...
   0ipu2p43c6jh        ingress             overlay             swarm
   697402c52a87        none                null                local
   @ubuntu ➜ ~   docker run -it --rm -v /var/run/docker/netns:/var/run/docker/netns --privileged=true nicolaka/netshoot
   Welcome to Netshoot! (github.com/nicolaka/netshoot)
   root @ /
   [1] 🐳  → ls /var/run/docker/netns/
   1-0ipu2p43c6  50def85bf6e2  83f9ffa847d7  default       ingress_sbox
   root @ /var/run/docker/netns
   [6] 🐳  → nsenter --net=/var/run/docker/netns/1-0ipu2p43c6 sh
   root @ /run/docker/netns
   [#] 🐳  → ifconfig
   br0     Link encap:Ethernet  HWaddr 02:61:F2:E4:26:3B
           inet addr:10.255.0.1  Bcast:10.255.255.255  Mask:255.255.0.0
           UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
   ...
   vxlan0  Link encap:Ethernet  HWaddr 02:61:F2:E4:26:3B
           UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
   root @ /run/docker/netns
   [#] 🐳  → bridge fdb show br0
   33:33:00:00:00:01 dev br0 self permanent
   01:00:5e:00:00:01 dev br0 self permanent
   02:61:f2:e4:26:3b dev vxlan0 master br0 permanent
   ...
   

   上面的命令首先我们是进入了1-0ipu2p43c6的命名空间，即那个叫 ingress 的 overlay 网络，然后可以通过查看这个 NIC 设备上的fdb 表
3. 我们也可以通过挂载 docker 的 unix sock 文件查看容器的 metrics

   docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock nicolaka/netshoot ctop
   

   如下图所示
   
   netshoot 工具非常强大，还有很多功能可以自行去探索，netshoot上有详细的说明。

0x30 docker-debug

上面介绍的 netshoot 主要定位于 docker 网络的诊断，从名字就可以看出来。而我们现在介绍的工具 docker-debug可以说是 netshoot 的升级版，他不仅可以进入目标容器的网络命名空间，还可以进入 pid,user,filesystem,ipc 的命名空间，所以我们可以操作的空间就更大了。话不多说，我们开始演示。

0x31 安装

首先我们要下载 docker-debug 的二进制文件

@ubuntu ➜ docker-debug  wget docker-debug https://github.com/zeromake/docker-debug/releases/download/0.6.2/docker-debug-linux-amd64 -O docker-debug
@ubuntu ➜ docker-debug  chmod +x docker-debug
@ubuntu ➜ docker-debug  mv docker-debug /usr/bin
@ubuntu ➜ docker-debug  docker-debug info
Version:	0.6.2
Platform:	TravisLinux
Commit:		cf4cc41
Time:		2019-06-20 05:40:52 +0000

然后我赋予了文件执行权限并移动到/usr/bin目录下，如果执行docker-debug info看到有正确输出，则说明安装成功了

0x32 使用

使用就很简单了，首先我们获取到容器的名字或者容器 ID

@ubuntu ➜ docker-debug  docker ps
CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS              PORTS                                          NAMES
0dde03166e02        jumpserver/jms_all:1.5.4   "entrypoint.sh"     3 weeks ago         Up 3 weeks          ...   mystifying_williamson

然后执行docker-debug <CONTAINER_ID|CONTAINER_NAME> COMMAND就可以了

@ubuntu ➜ docker-debug  docker-debug 0dde03166e02 bash
bash-5.0# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
...
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      52/python3.6
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      110/java
...
bash-5.0# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:32776325 errors:0 dropped:0 overruns:0 frame:0
...
bash-5.0# ls /mnt/container/
anaconda-post.log  dev                lib                mnt                root               srv                usr
bin                etc                lib64              opt                run                sys                var
config             home               media              proc               sbin               tmp

我们可以看到已经进入了目标容器的 ipc,network,filesystem,pid 的命名空间了，而目标容器的root则挂载在了/mnt/container目录下。
此外，我们还可以通过设置 docker-debug 在~/.docker-debug/config.toml的配置文件使用自定义的诊断镜像

version = "0.6.1"
image = "nicolaka/netshoot:latest"
mount_dir = "/mnt/container"
timeout = 10000000000
config_default = "default"

[config]
  [config.default]
    host = "unix:///var/run/docker.sock"
    tls = false
    cert_dir = ""
    cert_password = ""

其他就不做过多介绍了

0x40 kubectl-debug

在介绍了 docker 的 debug 的工具后，我们了解了容器诊断工具的原理和使用，接下来我们要学习 kubernetes 的容器诊断工具。虽然 kubernetes 上也可以用我上面介绍的那些工具，但 kubernetes 上的容器毕竟运行在不同的 node 上，用起来就不太方便，所以就要用到 kubectl-debug 这个工具了。

kubectl-debug 其实就是一个 kubectl 的插件，他的原理和 docker 容器诊断工具大同小异。kubectl-debug 可以帮我们在 某个 Pod 的节点上起一个容器，并将这个容器加入到目标容器的pid,network,user,icp 的命名空间。kubectl-debug 架构主要可以分为两部分：

• 客户端：kubectl-debug 二进制文件
• 服务端：agent 容器
  客户端通过控制 node 上的 agent 服务端与容器运行时通信，从而启动一个容器并进入到指定 Pod 的命名空间，可以说 agent 就是一个 debug 容器与客户端之间的中继。而从 kubectl-debug 的工作模式来看，可以分为两种模式：
• 非常驻服务端：agentless
• 常驻服务端： DaemonSet
  简单来说就是 agentless 模式只有在每次 kubectl-debug 进行调试 Pod 的时候才会启动一个 agent 服务端，调试完成后自动清理 agent，此模式的优点是不那么占用 kubernetes 集群资源，而 DaemonSet 模式就是在每个节点上都会常驻一个 DaemonSet 的 agent， 好处就是启动快。
  此外针对 node 节点无法直接访问的情况，kubectl-debug 还有一个 port-forward 模式，这里就不多介绍了。

由于 kubectl-debug 可能还不太完善，agentless 模式我这里用不了，所以我用的是 DaemonSet 模式，下面开始演示。

0x41 安装客户端

安装过程和 docker-debug 差不多

1. 下载二进制文件: wget https://github.com/aylei/kubectl-debug/releases/download/v0.1.0/kubectl-debug_0.1.0_linux_amd64.tar.gz -O kubectl-debug.tar.gz
2. 解压文件: tar -zxvf kubectl-debug.tar.gz kubectl-debug

0x42 安装 agent 服务端

1. 下载 DaemonSet 的 yaml 文件：wget -f https://raw.githubusercontent.com/aylei/kubectl-debug/master/scripts/agent_daemonset.yml
2. 修改agent_daemonset.yml 文件

     ...
     18       hostNetwork: true  # 需要加上 hostNetwork: true，hostPort：10027 才会生效
     19       hostPID: true
     20       tolerations:
     21         - key: node-role.kubernetes.io/master
     22           effect: NoSchedule
     23       containers:
     24         - name: debug-agent
     25           image: aylei/debug-agent:v0.1.1 # 老版本镜像有问题，使用 v0.1.1新版本
     ...
     39           ports:
     40             - containerPort: 10027
     41               hostPort: 10027
     ...
   

3. 创建 DaemonSet：kubectl apply -f agent_daemonset.yaml, 接下来我们可以看到每个节点上都创建了 debug-agent 的 DaemonSet，并且宿主机上都监听了10027端口。

    root @ master ➜  k8s-debug   kubectl get pods
    NAME                                   READY     STATUS    RESTARTS   AGE
    debug-agent-5gfk6                      1/1       Running   0          22h
    ...
    root @ master ➜  k8s-debug  netstat -lntp | grep 10027
    tcp6       0      0 :::10027                :::*                LISTEN      15510/debug-agent
   

4. 执行命令kubectl-debug <POD_NAME>就可以进行调试了

root @ master ➜  k8s-debug  kubectl get pods
NAME                             READY     STATUS             RESTARTS   AGE
licai-gwapi-77465b4c66-hdjlb     1/1       Running            0          3d
...
root @ master ➜  k8s-debug  kubectl-debug licai-gwapi-77465b4c66-hdjlb --agentless=false --port-forward=false
pulling image nicolaka/netshoot:latest...
...
bash-5.0# ps -ef | grep java
1 root     23:24 /usr/local/openjdk-8/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties...
192 root      0:00 grep java
bash-5.0# exit
exit
root @ master ➜  k8s-debug

我们可以看到已经进入了目标容器的命名空间了，而kubectl-debug 客户端正是与每个 node 上的 10027 端口通信来控制 agent 对 Pod 的调试。
除了这些之外，kubectl-debug 还有很多配置可以自定义，kubectl-debug页面也有详细的介绍，至此从 docker 到 kubernetes 的调试工具介绍完成了。