1# Multi-Cluster Management
2 
3---
4 
5## Cluster API
6 
7### Installation
8 
9```bash
10# Install clusterctl CLI
11curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.6.0/clusterctl-linux-amd64 -o clusterctl
12chmod +x clusterctl && sudo mv clusterctl /usr/local/bin/
13 
14# Initialize management cluster with AWS provider
15clusterctl init --infrastructure aws
16 
17# Initialize with multiple providers
18clusterctl init \
19  --infrastructure aws,azure \
20  --control-plane kubeadm \
21  --bootstrap kubeadm
22```
23 
24### Cluster Definition
25 
26```yaml
27apiVersion: cluster.x-k8s.io/v1beta1
28kind: Cluster
29metadata:
30  name: production-cluster
31  namespace: clusters
32spec:
33  clusterNetwork:
34    pods:
35      cidrBlocks: ["192.168.0.0/16"]
36    services:
37      cidrBlocks: ["10.96.0.0/12"]
38  controlPlaneRef:
39    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
40    kind: KubeadmControlPlane
41    name: production-control-plane
42  infrastructureRef:
43    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
44    kind: AWSCluster
45    name: production-cluster
46---
47apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
48kind: AWSCluster
49metadata:
50  name: production-cluster
51  namespace: clusters
52spec:
53  region: us-west-2
54  sshKeyName: production-key
55  network:
56    vpc:
57      cidrBlock: 10.0.0.0/16
58    subnets:
59      - availabilityZone: us-west-2a
60        cidrBlock: 10.0.1.0/24
61        isPublic: true
62      - availabilityZone: us-west-2b
63        cidrBlock: 10.0.2.0/24
64        isPublic: true
65```
66 
67### Control Plane
68 
69```yaml
70apiVersion: controlplane.cluster.x-k8s.io/v1beta1
71kind: KubeadmControlPlane
72metadata:
73  name: production-control-plane
74  namespace: clusters
75spec:
76  replicas: 3
77  version: v1.28.0
78  machineTemplate:
79    infrastructureRef:
80      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
81      kind: AWSMachineTemplate
82      name: production-control-plane
83  kubeadmConfigSpec:
84    clusterConfiguration:
85      apiServer:
86        extraArgs:
87          cloud-provider: aws
88      controllerManager:
89        extraArgs:
90          cloud-provider: aws
91    initConfiguration:
92      nodeRegistration:
93        kubeletExtraArgs:
94          cloud-provider: aws
95    joinConfiguration:
96      nodeRegistration:
97        kubeletExtraArgs:
98          cloud-provider: aws
99```
100 
101### Machine Deployment (Worker Nodes)
102 
103```yaml
104apiVersion: cluster.x-k8s.io/v1beta1
105kind: MachineDeployment
106metadata:
107  name: production-workers
108  namespace: clusters
109spec:
110  clusterName: production-cluster
111  replicas: 5
112  selector:
113    matchLabels:
114      cluster.x-k8s.io/cluster-name: production-cluster
115  template:
116    spec:
117      clusterName: production-cluster
118      version: v1.28.0
119      bootstrap:
120        configRef:
121          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
122          kind: KubeadmConfigTemplate
123          name: production-workers
124      infrastructureRef:
125        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
126        kind: AWSMachineTemplate
127        name: production-workers
128---
129apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
130kind: AWSMachineTemplate
131metadata:
132  name: production-workers
133  namespace: clusters
134spec:
135  template:
136    spec:
137      instanceType: m5.xlarge
138      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
139      sshKeyName: production-key
140      rootVolume:
141        size: 100
142        type: gp3
143```
144 
145## Cross-Cluster Networking
146 
147### Submariner Installation
148 
149```bash
150# Install subctl
151curl -Ls https://get.submariner.io | bash
152 
153# Join clusters to broker
154subctl deploy-broker --kubeconfig kubeconfig-cluster1
155 
156# Join workload clusters
157subctl join --kubeconfig kubeconfig-cluster1 broker-info.subm --clusterid cluster1
158subctl join --kubeconfig kubeconfig-cluster2 broker-info.subm --clusterid cluster2
159 
160# Verify connectivity
161subctl show all
162```
163 
164### ServiceExport/ServiceImport
165 
166```yaml
167# Export service from cluster1
168apiVersion: multicluster.x-k8s.io/v1alpha1
169kind: ServiceExport
170metadata:
171  name: myapp
172  namespace: production
173---
174# Service is auto-imported to other clusters as:
175# myapp.production.svc.clusterset.local
176```
177 
178### Cilium Cluster Mesh
179 
180```bash
181# Enable cluster mesh on both clusters
182cilium clustermesh enable --context cluster1
183cilium clustermesh enable --context cluster2
184 
185# Connect clusters
186cilium clustermesh connect --context cluster1 --destination-context cluster2
187 
188# Verify
189cilium clustermesh status --context cluster1
190```
191 
192```yaml
193# Global service accessible from all clusters
194apiVersion: v1
195kind: Service
196metadata:
197  name: myapp
198  namespace: production
199  annotations:
200    service.cilium.io/global: "true"
201spec:
202  type: ClusterIP
203  selector:
204    app: myapp
205  ports:
206    - port: 80
207```
208 
209## Multi-Cluster DNS
210 
211### ExternalDNS with Route53
212 
213```yaml
214apiVersion: apps/v1
215kind: Deployment
216metadata:
217  name: external-dns
218  namespace: kube-system
219spec:
220  template:
221    spec:
222      containers:
223        - name: external-dns
224          image: k8s.gcr.io/external-dns/external-dns:v0.14.0
225          args:
226            - --source=service
227            - --source=ingress
228            - --provider=aws
229            - --aws-zone-type=public
230            - --registry=txt
231            - --txt-owner-id=my-cluster
232            - --domain-filter=example.com
233```
234 
235### CoreDNS Federation
236 
237```yaml
238# Forward queries for other clusters
239apiVersion: v1
240kind: ConfigMap
241metadata:
242  name: coredns
243  namespace: kube-system
244data:
245  Corefile: |
246    .:53 {
247        errors
248        health
249        kubernetes cluster.local in-addr.arpa ip6.arpa {
250           pods insecure
251           fallthrough in-addr.arpa ip6.arpa
252        }
253        # Forward to cluster2 DNS
254        cluster2.local:53 {
255            forward . 10.0.0.10
256        }
257        forward . /etc/resolv.conf
258        cache 30
259        loop
260        reload
261        loadbalance
262    }
263```
264 
265## Workload Distribution
266 
267### Kubernetes Federation v2
268 
269```yaml
270apiVersion: types.kubefed.io/v1beta1
271kind: FederatedDeployment
272metadata:
273  name: myapp
274  namespace: production
275spec:
276  template:
277    metadata:
278      labels:
279        app: myapp
280    spec:
281      replicas: 3
282      selector:
283        matchLabels:
284          app: myapp
285      template:
286        metadata:
287          labels:
288            app: myapp
289        spec:
290          containers:
291            - name: myapp
292              image: myregistry.io/myapp:v1.0.0
293  placement:
294    clusters:
295      - name: cluster-us-west
296      - name: cluster-us-east
297      - name: cluster-eu-west
298  overrides:
299    - clusterName: cluster-us-west
300      clusterOverrides:
301        - path: "/spec/replicas"
302          value: 5
303    - clusterName: cluster-eu-west
304      clusterOverrides:
305        - path: "/spec/replicas"
306          value: 3
307```
308 
309### ArgoCD Multi-Cluster
310 
311```yaml
312apiVersion: argoproj.io/v1alpha1
313kind: ApplicationSet
314metadata:
315  name: myapp-global
316  namespace: argocd
317spec:
318  generators:
319    - clusters:
320        selector:
321          matchLabels:
322            environment: production
323  template:
324    metadata:
325      name: 'myapp-{{name}}'
326    spec:
327      project: default
328      source:
329        repoURL: https://github.com/myorg/myapp-manifests.git
330        targetRevision: main
331        path: overlays/production
332      destination:
333        server: '{{server}}'
334        namespace: production
335      syncPolicy:
336        automated:
337          prune: true
338          selfHeal: true
339```
340 
341## Disaster Recovery
342 
343### Velero Backup Configuration
344 
345```bash
346# Install Velero with S3
347velero install \
348  --provider aws \
349  --plugins velero/velero-plugin-for-aws:v1.8.0 \
350  --bucket velero-backups \
351  --backup-location-config region=us-west-2 \
352  --snapshot-location-config region=us-west-2 \
353  --secret-file ./credentials-velero
354```
355 
356```yaml
357# Scheduled backup
358apiVersion: velero.io/v1
359kind: Schedule
360metadata:
361  name: daily-backup
362  namespace: velero
363spec:
364  schedule: "0 2 * * *"
365  template:
366    includedNamespaces:
367      - production
368      - staging
369    excludedResources:
370      - events
371    storageLocation: default
372    volumeSnapshotLocations:
373      - default
374    ttl: 720h  # 30 days
375---
376# Restore to different cluster
377apiVersion: velero.io/v1
378kind: Restore
379metadata:
380  name: restore-production
381  namespace: velero
382spec:
383  backupName: daily-backup-20240115
384  includedNamespaces:
385    - production
386  restorePVs: true
387  preserveNodePorts: true
388```
389 
390### Active-Passive Failover
391 
392```yaml
393# Primary cluster ingress
394apiVersion: networking.k8s.io/v1
395kind: Ingress
396metadata:
397  name: myapp
398  annotations:
399    external-dns.alpha.kubernetes.io/hostname: myapp.example.com
400    external-dns.alpha.kubernetes.io/set-identifier: primary
401    external-dns.alpha.kubernetes.io/aws-weight: "100"
402spec:
403  rules:
404    - host: myapp.example.com
405      http:
406        paths:
407          - path: /
408            pathType: Prefix
409            backend:
410              service:
411                name: myapp
412                port:
413                  number: 80
414---
415# Secondary cluster ingress
416apiVersion: networking.k8s.io/v1
417kind: Ingress
418metadata:
419  name: myapp
420  annotations:
421    external-dns.alpha.kubernetes.io/hostname: myapp.example.com
422    external-dns.alpha.kubernetes.io/set-identifier: secondary
423    external-dns.alpha.kubernetes.io/aws-weight: "0"
424spec:
425  rules:
426    - host: myapp.example.com
427      # ... same backend config
428```
429 
430## Centralized Management Tools
431 
432### Rancher Setup
433 
434```bash
435# Install Rancher with Helm
436helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
437helm install rancher rancher-stable/rancher \
438  --namespace cattle-system \
439  --create-namespace \
440  --set hostname=rancher.example.com \
441  --set bootstrapPassword=admin
442```
443 
444### Kubeconfig Management
445 
446```yaml
447# Merge multiple kubeconfigs
448# ~/.kube/config
449apiVersion: v1
450kind: Config
451clusters:
452  - name: cluster-us-west
453    cluster:
454      server: https://cluster-us-west.example.com
455      certificate-authority-data: ...
456  - name: cluster-us-east
457    cluster:
458      server: https://cluster-us-east.example.com
459      certificate-authority-data: ...
460contexts:
461  - name: us-west
462    context:
463      cluster: cluster-us-west
464      user: admin-us-west
465      namespace: default
466  - name: us-east
467    context:
468      cluster: cluster-us-east
469      user: admin-us-east
470      namespace: default
471users:
472  - name: admin-us-west
473    user:
474      token: ...
475  - name: admin-us-east
476    user:
477      token: ...
478current-context: us-west
479```
480 
481```bash
482# Switch between clusters
483kubectl config use-context us-west
484kubectl config use-context us-east
485 
486# Run command against specific cluster
487kubectl --context=us-west get pods
488kubectl --context=us-east get pods
489 
490# Use kubectx for easier switching
491kubectx us-west
492```
493 
494## Best Practices
495 
4961. **Use Cluster API** for declarative cluster lifecycle management
4972. **Implement service mesh** for secure cross-cluster communication
4983. **Set up DNS-based routing** for global service discovery
4994. **Configure automated backups** with Velero across clusters
5005. **Use GitOps** (ArgoCD/Flux) for consistent multi-cluster deployments
5016. **Implement network policies** consistently across clusters
5027. **Centralize observability** with cross-cluster metrics and logs
5038. **Test failover procedures** regularly
5049. **Use namespaces consistently** across clusters
50510. **Document cluster topology** and dependencies
50611. **Implement RBAC** with cross-cluster access patterns
50712. **Monitor cluster health** from centralized dashboard
508