1# Custom Operators
2 
3---
4 
5## CustomResourceDefinition (CRD)
6 
7```yaml
8apiVersion: apiextensions.k8s.io/v1
9kind: CustomResourceDefinition
10metadata:
11  name: databases.mycompany.io
12spec:
13  group: mycompany.io
14  names:
15    kind: Database
16    listKind: DatabaseList
17    plural: databases
18    singular: database
19    shortNames:
20      - db
21  scope: Namespaced
22  versions:
23    - name: v1
24      served: true
25      storage: true
26      schema:
27        openAPIV3Schema:
28          type: object
29          required:
30            - spec
31          properties:
32            spec:
33              type: object
34              required:
35                - engine
36                - version
37                - storage
38              properties:
39                engine:
40                  type: string
41                  enum: [postgres, mysql, mongodb]
42                version:
43                  type: string
44                storage:
45                  type: string
46                  pattern: '^[0-9]+Gi$'
47                replicas:
48                  type: integer
49                  minimum: 1
50                  maximum: 5
51                  default: 1
52            status:
53              type: object
54              properties:
55                phase:
56                  type: string
57                  enum: [Pending, Creating, Running, Failed, Terminating]
58                ready:
59                  type: boolean
60                message:
61                  type: string
62                endpoint:
63                  type: string
64      subresources:
65        status: {}
66        scale:
67          specReplicasPath: .spec.replicas
68          statusReplicasPath: .status.replicas
69      additionalPrinterColumns:
70        - name: Engine
71          type: string
72          jsonPath: .spec.engine
73        - name: Version
74          type: string
75          jsonPath: .spec.version
76        - name: Status
77          type: string
78          jsonPath: .status.phase
79        - name: Age
80          type: date
81          jsonPath: .metadata.creationTimestamp
82```
83 
84## Custom Resource Instance
85 
86```yaml
87apiVersion: mycompany.io/v1
88kind: Database
89metadata:
90  name: orders-db
91  namespace: production
92spec:
93  engine: postgres
94  version: "15.4"
95  storage: 100Gi
96  replicas: 3
97```
98 
99## Operator SDK Project Structure
100 
101```
102my-operator/
103├── Dockerfile
104├── Makefile
105├── PROJECT                     # Kubebuilder project config
106├── config/
107│   ├── crd/                    # CRD manifests
108│   │   └── bases/
109│   │       └── mycompany.io_databases.yaml
110│   ├── manager/                # Operator deployment
111│   │   └── manager.yaml
112│   ├── rbac/                   # RBAC configuration
113│   │   ├── role.yaml
114│   │   ├── role_binding.yaml
115│   │   └── service_account.yaml
116│   └── samples/                # Example CRs
117│       └── mycompany_v1_database.yaml
118├── api/
119│   └── v1/
120│       ├── database_types.go   # API type definitions
121│       ├── groupversion_info.go
122│       └── zz_generated.deepcopy.go
123├── controllers/
124│   └── database_controller.go  # Reconciliation logic
125├── main.go                     # Entry point
126└── go.mod
127```
128 
129## API Type Definition (Go)
130 
131```go
132// api/v1/database_types.go
133package v1
134 
135import (
136	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
137)
138 
139// DatabaseSpec defines the desired state of Database
140type DatabaseSpec struct {
141	// Engine is the database engine type
142	// +kubebuilder:validation:Enum=postgres;mysql;mongodb
143	Engine string `json:"engine"`
144 
145	// Version is the database version
146	Version string `json:"version"`
147 
148	// Storage is the size of persistent storage
149	// +kubebuilder:validation:Pattern=`^[0-9]+Gi$`
150	Storage string `json:"storage"`
151 
152	// Replicas is the number of database instances
153	// +kubebuilder:validation:Minimum=1
154	// +kubebuilder:validation:Maximum=5
155	// +kubebuilder:default=1
156	// +optional
157	Replicas int32 `json:"replicas,omitempty"`
158}
159 
160// DatabaseStatus defines the observed state of Database
161type DatabaseStatus struct {
162	// Phase represents the current lifecycle phase
163	Phase string `json:"phase,omitempty"`
164 
165	// Ready indicates if the database is ready to accept connections
166	Ready bool `json:"ready,omitempty"`
167 
168	// Message provides additional status information
169	Message string `json:"message,omitempty"`
170 
171	// Endpoint is the connection endpoint
172	Endpoint string `json:"endpoint,omitempty"`
173 
174	// Replicas is the current number of running replicas
175	Replicas int32 `json:"replicas,omitempty"`
176}
177 
178// +kubebuilder:object:root=true
179// +kubebuilder:subresource:status
180// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas
181// +kubebuilder:printcolumn:name="Engine",type=string,JSONPath=`.spec.engine`
182// +kubebuilder:printcolumn:name="Version",type=string,JSONPath=`.spec.version`
183// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.phase`
184// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
185 
186// Database is the Schema for the databases API
187type Database struct {
188	metav1.TypeMeta   `json:",inline"`
189	metav1.ObjectMeta `json:"metadata,omitempty"`
190 
191	Spec   DatabaseSpec   `json:"spec,omitempty"`
192	Status DatabaseStatus `json:"status,omitempty"`
193}
194 
195// +kubebuilder:object:root=true
196 
197// DatabaseList contains a list of Database
198type DatabaseList struct {
199	metav1.TypeMeta `json:",inline"`
200	metav1.ListMeta `json:"metadata,omitempty"`
201	Items           []Database `json:"items"`
202}
203 
204func init() {
205	SchemeBuilder.Register(&Database{}, &DatabaseList{})
206}
207```
208 
209## Controller Implementation
210 
211```go
212// controllers/database_controller.go
213package controllers
214 
215import (
216	"context"
217	"fmt"
218	"time"
219 
220	appsv1 "k8s.io/api/apps/v1"
221	corev1 "k8s.io/api/core/v1"
222	"k8s.io/apimachinery/pkg/api/errors"
223	"k8s.io/apimachinery/pkg/api/resource"
224	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
225	"k8s.io/apimachinery/pkg/runtime"
226	"k8s.io/apimachinery/pkg/types"
227	ctrl "sigs.k8s.io/controller-runtime"
228	"sigs.k8s.io/controller-runtime/pkg/client"
229	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
230	"sigs.k8s.io/controller-runtime/pkg/log"
231 
232	mycompanyv1 "github.com/mycompany/database-operator/api/v1"
233)
234 
235const databaseFinalizer = "databases.mycompany.io/finalizer"
236 
237type DatabaseReconciler struct {
238	client.Client
239	Scheme *runtime.Scheme
240}
241 
242// +kubebuilder:rbac:groups=mycompany.io,resources=databases,verbs=get;list;watch;create;update;patch;delete
243// +kubebuilder:rbac:groups=mycompany.io,resources=databases/status,verbs=get;update;patch
244// +kubebuilder:rbac:groups=mycompany.io,resources=databases/finalizers,verbs=update
245// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
246// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
247// +kubebuilder:rbac:groups=core,resources=persistentvolumeclaims,verbs=get;list;watch
248 
249func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
250	logger := log.FromContext(ctx)
251 
252	// Fetch the Database instance
253	database := &mycompanyv1.Database{}
254	if err := r.Get(ctx, req.NamespacedName, database); err != nil {
255		if errors.IsNotFound(err) {
256			return ctrl.Result{}, nil
257		}
258		return ctrl.Result{}, err
259	}
260 
261	// Handle deletion with finalizer
262	if !database.DeletionTimestamp.IsZero() {
263		if controllerutil.ContainsFinalizer(database, databaseFinalizer) {
264			if err := r.cleanupResources(ctx, database); err != nil {
265				return ctrl.Result{}, err
266			}
267			controllerutil.RemoveFinalizer(database, databaseFinalizer)
268			if err := r.Update(ctx, database); err != nil {
269				return ctrl.Result{}, err
270			}
271		}
272		return ctrl.Result{}, nil
273	}
274 
275	// Add finalizer if not present
276	if !controllerutil.ContainsFinalizer(database, databaseFinalizer) {
277		controllerutil.AddFinalizer(database, databaseFinalizer)
278		if err := r.Update(ctx, database); err != nil {
279			return ctrl.Result{}, err
280		}
281	}
282 
283	// Reconcile StatefulSet
284	statefulSet := r.buildStatefulSet(database)
285	if err := controllerutil.SetControllerReference(database, statefulSet, r.Scheme); err != nil {
286		return ctrl.Result{}, err
287	}
288 
289	found := &appsv1.StatefulSet{}
290	err := r.Get(ctx, types.NamespacedName{Name: statefulSet.Name, Namespace: statefulSet.Namespace}, found)
291	if err != nil && errors.IsNotFound(err) {
292		logger.Info("Creating StatefulSet", "name", statefulSet.Name)
293		if err := r.Create(ctx, statefulSet); err != nil {
294			return ctrl.Result{}, err
295		}
296		return r.updateStatus(ctx, database, "Creating", false, "StatefulSet created")
297	} else if err != nil {
298		return ctrl.Result{}, err
299	}
300 
301	// Reconcile Service
302	service := r.buildService(database)
303	if err := controllerutil.SetControllerReference(database, service, r.Scheme); err != nil {
304		return ctrl.Result{}, err
305	}
306 
307	foundSvc := &corev1.Service{}
308	err = r.Get(ctx, types.NamespacedName{Name: service.Name, Namespace: service.Namespace}, foundSvc)
309	if err != nil && errors.IsNotFound(err) {
310		if err := r.Create(ctx, service); err != nil {
311			return ctrl.Result{}, err
312		}
313	}
314 
315	// Update status based on StatefulSet state
316	if found.Status.ReadyReplicas == *found.Spec.Replicas {
317		return r.updateStatus(ctx, database, "Running", true,
318			fmt.Sprintf("%d/%d replicas ready", found.Status.ReadyReplicas, *found.Spec.Replicas))
319	}
320 
321	// Requeue to check status
322	return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
323}
324 
325func (r *DatabaseReconciler) buildStatefulSet(db *mycompanyv1.Database) *appsv1.StatefulSet {
326	replicas := db.Spec.Replicas
327	labels := map[string]string{
328		"app":        db.Name,
329		"controller": db.Name,
330	}
331 
332	return &appsv1.StatefulSet{
333		ObjectMeta: metav1.ObjectMeta{
334			Name:      db.Name,
335			Namespace: db.Namespace,
336		},
337		Spec: appsv1.StatefulSetSpec{
338			Replicas: &replicas,
339			Selector: &metav1.LabelSelector{
340				MatchLabels: labels,
341			},
342			ServiceName: db.Name,
343			Template: corev1.PodTemplateSpec{
344				ObjectMeta: metav1.ObjectMeta{
345					Labels: labels,
346				},
347				Spec: corev1.PodSpec{
348					Containers: []corev1.Container{{
349						Name:  "database",
350						Image: fmt.Sprintf("%s:%s", db.Spec.Engine, db.Spec.Version),
351						Ports: []corev1.ContainerPort{{
352							ContainerPort: 5432,
353							Name:          "db",
354						}},
355					}},
356				},
357			},
358			VolumeClaimTemplates: []corev1.PersistentVolumeClaim{{
359				ObjectMeta: metav1.ObjectMeta{
360					Name: "data",
361				},
362				Spec: corev1.PersistentVolumeClaimSpec{
363					AccessModes: []corev1.PersistentVolumeAccessMode{
364						corev1.ReadWriteOnce,
365					},
366					Resources: corev1.VolumeResourceRequirements{
367						Requests: corev1.ResourceList{
368							corev1.ResourceStorage: resource.MustParse(db.Spec.Storage),
369						},
370					},
371				},
372			}},
373		},
374	}
375}
376 
377func (r *DatabaseReconciler) buildService(db *mycompanyv1.Database) *corev1.Service {
378	return &corev1.Service{
379		ObjectMeta: metav1.ObjectMeta{
380			Name:      db.Name,
381			Namespace: db.Namespace,
382		},
383		Spec: corev1.ServiceSpec{
384			Selector: map[string]string{"app": db.Name},
385			Ports: []corev1.ServicePort{{
386				Port: 5432,
387				Name: "db",
388			}},
389			ClusterIP: "None", // Headless service for StatefulSet
390		},
391	}
392}
393 
394func (r *DatabaseReconciler) updateStatus(ctx context.Context, db *mycompanyv1.Database,
395	phase string, ready bool, message string) (ctrl.Result, error) {
396	db.Status.Phase = phase
397	db.Status.Ready = ready
398	db.Status.Message = message
399	db.Status.Endpoint = fmt.Sprintf("%s.%s.svc.cluster.local:5432", db.Name, db.Namespace)
400	return ctrl.Result{}, r.Status().Update(ctx, db)
401}
402 
403func (r *DatabaseReconciler) cleanupResources(ctx context.Context, db *mycompanyv1.Database) error {
404	// Custom cleanup logic (e.g., backup before deletion)
405	return nil
406}
407 
408func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
409	return ctrl.NewControllerManagedBy(mgr).
410		For(&mycompanyv1.Database{}).
411		Owns(&appsv1.StatefulSet{}).
412		Owns(&corev1.Service{}).
413		Complete(r)
414}
415```
416 
417## Operator RBAC
418 
419```yaml
420apiVersion: v1
421kind: ServiceAccount
422metadata:
423  name: database-operator
424  namespace: operators
425---
426apiVersion: rbac.authorization.k8s.io/v1
427kind: ClusterRole
428metadata:
429  name: database-operator-role
430rules:
431  - apiGroups: ["mycompany.io"]
432    resources: ["databases"]
433    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
434  - apiGroups: ["mycompany.io"]
435    resources: ["databases/status"]
436    verbs: ["get", "update", "patch"]
437  - apiGroups: ["mycompany.io"]
438    resources: ["databases/finalizers"]
439    verbs: ["update"]
440  - apiGroups: ["apps"]
441    resources: ["statefulsets"]
442    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
443  - apiGroups: [""]
444    resources: ["services", "configmaps", "secrets"]
445    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
446  - apiGroups: [""]
447    resources: ["persistentvolumeclaims"]
448    verbs: ["get", "list", "watch"]
449  - apiGroups: [""]
450    resources: ["events"]
451    verbs: ["create", "patch"]
452---
453apiVersion: rbac.authorization.k8s.io/v1
454kind: ClusterRoleBinding
455metadata:
456  name: database-operator-rolebinding
457roleRef:
458  apiGroup: rbac.authorization.k8s.io
459  kind: ClusterRole
460  name: database-operator-role
461subjects:
462  - kind: ServiceAccount
463    name: database-operator
464    namespace: operators
465```
466 
467## Operator Deployment
468 
469```yaml
470apiVersion: apps/v1
471kind: Deployment
472metadata:
473  name: database-operator
474  namespace: operators
475  labels:
476    app: database-operator
477spec:
478  replicas: 1
479  selector:
480    matchLabels:
481      app: database-operator
482  template:
483    metadata:
484      labels:
485        app: database-operator
486    spec:
487      serviceAccountName: database-operator
488      securityContext:
489        runAsNonRoot: true
490        seccompProfile:
491          type: RuntimeDefault
492      containers:
493        - name: manager
494          image: myregistry.io/database-operator:v1.0.0
495          args:
496            - --leader-elect
497            - --health-probe-bind-address=:8081
498          securityContext:
499            allowPrivilegeEscalation: false
500            capabilities:
501              drop: ["ALL"]
502            readOnlyRootFilesystem: true
503          ports:
504            - containerPort: 8080
505              name: metrics
506          livenessProbe:
507            httpGet:
508              path: /healthz
509              port: 8081
510            initialDelaySeconds: 15
511            periodSeconds: 20
512          readinessProbe:
513            httpGet:
514              path: /readyz
515              port: 8081
516            initialDelaySeconds: 5
517            periodSeconds: 10
518          resources:
519            limits:
520              cpu: 500m
521              memory: 128Mi
522            requests:
523              cpu: 10m
524              memory: 64Mi
525```
526 
527## Operator SDK Commands
528 
529```bash
530# Initialize new operator project
531operator-sdk init --domain mycompany.io --repo github.com/mycompany/database-operator
532 
533# Create new API (CRD + controller)
534operator-sdk create api --group mycompany --version v1 --kind Database --resource --controller
535 
536# Generate manifests (CRD, RBAC)
537make manifests
538 
539# Generate deep copy methods
540make generate
541 
542# Build operator image
543make docker-build docker-push IMG=myregistry.io/database-operator:v1.0.0
544 
545# Deploy to cluster
546make deploy IMG=myregistry.io/database-operator:v1.0.0
547 
548# Undeploy
549make undeploy
550```
551 
552## Best Practices
553 
5541. **Use finalizers** for cleanup of external resources before CR deletion
5552. **Set owner references** so owned resources are garbage collected with the CR
5563. **Implement idempotent reconciliation** - same input should produce same output
5574. **Use status subresource** to separate desired state (spec) from observed state (status)
5585. **Add validation** via OpenAPI schema or webhooks
5596. **Emit events** for significant state changes
5607. **Use leader election** for high availability
5618. **Set resource limits** on the operator deployment
5629. **Follow least privilege** RBAC principles
56310. **Test with envtest** for unit testing controllers
564