自定义 Kubernetes 调度：动手实验

Kubernetes 已成为容器编排的事实标准，随着其广泛应用，理解其核心组件（如调度器）已成为任何 DevOps 工程师必备的技能。调度器负责以优化资源使用、避免冲突并遵循用户定义约束的方式将 Pod 放置到节点上。虽然 Kubernetes 提供了强大的默认调度器，但在某些场景下可能需要自定义调度器。本文将带您深入了解 Kubernetes 调度器的内部工作原理，并指导您使用简单的 shell 脚本从头开始构建自定义调度器。

理解 Kubernetes 调度器

在深入终端操作之前，理解 Kube-Scheduler 的角色和功能至关重要。调度器是决定新 Pod 应该运行在哪个节点上的组件。它会评估可用节点与 Pod 的需求（如 CPU、内存和存储）是否匹配，并将 Pod 放置在最能满足这些需求的节点上。调度器还会考虑亲和性、反亲和性和污点等约束，以确保 Pod 的放置方式符合集群策略。

默认的 Kube-Scheduler 非常复杂，但它并不总是足够灵活以应对所有用例。例如，您可能需要根据特定的业务逻辑优先处理 Pod，优化网络邻近性，或执行自定义资源分配规则。在这些情况下，构建自定义调度器可能是一个可行的解决方案。

调度器的工作流程

要构建自定义调度器，了解默认调度器的工作流程很有帮助。以下是高级概述：

Pod 注册：当创建新 Pod 时，它会被注册到 Kubernetes API 服务器。API 服务器随后会通知调度器有新 Pod。
过滤：调度器接收 Pod 并过滤掉不满足 Pod 需求的节点。这包括检查资源可用性、节点亲和性、反亲和性和污点。
评分：剩余的节点根据它们与 Pod 需求的匹配程度进行评分。调度器使用评分算法为每个节点分配分数，分数越高表示匹配度越好。
调度：选择得分最高的节点来运行 Pod。调度器随后将 Pod 放置在节点上，并在 API 服务器中更新 Pod 的状态。

使用 Shell 脚本构建自定义调度器

现在，让我们逐步创建一个简单的自定义调度器，使用 shell 脚本实现。本例将重点关注一个基本的评分机制，该机制根据可用的 CPU 资源优先考虑节点。

第 1 步：设置环境

首先，确保您有一个正在运行的 Kubernetes 集群并可以访问它。您还需要安装 kubectl 以与集群交互。对于本实验，我们将使用一个简单的 shell 脚本与 Kubernetes API 交互。

#!/bin/bash

# 获取集群中的节点列表
NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}')

第 2 步：定义评分函数

接下来，定义一个函数来根据可用的 CPU 资源对每个节点进行评分。Kubernetes API 提供了获取节点指标的端点，我们将使用这些端点来确定分数。

score_node() {
    NODE_NAME=$1
    # 获取节点上的可用 CPU
    CPU_AVAIL=$(kubectl get node $NODE_NAME -o jsonpath='{.status.allocationPressure.availableCpu}')
    # 根据可用 CPU 计算简单分数
    SCORE=$((CPU_AVAIL / 1000))
    echo $SCORE
}

第 3 步：选择最佳节点

现在，迭代所有节点，对它们进行评分，并选择得分最高的节点。

# 初始化变量以跟踪最佳节点
BEST_NODE=""
BEST_SCORE=0

# 迭代所有节点
for NODE in $NODES; do
    # 评分节点
    SCORE=$(score_node $NODE)
    echo "节点 $NODE 得分 $SCORE"

    # 检查该节点是否是目前最佳节点
    if [ $SCORE -gt $BEST_SCORE ]; then
        BEST_SCORE=$SCORE
        BEST_NODE=$NODE
    fi
done

echo "Pod 的最佳节点是 $BEST_NODE，得分 $BEST_SCORE"

第 4 步：放置 Pod

最后，将 Pod 放置在选定的节点上。为了简单起见，我们只需打印放置 Pod 的命令。

# Pod 详情（通常您会从 API 中获取这些信息）
POD_NAME="my-pod"
NAMESPACE="default"

# 将 Pod 放置在最佳节点的命令
echo "kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: $POD_NAME
  namespace: $NAMESPACE
spec:
  containers:
  - name: my-container
    image: nginx
  nodeSelector:
    kubernetes.io/hostname: $BEST_NODE
EOF"

运行自定义调度器

将脚本保存为 custom-scheduler.sh 并运行它：

chmod +x custom-scheduler.sh
./custom-scheduler.sh

此脚本将根据可用的 CPU 资源输出 Pod 的最佳节点，并打印将 Pod 放置在该节点的命令。

洞察与注意事项

虽然这个示例很简单，但它展示了构建自定义 Kubernetes 调度器的核心原理。以下是一些洞察和注意事项：

评分机制：评分函数可以通过包含其他指标（如内存、存储、网络带宽和节点标签）来变得更加复杂。您还可以实现自定义业务逻辑，根据特定需求优先考虑某些节点。
API 交互：对于生产就绪的调度器，您需要更健壮地与 Kubernetes API 交互。这包括处理 API 身份验证、错误处理和异步操作。
性能：自定义调度器应设计为注重性能。调度过程应高效，以避免 Pod 放置延迟，这会影响应用程序的可用性和性能。
测试：在生产环境中部署自定义调度器之前，在受控环境中彻底测试它。确保它处理各种边缘情况，并与其他集群组件良好集成。

总结

构建自定义 Kubernetes 调度器是针对特定需求定制 Pod 放置的一种强大方式。虽然默认调度器功能强大，但自定义调度器提供了灵活性，以优化资源使用、执行自定义策略并与独特的业务需求保持一致。通过理解调度器的工作流程并利用 shell 脚本和 Kubernetes API 等工具，您可以创建满足特定用例的调度器。这种动手方法不仅加深了您对 Kubernetes 内部的理解，还使您能够构建更高效、更有效的容器编排解决方案。

Customizing Kubernetes Scheduling: A Hands-On Lab

Kubernetes has become the de facto standard for container orchestration, and with its widespread adoption, understanding its core components—like the scheduler—has become essential for any DevOps engineer. The scheduler is responsible for placing pods onto nodes in a way that optimizes resource usage, avoids conflicts, and adheres to user-defined constraints. While Kubernetes provides a robust default scheduler, there are scenarios where a custom scheduler might be needed. This article takes you through the inner workings of the Kubernetes scheduler and guides you through building a custom scheduler from scratch using simple shell scripts.

Understanding the Kubernetes Scheduler

Before diving into the terminal, it's crucial to grasp the role and functionality of the Kube-Scheduler. The scheduler is the component that decides which node a new pod should run on. It evaluates available nodes against pod requirements—such as CPU, memory, and storage—and places the pod on the node that best meets those requirements. The scheduler also considers constraints like affinity, anti-affinity, and taints to ensure that pods are placed in a manner that aligns with the cluster's policies.

The default Kube-Scheduler is highly sophisticated, but it's not always flexible enough to handle every use case. For example, you might need to prioritize pods based on specific business logic, optimize for network proximity, or enforce custom resource allocation rules. In such cases, building a custom scheduler can be a viable solution.

The Scheduler's Workflow

To build a custom scheduler, it's helpful to understand the default scheduler's workflow. Here's a high-level overview:

Pod Registration: When a new pod is created, it's registered with the Kubernetes API server. The API server then notifies the scheduler of the new pod.
Filtering: The scheduler receives the pod and filters out nodes that don't meet the pod's requirements. This includes checking resource availability, node affinity, anti-affinity, and taints.
Scoring: The remaining nodes are scored based on how well they match the pod's requirements. The scheduler uses a scoring algorithm to assign a score to each node, with higher scores indicating better matches.
Scheduling: The node with the highest score is selected to run the pod. The scheduler then places the pod on the node and updates the pod's status in the API server.

Building a Custom Scheduler with Shell Scripts

Now, let's walk through creating a simple custom scheduler using shell scripts. This example will focus on a basic scoring mechanism that prioritizes nodes based on available CPU resources.

Step 1: Setting Up the Environment

First, ensure you have a Kubernetes cluster running and access to it. You'll also need kubectl installed to interact with the cluster. For this lab, we'll use a simple shell script to interact with the Kubernetes API.

#!/bin/bash

# Get the list of nodes in the cluster
NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}')

Step 2: Defining the Scoring Function

Next, define a function to score each node based on available CPU resources. The Kubernetes API provides endpoints to fetch node metrics, which we'll use to determine the score.

score_node() {
    NODE_NAME=$1
    # Get CPU available on the node
    CPU_AVAIL=$(kubectl get node $NODE_NAME -o jsonpath='{.status.allocationPressure.availableCpu}')
    # Calculate a simple score based on available CPU
    SCORE=$((CPU_AVAIL / 1000))
    echo $SCORE
}

Step 3: Selecting the Best Node

Now, iterate over all nodes, score them, and select the node with the highest score.

# Initialize variables to track the best node
BEST_NODE=""
BEST_SCORE=0

# Iterate over all nodes
for NODE in $NODES; do
    # Score the node
    SCORE=$(score_node $NODE)
    echo "Node $NODE scored $SCORE"

    # Check if this node is the best so far
    if [ $SCORE -gt $BEST_SCORE ]; then
        BEST_SCORE=$SCORE
        BEST_NODE=$NODE
    fi
done

echo "Best node for the pod is $BEST_NODE with score $BEST_SCORE"

Step 4: Placing the Pod

Finally, place the pod on the selected node. For simplicity, we'll just print the command that would place the pod.

# Pod details (you would typically fetch these from the API)
POD_NAME="my-pod"
NAMESPACE="default"

# Command to place the pod on the best node
echo "kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: $POD_NAME
  namespace: $NAMESPACE
spec:
  containers:
  - name: my-container
    image: nginx
  nodeSelector:
    kubernetes.io/hostname: $BEST_NODE
EOF"

Running the Custom Scheduler

Save the script to custom-scheduler.sh and run it:

chmod +x custom-scheduler.sh
./custom-scheduler.sh

This script will output the best node for the pod based on available CPU resources and print the command to place the pod on that node.

Insights and Considerations

While this example is simplistic, it demonstrates the core principles of building a custom Kubernetes scheduler. Here are some insights and considerations:

Scoring Mechanism: The scoring function can be made more sophisticated by incorporating additional metrics such as memory, storage, network bandwidth, and node labels. You can also implement custom business logic to prioritize certain nodes based on specific requirements.
API Interaction: For a production-ready scheduler, you would need to interact with the Kubernetes API more robustly. This includes handling API authentication, error handling, and asynchronous operations.
Performance: A custom scheduler should be designed with performance in mind. The scheduling process should be efficient to avoid delays in pod placement, which can impact application availability and performance.
Testing: Thoroughly test your custom scheduler in a controlled environment before deploying it to production. Ensure it handles various edge cases and integrates well with other cluster components.

Takeaway

Building a custom Kubernetes scheduler can be a powerful way to tailor pod placement to your specific needs. While the default scheduler is highly capable, custom schedulers offer flexibility to optimize resource usage, enforce custom policies, and align with unique business requirements. By understanding the scheduler's workflow and leveraging tools like shell scripts and the Kubernetes API, you can create a scheduler that meets your specific use cases. This hands-on approach not only deepens your understanding of Kubernetes internals but also empowers you to build more efficient and effective container orchestration solutions.

使用 Shell 脚本构建自定义 Kubernetes 调度器：动手实验之旅