跳到主要内容

ai-rate-limiting

ai-rate-limiting 插件为发送到 LLM 服务的请求强制执行基于 Token 的速率限制。它通过控制指定时间范围内消耗的 Token 数量来帮助管理 API 使用情况，确保公平的资源分配并防止服务过载。该插件通常与 ai-proxy-multi 插件一起使用。

本地限速与基于 Redis 的限速​

ai-rate-limiting 插件支持两种限速模式：

  • 本地限速：每个网关实例独立实施限速。每个实例维护自己的计数器，因此当流量分散到多个实例时，实际限额约为“限额 × 实例数”。未设置 policy 或将其设置为 local 时，这是默认模式。
  • 基于 Redis 的限速：通过 Redis 在所有网关实例之间共享限速计数。所有实例共享同一配额，因此配置的限额适用于所有网关实例。

在多实例部署中，请通过不同的网关地址发送验证请求，确认它们消耗的是同一个 Redis 共享配额。

Redis 示例是使用教程密码和内部服务地址的评估用配置。对于生产部署，请使用 Secret 管理凭据、限制网络访问，并在 Redis 服务支持 TLS 时启用 redis_ssl 或 redis_cluster_ssl 及证书校验。

演示​

以下演示展示了如何在 API7 企业版控制台中为两个模型配置不同优先级，并对高优先级实例应用限速。当 fallback_strategy 设置为 ["rate_limiting"] 时，第一个实例的配额耗尽后，请求会回退到低优先级实例。配置方法请参阅配置实例优先级和速率限制。

示例​

以下示例展示了如何在不同场景下配置 ai-rate-limiting。

示例使用 OpenAI 和 DeepSeek 作为上游 LLM 服务。请为示例使用的服务提供方创建 API Key，并保存到环境变量：

export OPENAI_API_KEY=replace-with-openai-api-key
export DEEPSEEK_API_KEY=replace-with-deepseek-api-key

限制单个实例的速率​

以下示例配置两个模型实例，按 80/20 的比例分配流量，并对接收 80% 流量的实例实施限速。该实例的配额耗尽后，rate_limiting 回退策略会将其他请求转发到另一个实例。

创建一个路由，并更新为你的模型服务提供方、模型、API Key 和端点：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rate-limiting-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "deepseek-instance-1",
            "provider": "deepseek",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-flash"
            }
          },
          {
            "name": "deepseek-instance-2",
            "provider": "deepseek",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-flash"
            }
          }
        ]
      },
      "ai-rate-limiting": {
        "policy": "local",
        "limit_strategy": "total_tokens",
        "instances": [
          {
            "name": "deepseek-instance-1",
            "limit": 100,
            "time_window": 30
          }
        ]
      }
    }
  }
EOF

❶ 基于 total_tokens 应用速率限制。

❷ 对 deepseek-instance-1 实例应用速率限制。

❸ 配置 100 个 Token 的配额。

❹ 配置时间窗口为 30 秒。

ADC 和 Gateway API 标签页包含常规后端。当插件自行处理请求时，不会联系该后端。请验证下方的插件特定结果；仅收到后端响应并不能证明插件已运行。

发送一个 POST 请求到该路由，请求体中包含系统提示词和一个示例用户问题：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

你应该会收到类似以下的响应：

{
  ...
  "model": "deepseek-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1 + 1 equals 2. This is a fundamental arithmetic operation where adding one unit to another results in a total of two units."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

当 deepseek-instance-1 在 30 秒窗口内耗尽 100 个 Token 的配额后，rate_limiting 回退策略会将其排除在选择范围之外。其他请求会转发到未受限速的 deepseek-instance-2。

对所有实例应用一个共享限额​

以下示例为路由上的所有 LLM 上游实例应用一个共享的 500 Token 限额。

为了演示和更易于区分，你将配置一个 OpenAI 实例和一个 DeepSeek 实例作为上游 LLM 服务。

创建一个路由，并更新为你的模型服务提供方、模型、API Key 和端点：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rate-limiting-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-flash"
            }
          }
        ]
      },
      "ai-rate-limiting": {
        "policy": "local",
        "limit": 500,
        "time_window": 60,
        "rejected_code": 429,
        "limit_strategy": "total_tokens"
      }
    }
  }
EOF

❶ 在顶层应用一个由所有选定 LLM 实例共享的 500 Token 限额。若要为各个实例分别配置限额或时间窗口，请使用 ai-rate-limiting.instances 条目。

❷ 配置时间窗口为 60 秒。

❸ 设置拒绝响应的 HTTP 状态码为 429。

❹ 基于 total_tokens 应用速率限制。

发送一个 POST 请求到该路由，请求体中包含系统提示词和一个示例用户问题：

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws" }
    ]
  }'

你应该会收到来自任一 LLM 实例的响应，类似于：

{
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Sure! Sir Isaac Newton formulated three laws of motion that describe the motion of objects. These laws are widely used in physics and engineering for studying and understanding how things move. Here they are:\n\n1. Newton's First Law - Law of Inertia: An object at rest tends to stay at rest and an object in motion tends to stay in motion with the same speed and in the same direction unless acted upon by an unbalanced force. This is also known as the principle of inertia.\n\n2. Newton's Second Law of Motion - Force and Acceleration: The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. This is usually formulated as F=ma where F is the force applied, m is the mass of the object and a is the acceleration produced.\n\n3. Newton's Third Law - Action and Reaction: For every action, there is an equal and opposite reaction. This means that any force exerted on a body will create a force of equal magnitude but in the opposite direction on the object that exerted the first force.\n\nIn simple terms: \n1. If you slide a book on a table and let go, it will stop because of the friction (or force) between it and the table.\n2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 256,
    "total_tokens": 279,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

该响应会从共享配额中扣除 279 个 Token，剩余 221 个 Token。

在相同的 60 秒窗口内，发送另一个 POST 请求到该路由：

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws" }
    ]
  }'

你应该会再次收到来自任一 LLM 实例的成功响应，类似于：

{
  "model": "deepseek-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Sure! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics. Here's an explanation of each law:\n\n---\n\n### **1. Newton's First Law (Law of Inertia)**\n- **Statement**: An object will remain at rest or in uniform motion in a straight line unless acted upon by an external force.\n- **What it means**: This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion. If no net force acts on an object, its velocity (speed and direction) will not change.\n- **Example**: A book lying on a table will stay at rest unless you push it. Similarly, a hockey puck sliding on ice will keep moving at a constant speed unless friction or another force slows it down.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration)**\n- **Statement**: The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n"
      },
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 256,
    "total_tokens": 269,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "prompt_cache_hit_tokens": 0,
    "prompt_cache_miss_tokens": 13
  },
  "system_fingerprint": "fp_3a5770e1b4_prod0225"
}

两次响应会从同一个共享计数器中扣除 548 个 Token。准入检查只会在请求开始前确认至少还剩 1 个 Token，实际用量会在响应后扣除。因此，第二个请求仍可能成功，并使共享计数器超过 500 Token 限额。

在相同的 60 秒窗口内，发送第三个 POST 请求到该路由：

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws" }
    ]
  }'

按照上述示例总量，第三个请求应被拒绝并返回 429，因为共享计数器已经耗尽。实际 Token 用量可能不同，验证时请重复请求直至配额耗尽。尽管所有实例使用同一个底层计数器，限速响应头的后缀仍会标识所选实例：

X-AI-RateLimit-Limit-{instance-name}: 500
X-AI-RateLimit-Remaining-{instance-name}: 0
X-AI-RateLimit-Reset-{instance-name}: 42.137000083923

配置实例优先级和速率限制​

以下示例配置两个具有不同优先级的模型，并对高优先级实例实施限速。当 fallback_strategy 设置为 ["rate_limiting"] 时，第一个实例的配额耗尽后，请求会回退到低优先级实例。

创建一个路由，并更新为你的模型服务提供方、模型、API Key 和端点：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rate-limiting-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "priority": 1,
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "priority": 0,
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-flash"
            }
          }
        ]
      },
      "ai-rate-limiting": {
        "policy": "local",
        "instances": [
          {
            "name": "openai-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "limit_strategy": "total_tokens"
      }
    }
  }
EOF

❶ 将 fallback_strategy 设置为 ["rate_limiting"]。

❷ 在 openai-instance 实例上设置更高的优先级。

❸ 在 deepseek-instance 实例上设置较低的优先级。

❹ 对 openai-instance 实例应用速率限制。

❺ 配置 10 个 Token 的配额。

❻ 配置时间窗口为 60 秒。

❼ 基于 total_tokens 应用速率限制。

发送一个 POST 请求到该路由，请求体中包含系统提示词和一个示例用户问题：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

你应该会收到类似以下的响应：

{
  ...,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

由于 total_tokens 值超过了配置的 10 个 Token 配额，因此在 60 秒窗口内的下一个请求预计将被转发到另一个实例。

在相同的 60 秒窗口内，发送另一个 POST 请求到该路由：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newton law" }
    ]
  }'

你应该看到类似于以下的响应：

{
  ...,
  "model": "deepseek-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Certainly! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics.\n\n---\n\n### **1. Newton's First Law (Law of Inertia):**\n- **Statement:** An object at rest will remain at rest, and an object in motion will continue moving at a constant velocity (in a straight line at a constant speed), unless acted upon by an external force.\n- **Key Idea:** This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion.\n- **Example:** If you slide a book across a table, it eventually stops because of the force of friction acting on it. Without friction, the book would keep moving indefinitely.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration):**\n- **Statement:** The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n  where:\n  - \\( F \\) = net force applied (in Newtons),\n  -"
      },
      ...
    }
  ],
  ...
}

按消费者负载均衡和速率限制​

以下示例展示了如何配置两个模型进行负载均衡，并按消费者应用速率限制。

创建消费者 johndoe，并在 openai-instance 实例上设置 60 秒窗口内 10 个 Token 的速率限制配额：

curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "username": "johndoe",
    "plugins": {
      "ai-rate-limiting": {
        "policy": "local",
        "instances": [
          {
            "name": "openai-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "rejected_code": 429,
        "limit_strategy": "total_tokens"
      }
    }
  }'

配置 johndoe 的 key-auth 凭证：

curl "http://127.0.0.1:9180/apisix/admin/consumers/johndoe/credentials" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "cred-john-key-auth",
    "plugins": {
      "key-auth": {
        "key": "john-key"
      }
    }
  }'

创建另一个消费者 janedoe，并在 deepseek-instance 实例上设置 60 秒窗口内 10 个 Token 的速率限制配额：

curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "username": "janedoe",
    "plugins": {
      "ai-rate-limiting": {
        "policy": "local",
        "instances": [
          {
            "name": "deepseek-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "rejected_code": 429,
        "limit_strategy": "total_tokens"
      }
    }
  }'

配置 janedoe 的 key-auth 凭证：

curl "http://127.0.0.1:9180/apisix/admin/consumers/janedoe/credentials" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "cred-jane-key-auth",
    "plugins": {
      "key-auth": {
        "key": "jane-key"
      }
    }
  }'

创建一个路由，并更新为你的模型服务提供方、模型、API Key 和端点：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rate-limiting-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "key-auth": {},
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-flash"
            }
          }
        ]
      }
    }
  }
EOF

❶ 在路由上启用 key-auth。

❷ 配置 openai 实例。

❸ 配置 deepseek 实例。

发送一个不带任何消费者密钥的 POST 请求到该路由：

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

你应该会收到一个 HTTP/1.1 401 Unauthorized 响应。

使用 johndoe 的密钥发送一个 POST 请求到该路由：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: john-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

由于两个实例具有相同的优先级和权重，第一个响应可能来自任一模型。OpenAI 响应类似如下：

{
  ...,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

如果 OpenAI 响应耗尽了 johndoe 在该实例上的配额，那么 60 秒窗口内的后续请求会排除 OpenAI 并使用 DeepSeek。如果第一次选择的是 DeepSeek，请重复请求，直到选中 OpenAI 且其配额耗尽。

在相同的 60 秒窗口内，使用 johndoe 的密钥发送另一个 POST 请求到该路由：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: john-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws to me" }
    ]
  }'

排除 johndoe 的 OpenAI 实例后，你应该会看到类似如下的 DeepSeek 响应：

{
  ...,
  "model": "deepseek-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Certainly! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics.\n\n---\n\n### **1. Newton's First Law (Law of Inertia):**\n- **Statement:** An object at rest will remain at rest, and an object in motion will continue moving at a constant velocity (in a straight line at a constant speed), unless acted upon by an external force.\n- **Key Idea:** This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion.\n- **Example:** If you slide a book across a table, it eventually stops because of the force of friction acting on it. Without friction, the book would keep moving indefinitely.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration):**\n- **Statement:** The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n  where:\n  - \\( F \\) = net force applied (in Newtons),\n  -"
      },
      ...
    }
  ],
  ...
}

使用 janedoe 的密钥发送一个 POST 请求到该路由：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: jane-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

第一个响应同样可能来自任一模型。DeepSeek 响应类似如下：

{
  ...,
  "model": "deepseek-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The sum of 1 and 1 is 2. This is a basic arithmetic operation where you combine two units to get a total of two units."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 31,
    "total_tokens": 45,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "prompt_cache_hit_tokens": 0,
    "prompt_cache_miss_tokens": 14
  },
  "system_fingerprint": "fp_3a5770e1b4_prod0225"
}

如果 DeepSeek 响应耗尽了 janedoe 在该实例上的配额，那么 60 秒窗口内的后续请求会排除 DeepSeek 并使用 OpenAI。如果第一次选择的是 OpenAI，请重复请求，直到选中 DeepSeek 且其配额耗尽。

在相同的 60 秒窗口内，使用 janedoe 的密钥发送另一个 POST 请求到该路由：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: jane-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws to me" }
    ]
  }'

排除 janedoe 的 DeepSeek 实例后，你应该会看到类似如下的 OpenAI 响应：

{
  ...,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Sure, here are Newton's three laws of motion:\n\n1) Newton's First Law, also known as the Law of Inertia, states that an object at rest will stay at rest, and an object in motion will stay in motion, unless acted on by an external force. In simple words, this law suggests that an object will keep doing whatever it is doing until something causes it to do otherwise. \n\n2) Newton's Second Law states that the force acting on an object is equal to the mass of that object times its acceleration (F=ma). This means that force is directly proportional to mass and acceleration. The heavier the object and the faster it accelerates, the greater the force.\n\n3) Newton's Third Law, also known as the law of action and reaction, states that for every action, there is an equal and opposite reaction. Essentially, any force exerted onto a body will create a force of equal magnitude but in the opposite direction on the object that exerted the first force.\n\nRemember, these laws become less accurate when considering speeds near the speed of light (where Einstein's theory of relativity becomes more appropriate) or objects very small or very large. However, for everyday situations, they provide a good model of how things move.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

这表明 ai-proxy-multi 会在 ai-rate-limiting 为每个消费者应用不同的实例配额时平衡流量。

基于规则的速率限制​

以下示例根据请求属性配置不同的限流规则。规则自 API7 企业版 3.8.17 和 APISIX 3.17.0 起可用。示例还根据服务提供方返回的原始用量计算请求成本，并对补全 Token 按两倍计费。表达式成本计算自 API7 企业版 3.9.8 和 APISIX 3.17.0 起可用。

请注意，所有规则按顺序应用。如果配置的键不存在，则将跳过相应的规则。

提示

除 HTTP 请求头外，还可以根据其他内置变量配置规则，以实现更灵活、更细粒度的限速策略。

创建一个根据请求头应用速率限制的路由。该路由使用 X-Subscription-ID 为每个订阅限速，并使用 X-Trial-ID 对试用用户应用更严格的限制：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rate-limiting-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4o-mini"
        }
      },
      "ai-rate-limiting": {
        "policy": "local",
        "rejected_code": 429,
        "limit_strategy": "expression",
        "cost_expr": "prompt_tokens + completion_tokens * 2",
        "rules": [
          {
            "key": "\${http_x_subscription_id}",
            "count": "\${http_x_custom_count ?? 500}",
            "time_window": 60
          },
          {
            "key": "\${http_x_trial_id}",
            "count": 50,
            "time_window": 60
          }
        ]
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }
EOF

❶ 使用 X-Subscription-ID 请求头的值作为速率限制键。

❷ 基于 X-Custom-Count 头动态设置请求限制。如果未提供该头，则应用 500 个 Token 的默认计数。

❸ 使用 X-Trial-ID 请求头的值作为速率限制键。

成本表达式使用 OpenAI 兼容服务提供方返回的原始 prompt_tokens 和 completion_tokens 字段。对于原生 Anthropic 用量，表达式可以组合 input_tokens、cache_creation_input_tokens、cache_read_input_tokens 和 output_tokens。缺失变量按 0 计算；负数结果会钳制为 0，小数结果会四舍五入到最近的整数，然后再提交到限流计数器。仅使用配置参考中记录的算术运算符和受支持数学函数。

网关会在当前响应的 Token 用量可用前计算配额响应头，并在日志阶段提交该用量。因此，第一个成功响应会显示完整配额，下一个响应才会反映前一个请求消耗的 Token。

要验证速率限制，请使用相同的订阅 ID 向该路由发送多个以下请求：

curl "http://127.0.0.1:9080/anything" -i -X POST \
  -H "X-Subscription-ID: sub-123456789" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

这些请求应匹配具有 500 个默认 Token 计数的第一个规则。你应该看到配额内的请求返回 HTTP/1.1 200 OK，而超过配额的请求返回 HTTP/1.1 429 Too Many Requests：

HTTP/1.1 200 OK
...
X-AI-1-RateLimit-Limit: 500
X-AI-1-RateLimit-Remaining: 500
X-AI-1-RateLimit-Reset: 60

HTTP/1.1 200 OK
...
X-AI-1-RateLimit-Limit: 500
X-AI-1-RateLimit-Remaining: 344
X-AI-1-RateLimit-Reset: 57.989000082016

HTTP/1.1 429 Too Many Requests
...
X-AI-1-RateLimit-Limit: 500
X-AI-1-RateLimit-Remaining: 0
X-AI-1-RateLimit-Reset: 5.871000051498

等待时间窗口重置。使用相同的订阅 ID 发送多个以下请求到该路由，并将 X-Custom-Count 头设置为 10：

curl "http://127.0.0.1:9080/anything" -i -X POST \
  -H "X-Subscription-ID: sub-123456789" \
  -H "X-Custom-Count: 10" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

这些请求应匹配具有 10 个自定义 Token 计数的第一个规则。你应该看到配额内的请求返回 HTTP/1.1 200 OK，而超过配额的请求返回 HTTP/1.1 429 Too Many Requests：

HTTP/1.1 200 OK
...
X-AI-1-RateLimit-Limit: 10
X-AI-1-RateLimit-Remaining: 10
X-AI-1-RateLimit-Reset: 60

HTTP/1.1 429 Too Many Requests
...
X-AI-1-RateLimit-Limit: 10
X-AI-1-RateLimit-Remaining: 0
X-AI-1-RateLimit-Reset: 40.422000169754

最后，发送多个不带订阅请求头的请求。以下请求只包含试用 ID：

curl "http://127.0.0.1:9080/anything" -i -X POST \
  -H "X-Trial-ID: trial-123456789" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

这些请求应匹配具有 50 个 Token 计数的第二个规则。你应该看到配额内的请求返回 HTTP/1.1 200 OK，而超过配额的请求返回 HTTP/1.1 429 Too Many Requests：

HTTP/1.1 200 OK
...
X-AI-2-RateLimit-Limit: 50
X-AI-2-RateLimit-Remaining: 50
X-AI-2-RateLimit-Reset: 60

HTTP/1.1 429 Too Many Requests
...
X-AI-2-RateLimit-Limit: 50
X-AI-2-RateLimit-Remaining: 0
X-AI-2-RateLimit-Reset: 44

使用 Redis 服务器在网关之间共享配额​

此示例将限速计数器存储在单个 Redis 服务器中，使多个网关实例执行同一份配额。

基于 Redis 的计数器自 API7 企业版 3.8.19 和 APISIX 3.18.0 起可用。

启动 Redis​

在网关所在的环境中启动 Redis。

将 GATEWAY_CONTAINER 设置为正在运行的 APISIX 或 API7 Gateway 容器。创建专用网络并将网关连接到该网络：

export GATEWAY_CONTAINER=replace-with-gateway-container-name

docker network create gateway-redis-net
docker network connect gateway-redis-net "$GATEWAY_CONTAINER"

在共享网络上启动 Redis：

docker run -d \
  --name redis-standalone \
  --network gateway-redis-net \
  redis:8.10.2-alpine \
  redis-server --requirepass redis-password

验证经过身份认证的连接：

docker exec -e REDISCLI_AUTH=redis-password redis-standalone redis-cli ping

你应该收到响应 PONG，表示连接成功。

设置 Admin API 和 ADC 示例使用的 Redis 主机名：

export REDIS_HOST=redis-standalone

创建路由并配置速率限制​

在网关组中创建一个具有以下配置的路由：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rate-limiting-redis-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-flash"
            }
          },
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            }
          }
        ]
      },
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "deepseek-instance",
            "limit": 100,
            "time_window": 30
          },
          {
            "name": "openai-instance",
            "limit": 50,
            "time_window": 30
          }
        ],
        "limit_strategy": "total_tokens",
        "policy": "redis",
        "redis_host": "$REDIS_HOST",
        "redis_port": 6379,
        "redis_password": "redis-password",
        "allow_degradation": false,
        "rejected_code": 429
      }
    }
  }
EOF

❶ policy：设置为 redis 以使用 Redis 实例进行速率限制。

❷ redis_host：设置为网关可访问的 Redis 主机名。

❸ redis_port：设置为 Redis 实例的监听端口。

❹ redis_password：如果有，设置为 Redis 实例的密码。

❺ allow_degradation: 设置为 false，如果 Redis 不可用则拒绝请求。

验证​

发送一个 POST 请求到该路由，请求体中包含系统提示词和一个示例用户问题。

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

响应应为 HTTP/1.1 200 OK，并包含所选模型实例的限速响应头。

逐个重复发送相同请求。分配到配额已耗尽的模型实例时，请求应返回 HTTP/1.1 429 Too Many Requests。由于实例采用加权选择，并且不同模型报告的 Token 用量不同，在两个实例的配额都耗尽之前，请求仍可能成功。

使用 Redis 集群在网关节点之间共享配额​

此示例将限速计数器存储在 Redis Cluster 中，使多个网关节点共享经过分片和复制的配额存储。

基于 Redis 的计数器自 API7 企业版 3.8.19 和 APISIX 3.18.0 起可用。

启动 Redis Cluster​

将 GATEWAY_CONTAINER 设置为正在运行的 APISIX 或 API7 Gateway 容器。创建专用网络并将网关连接到该网络：

export GATEWAY_CONTAINER=replace-with-gateway-container-name

docker network create gateway-redis-cluster-net
docker network connect gateway-redis-cluster-net "$GATEWAY_CONTAINER"

为三个主节点和三个副本创建以下 Docker Compose 文件：

docker-compose.yml
x-redis-node: &redis-node
  image: redis:8.10.2-alpine
  command:
    - redis-server
    - --cluster-enabled
    - "yes"
    - --cluster-config-file
    - nodes.conf
    - --cluster-node-timeout
    - "5000"
    - --appendonly
    - "no"
    - --requirepass
    - redis-cluster-password
    - --masterauth
    - redis-cluster-password
  healthcheck:
    test: ["CMD", "redis-cli", "-a", "redis-cluster-password", "ping"]
    interval: 2s
    timeout: 2s
    retries: 30
  networks:
    - gateway-redis-cluster-net

services:
  redis-cluster-1:
    <<: *redis-node
    container_name: redis-cluster-1
  redis-cluster-2:
    <<: *redis-node
    container_name: redis-cluster-2
  redis-cluster-3:
    <<: *redis-node
    container_name: redis-cluster-3
  redis-cluster-4:
    <<: *redis-node
    container_name: redis-cluster-4
  redis-cluster-5:
    <<: *redis-node
    container_name: redis-cluster-5
  redis-cluster-6:
    <<: *redis-node
    container_name: redis-cluster-6

networks:
  gateway-redis-cluster-net:
    external: true

启动节点并等待健康检查通过：

docker compose up -d --wait

初始化集群，并为每个主节点分配一个副本：

docker exec -e REDISCLI_AUTH=redis-cluster-password redis-cluster-1 \
  redis-cli --cluster create \
  redis-cluster-1:6379 \
  redis-cluster-2:6379 \
  redis-cluster-3:6379 \
  redis-cluster-4:6379 \
  redis-cluster-5:6379 \
  redis-cluster-6:6379 \
  --cluster-replicas 1 \
  --cluster-yes

输出末尾应显示 [OK] All 16384 slots covered.。

验证所有哈希槽均可用：

docker exec -e REDISCLI_AUTH=redis-cluster-password redis-cluster-1 \
  redis-cli cluster info

输出应包含 cluster_state:ok 和 cluster_slots_assigned:16384。

设置 Admin API 和 ADC 示例使用的 Redis 节点主机名：

export REDIS_CLUSTER_NODE_1=redis-cluster-1
export REDIS_CLUSTER_NODE_2=redis-cluster-2
export REDIS_CLUSTER_NODE_3=redis-cluster-3
export REDIS_CLUSTER_NODE_4=redis-cluster-4
export REDIS_CLUSTER_NODE_5=redis-cluster-5
export REDIS_CLUSTER_NODE_6=redis-cluster-6

创建路由并配置速率限制​

在网关组中创建一个具有以下配置的路由：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rate-limiting-redis-cluster-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-flash"
            }
          },
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            }
          }
        ]
      },
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "deepseek-instance",
            "limit": 200,
            "time_window": 60
          },
          {
            "name": "openai-instance",
            "limit": 100,
            "time_window": 60
          }
        ],
        "limit_strategy": "total_tokens",
        "policy": "redis-cluster",
        "redis_cluster_nodes": [
          "$REDIS_CLUSTER_NODE_1:6379",
          "$REDIS_CLUSTER_NODE_2:6379",
          "$REDIS_CLUSTER_NODE_3:6379",
          "$REDIS_CLUSTER_NODE_4:6379",
          "$REDIS_CLUSTER_NODE_5:6379",
          "$REDIS_CLUSTER_NODE_6:6379"
        ],
        "redis_password": "redis-cluster-password",
        "redis_cluster_name": "docs-redis-cluster",
        "redis_timeout": 1000,
        "allow_degradation": false,
        "rejected_code": 429
      }
    }
  }
EOF

❶ policy：设置为 redis-cluster 以使用 Redis 集群进行速率限制。

❷ redis_cluster_nodes：设置为 Redis 集群中的 Redis 节点地址。

❸ redis_password：如果有，设置为 Redis 集群的密码。

❹ redis_cluster_name：设置为 Redis 集群名称。

验证​

发送一个 POST 请求到该路由，请求体中包含系统提示词和一个示例用户问题。

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

响应应为 HTTP/1.1 200 OK，并包含所选模型实例的限速响应头。

逐个重复发送相同请求。分配到配额已耗尽的模型实例时，请求应返回 HTTP/1.1 429 Too Many Requests。由于实例采用加权选择，并且不同模型报告的 Token 用量不同，在两个实例的配额都耗尽之前，请求仍可能成功。

使用 Redis Sentinel 在网关节点之间共享配额​

Redis Sentinel 数据节点的认证连接自 API7 企业版 3.10.5 和 APISIX 3.18.0 起可用。

如果需要自动故障转移和高可用，但不需要数据分区，请使用 Redis Sentinel。此模式管理起来更简单，适用于大多数高可用场景。

此示例在与网关共享的网络上运行一个主节点、两个副本和三个 Sentinel。

警告

redis-sentinel 策略没有为 Sentinel 发现或选中的 Redis 数据节点连接提供 TLS 配置。请将网关、Sentinel 和 Redis 节点部署在可信的私有网络中。

启动 Redis Sentinel​

将 GATEWAY_CONTAINER 设置为正在运行的 APISIX 或 API7 Gateway 容器。创建专用网络并将网关连接到该网络：

export GATEWAY_CONTAINER=replace-with-gateway-container-name

docker network create gateway-redis-sentinel-net
docker network connect gateway-redis-sentinel-net "$GATEWAY_CONTAINER"

启动 Redis 主节点：

docker run -d \
  --name redis-master \
  --network gateway-redis-sentinel-net \
  redis:8.10.2-alpine \
  redis-server --requirepass redis-password

启动第一个副本：

docker run -d \
  --name redis-replica-1 \
  --network gateway-redis-sentinel-net \
  redis:8.10.2-alpine \
  redis-server --replicaof redis-master 6379 \
    --masterauth redis-password \
    --requirepass redis-password \
    --replica-announce-ip redis-replica-1

启动第二个副本：

docker run -d \
  --name redis-replica-2 \
  --network gateway-redis-sentinel-net \
  redis:8.10.2-alpine \
  redis-server --replicaof redis-master 6379 \
    --masterauth redis-password \
    --requirepass redis-password \
    --replica-announce-ip redis-replica-2

创建使用 Docker DNS 名称而非容器 IP 的 Sentinel 配置：

sentinel.conf
port 26379
protected-mode no
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel monitor mymaster redis-master 6379 2
sentinel auth-pass mymaster redis-password
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1
requirepass sentinel-password

使用配置文件的可写副本启动第一个 Sentinel：

docker run -d \
  --name redis-sentinel-1 \
  --network gateway-redis-sentinel-net \
  -v "$PWD/sentinel.conf:/etc/redis/sentinel.conf:ro" \
  redis:8.10.2-alpine \
  sh -c 'cp /etc/redis/sentinel.conf /data/sentinel.conf && redis-sentinel /data/sentinel.conf'

启动第二个 Sentinel：

docker run -d \
  --name redis-sentinel-2 \
  --network gateway-redis-sentinel-net \
  -v "$PWD/sentinel.conf:/etc/redis/sentinel.conf:ro" \
  redis:8.10.2-alpine \
  sh -c 'cp /etc/redis/sentinel.conf /data/sentinel.conf && redis-sentinel /data/sentinel.conf'

启动第三个 Sentinel：

docker run -d \
  --name redis-sentinel-3 \
  --network gateway-redis-sentinel-net \
  -v "$PWD/sentinel.conf:/etc/redis/sentinel.conf:ro" \
  redis:8.10.2-alpine \
  sh -c 'cp /etc/redis/sentinel.conf /data/sentinel.conf && redis-sentinel /data/sentinel.conf'

验证 Sentinel 仲裁是否可用：

docker exec -e REDISCLI_AUTH=sentinel-password redis-sentinel-1 \
  redis-cli -p 26379 SENTINEL ckquorum mymaster

命令应报告 OK 3 usable Sentinels。

设置 Admin API 和 ADC 示例使用的 Sentinel 主机名：

export REDIS_SENTINEL_1=redis-sentinel-1
export REDIS_SENTINEL_2=redis-sentinel-2
export REDIS_SENTINEL_3=redis-sentinel-3

创建路由并配置速率限制​

在网关组中创建一个具有以下配置的路由：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rate-limiting-redis-sentinel-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-flash"
            }
          },
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            }
          }
        ]
      },
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "deepseek-instance",
            "limit": 100,
            "time_window": 60
          },
          {
            "name": "openai-instance",
            "limit": 100,
            "time_window": 60
          }
        ],
        "limit_strategy": "total_tokens",
        "policy": "redis-sentinel",
        "redis_sentinels": [
          {"host": "$REDIS_SENTINEL_1", "port": 26379},
          {"host": "$REDIS_SENTINEL_2", "port": 26379},
          {"host": "$REDIS_SENTINEL_3", "port": 26379}
        ],
        "redis_master_name": "mymaster",
        "redis_password": "redis-password",
        "redis_role": "master",
        "sentinel_password": "sentinel-password",
        "rejected_code": 429
      }
    }
  }
EOF

❶ policy：设置为 redis-sentinel 以使用 Sentinel 模式下的 Redis 进行速率限制。

❷ redis_sentinels：配置 Sentinel 节点地址列表（主机和端口）。

❸ redis_master_name：配置 Sentinel 监控的 Redis 主组名称。

❹ redis_password：如果有，设置为 Redis 实例的密码。

❺ redis_role：设置为 master，以连接当前 Redis 主节点。

❻ sentinel_password：配置用于向 Redis Sentinel 进行身份认证的密码。

验证​

发送一个 POST 请求到该路由，请求体中包含系统提示词和一个示例用户问题。

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

响应应为 HTTP/1.1 200 OK，并包含所选模型实例的限速响应头。

逐个重复发送相同请求。分配到配额已耗尽的模型实例时，请求应返回 HTTP/1.1 429 Too Many Requests。由于实例采用加权选择，并且不同模型报告的 Token 用量不同，在两个实例的配额都耗尽之前，请求仍可能成功。