跳到主要内容

ai-proxy-multi

ai-proxy-multi 插件在 OpenAI、DeepSeek、Gemini、Vertex AI 以及其他兼容的模型服务提供方实例之间分发 LLM 和嵌入请求，并提供负载均衡、重试、故障回退和健康检查功能。

转发请求前，插件会根据请求 URI 和请求体识别客户端协议。当客户端使用 Anthropic Messages，而所选实例的服务提供方支持 OpenAI Chat Completions 而非原生 Anthropic Messages 时，插件会执行协议转换：将客户端请求转换为 OpenAI 格式，并将后端响应转换为 Anthropic 格式。该转换只支持 Anthropic Messages 的一部分功能。有关检测规则、字段映射、版本边界和限制，请参阅协议参考。

该插件还可以在访问日志而非错误日志中记录 LLM 请求信息，包括 Token 用量、模型和首字节响应时间。日志插件可以采集这些条目。

演示​

演示展示了如何在 API7 企业版控制台中配置实例优先级和限流。示例为两个模型分配不同的优先级，并对高优先级实例进行限流。当该实例耗尽配额，且 fallback_strategy 为 ["rate_limiting"] 时，请求会继续转发到低优先级实例。

示例​

以下示例展示如何针对常见的路由和弹性场景配置 ai-proxy-multi。

请为每个实例设置唯一名称，即使实例的优先级不同。重命名实例时，还需更新对应的 AI 速率限制条目和语义后备引用。

实例间负载均衡​

本示例按 80:20 的比例在两个模型之间分发流量。如果第一次尝试在两秒内返回 429 或 5xx，插件会在另一个实例上重试一次。max_retries 和 retry_on_failure_within_ms 字段在 API7 企业版 3.9.14 和 APISIX 3.17.0 中引入。

该示例使用 OpenAI 和 DeepSeek 实例，以便清楚区分所选的上游。

创建路由，并按需替换模型服务提供方相关的值：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "fallback_strategy": ["http_429", "http_5xx"],
        "max_retries": 1,
        "retry_on_failure_within_ms": 2000,
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ]
      }
    }
  }
EOF

❶ 将 openai-instance 的权重配置为 8。

❷ 将 deepseek-instance 的权重配置为 2。

ADC 和 Gateway API 标签页包含一个常规后端。当插件自行处理请求时，不会访问该后端。请验证下文中插件特有的结果；仅收到后端响应并不能证明插件已运行。

使用系统提示词和示例用户问题发送 10 个 POST 请求，并统计分别转发到 OpenAI 和 DeepSeek 的请求数量：

openai_count=0
deepseek_count=0

for i in {1..10}; do
  model=$(curl -s "http://127.0.0.1:9080/anything" -X POST \
    -H "Content-Type: application/json" \
    -d '{
      "messages": [
        { "role": "system", "content": "You are a mathematician" },
        { "role": "user", "content": "What is 1+1?" }
      ]
    }' | jq -r '.model')

  if [[ "$model" == *"gpt-4"* ]]; then
    ((openai_count++))
  elif [[ "$model" == "deepseek-chat" ]]; then
    ((deepseek_count++))
  fi
done

echo "OpenAI responses: $openai_count"
echo "DeepSeek responses: $deepseek_count"

你应该看到类似于以下的响应：

OpenAI responses: 8
DeepSeek responses: 2

对 Responses API 请求进行负载均衡​

以下示例在 OpenAI Responses API 路由上配置 ai-proxy-multi，并将请求分配到两个 OpenAI 模型。

Responses 和 Embeddings 请求都包含 input。请保留 /v1/responses URI 后缀，使插件将该请求识别为 Responses，而不是 Embeddings。请参阅请求协议检测。

创建一条路由，并将 uri 设置为 /v1/responses：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-responses-route",
    "uri": "/v1/responses",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-responses-primary",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1"
            }
          },
          {
            "name": "openai-responses-secondary",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          }
        ]
      }
    }
  }
EOF

使用 OpenAI Responses API 格式发送请求：

curl "http://127.0.0.1:9080/v1/responses" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Write one sentence about API gateways."
  }'

请求会被转发到已配置的某个 OpenAI 实例，并以 Responses API 格式返回响应。

按语义相似度路由​

semantic 负载均衡算法通过比较请求提示词与分配给各实例的示例语句来选择实例。此功能自 API7 企业版 3.9.18 和 3.10.5 以及 APISIX 3.18.0 起可用。

导出向量嵌入请求和 LLM 请求所需的 API Key：

export EMBEDDING_API_KEY="<YOUR_EMBEDDING_API_KEY>"
export LLM_API_KEY="<YOUR_LLM_API_KEY>"

创建一条路由，分别配置适用于编程、翻译和通用提示词的实例。当所有得分均未达到配置的阈值，或向量嵌入请求失败时，也会使用通用实例作为后备实例：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-semantic-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "balancer": {
          "algorithm": "semantic"
        },
        "semantic_opts": {
          "embeddings": {
            "provider": "openai",
            "model": "text-embedding-3-small",
            "auth": {
              "header": {
                "Authorization": "Bearer $EMBEDDING_API_KEY"
              }
            }
          },
          "threshold": 0.5,
          "fallback": "default",
          "debugging": true
        },
        "instances": [
          {
            "name": "code",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $LLM_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o"
            },
            "examples": [
              "write a Python function",
              "debug this code"
            ]
          },
          {
            "name": "translate",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $LLM_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            },
            "examples": [
              "translate this sentence into English"
            ]
          },
          {
            "name": "default",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $LLM_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            },
            "examples": [
              "what is the weather today",
              "tell me a joke"
            ]
          }
        ]
      }
    }
  }
EOF

发送一个编程请求：

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Help me debug this Python code"
      }
    ]
  }'

你应该会收到 HTTP/1.1 200 OK。响应头会显示选中的实例以及用于决策的得分：

X-AI-Semantic-Picked-Instance: code
X-AI-Semantic-Scores: code:0.8213,default:0.2451,translate:0.1904

得分取决于向量嵌入模型和示例语句。你可以据此选择一个阈值，将应按相似度路由的提示词与应使用后备实例的提示词区分开。后备实例仍然需要配置 examples，因为它也会参与常规语义排序。

启用语义选择时，不会使用已配置的健康检查设置，且发生上游失败后不会重试。因此，fallback_strategy、fallback_http_statuses 和 max_retries 不会影响语义选择。选中实例的上游失败会直接返回给客户端。只有在没有实例达到阈值或向量嵌入请求失败时，才会使用语义后备实例。

调试响应头会暴露实例名称和得分。调优完成后，请禁用 semantic_opts.debugging，并在生产环境中使用为后备决策输出的 warning 日志。

在 Gemini 和 Vertex AI 之间进行负载均衡​

本示例在 Google AI Studio 和 Vertex AI 中的 Gemini 之间对兼容 OpenAI Chat Completions 格式的请求进行负载均衡。实例权重会将大约 70% 的请求发送到 Google AI Studio，将 30% 的请求发送到 Vertex AI。API7 企业版 3.9.2 和 APISIX 3.15.0 开始支持此功能。

在继续之前：

  1. 对于 Google AI Studio Gemini，请获取 Gemini API 密钥。
  2. 对于 Vertex AI Gemini，请按照 Vertex AI 项目设置说明启用结算和 Vertex AI API。创建服务账号，为其分配“Vertex AI User”角色，并创建 JSON 格式的服务账号密钥。

分别保存 API 密钥、供 ADC 示例使用的原始服务账号 JSON，以及供 Admin API 示例使用的 JSON 编码字符串：

export GEMINI_API_KEY="<GEMINI_API_KEY>"
export GCP_SA_JSON="$(cat credentials.json)"
export GCP_SA_JSON_STRING="$(jq -Rs . < credentials.json)"

创建路由，并替换项目相关的值：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-google-ai-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "gemini-instance",
            "provider": "gemini",
            "weight": 7,
            "auth": {
              "header": {
                "Authorization": "Bearer $GEMINI_API_KEY"
              }
            },
            "options": {
              "model": "gemini-3.5-flash"
            }
          },
          {
            "name": "vertex-ai-instance",
            "provider": "vertex-ai",
            "weight": 3,
            "auth": {
              "gcp": {
                "service_account_json": $GCP_SA_JSON_STRING
              }
            },
            "provider_conf": {
              "project_id": "<GCP_PROJECT_ID>",
              "region": "northamerica-northeast1"
            },
            "options": {
              "model": "google/gemini-3.5-flash"
            }
          }
        ]
      }
    }
  }
EOF

❶ 将提供商配置为 gemini，以访问 Google AI Studio Gemini。

❷ 在 Authorization 请求头中替换为你的 Gemini API Key。

❸ 以 <model> 格式指定通过 Google AI Studio 使用的 Gemini 模型名称。

❹ 将提供商配置为 vertex-ai，以访问 Vertex AI Gemini。

❺ 以 JSON 字符串形式提供服务账号凭据。

❻ 替换为你的 Vertex AI 项目 ID 和区域。

❼ 以 <publisher>/<model> 格式指定通过 Vertex AI 使用的 Gemini 模型名称。

向该路由发送 10 个 POST 请求，以查看负载均衡分布：

studio_count=0
vertex_count=0

for i in {1..10}; do
  model=$(curl -s "http://127.0.0.1:9080/anything" -X POST \
    -H "Content-Type: application/json" \
    -d '{
      "messages": [
        { "role": "system", "content": "You are a mathematician" },
        { "role": "user", "content": "What is 1+1?" }
      ]
    }' | jq -r '.model')

  if [[ "$model" == "gemini-3.5-flash" ]]; then
    ((studio_count++))
  elif [[ "$model" == "google/gemini-3.5-flash" ]]; then
    ((vertex_count++))
  fi
done

echo "Google AI Studio Gemini responses: $studio_count"
echo "Vertex AI Gemini responses: $vertex_count"

你应该看到类似于以下的响应：

Google AI Studio Gemini responses: 7
Vertex AI Gemini responses: 3

配置实例优先级和速率限制​

本示例为两个模型分配不同的优先级，并对高优先级实例进行限流。当 fallback_strategy 设置为 ["rate_limiting"] 时，高优先级实例耗尽配额后，请求会转发到低优先级实例。

创建路由，并按需替换模型服务提供方相关的值：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "priority": 1,
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "priority": 0,
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ]
      },
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "openai-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "limit_strategy": "total_tokens"
      }
    }
  }
EOF

❶ 将 fallback_strategy 设置为 ["rate_limiting"]。

❷ 为 openai-instance 设置较高的优先级。

❸ 为 deepseek-instance 设置较低的优先级。

❹ 配置 10 个 Token 的配额。

❺ 将时间窗口配置为 60 秒。

❻ 将 limit_strategy 配置为 total_tokens。

发送一个 POST 请求到该路由，请求体中包含系统提示词和一个示例用户问题：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

你应该会收到类似以下的响应：

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

由于 total_tokens 值超过了配置的 10 个 Token 配额，因此在 60 秒窗口内的下一个请求预计将被转发到另一个实例。

在相同的 60 秒窗口内，发送另一个 POST 请求到该路由：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newton law" }
    ]
  }'

你应该看到类似于以下的响应：

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Certainly! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics.\n\n---\n\n### **1. Newton's First Law (Law of Inertia):**\n- **Statement:** An object at rest will remain at rest, and an object in motion will continue moving at a constant velocity (in a straight line at a constant speed), unless acted upon by an external force.\n- **Key Idea:** This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion.\n- **Example:** If you slide a book across a table, it eventually stops because of the force of friction acting on it. Without friction, the book would keep moving indefinitely.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration):**\n- **Statement:** The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n  where:\n  - \\( F \\) = net force applied (in Newtons),\n  -"
      },
      ...
    }
  ],
  ...
}

按消费者负载均衡和速率限制​

本示例在两个模型之间进行负载均衡，并为每个消费者分别应用限流。

创建消费者 ai-multi-johndoe，并为 openai-instance 配置每 60 秒 10 个 Token 的配额：

curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "username": "ai-multi-johndoe",
    "plugins": {
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "openai-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "rejected_code": 429,
        "policy": "local",
        "limit_strategy": "total_tokens"
      }
    }
  }'

为 ai-multi-johndoe 添加 key-auth 凭据：

curl "http://127.0.0.1:9180/apisix/admin/consumers/ai-multi-johndoe/credentials" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "cred-john-key-auth",
    "plugins": {
      "key-auth": {
        "key": "john-key"
      }
    }
  }'

创建消费者 ai-multi-janedoe，并为 deepseek-instance 配置每 60 秒 10 个 Token 的配额：

curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "username": "ai-multi-janedoe",
    "plugins": {
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "deepseek-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "rejected_code": 429,
        "policy": "local",
        "limit_strategy": "total_tokens"
      }
    }
  }'

为 ai-multi-janedoe 添加 key-auth 凭据：

curl "http://127.0.0.1:9180/apisix/admin/consumers/ai-multi-janedoe/credentials" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "cred-jane-key-auth",
    "plugins": {
      "key-auth": {
        "key": "jane-key"
      }
    }
  }'

创建路由，并按需替换模型服务提供方相关的值：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "key-auth": {},
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ]
      }
    }
  }
EOF

❶ 在路由上启用 key-auth。

❷ 配置 openai 实例。

❸ 配置 deepseek 实例。

发送一个不带任何消费者密钥的 POST 请求到该路由：

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

你应该会收到一个 HTTP/1.1 401 Unauthorized 响应。

使用 ai-multi-johndoe 的密钥向路由发送 POST 请求：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: john-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

你应该会收到类似以下的响应：

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

由于 total_tokens 的值超过了为 ai-multi-johndoe 配置的 openai 实例配额，因此在同一个 60 秒窗口内，ai-multi-johndoe 的下一个请求应转发到 deepseek 实例。

在同一个 60 秒窗口内，再使用 ai-multi-johndoe 的密钥向路由发送一个 POST 请求：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: john-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws to me" }
    ]
  }'

你应该看到类似于以下的响应：

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Certainly! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics.\n\n---\n\n### **1. Newton's First Law (Law of Inertia):**\n- **Statement:** An object at rest will remain at rest, and an object in motion will continue moving at a constant velocity (in a straight line at a constant speed), unless acted upon by an external force.\n- **Key Idea:** This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion.\n- **Example:** If you slide a book across a table, it eventually stops because of the force of friction acting on it. Without friction, the book would keep moving indefinitely.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration):**\n- **Statement:** The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n  where:\n  - \\( F \\) = net force applied (in Newtons),\n  -"
      },
      ...
    }
  ],
  ...
}

使用 ai-multi-janedoe 的密钥向路由发送 POST 请求：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: jane-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

你应该会收到类似以下的响应：

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The sum of 1 and 1 is 2. This is a basic arithmetic operation where you combine two units to get a total of two units."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 31,
    "total_tokens": 45,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "prompt_cache_hit_tokens": 0,
    "prompt_cache_miss_tokens": 14
  },
  "system_fingerprint": "fp_3a5770e1b4_prod0225"
}

由于 total_tokens 的值超过了为 ai-multi-janedoe 配置的 deepseek 实例配额，因此在同一个 60 秒窗口内，ai-multi-janedoe 的下一个请求应转发到 openai 实例。

在同一个 60 秒窗口内，再使用 ai-multi-janedoe 的密钥向路由发送一个 POST 请求：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: jane-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws to me" }
    ]
  }'

你应该看到类似于以下的响应：

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Sure, here are Newton's three laws of motion:\n\n1) Newton's First Law, also known as the Law of Inertia, states that an object at rest will stay at rest, and an object in motion will stay in motion, unless acted on by an external force. In simple words, this law suggests that an object will keep doing whatever it is doing until something causes it to do otherwise. \n\n2) Newton's Second Law states that the force acting on an object is equal to the mass of that object times its acceleration (F=ma). This means that force is directly proportional to mass and acceleration. The heavier the object and the faster it accelerates, the greater the force.\n\n3) Newton's Third Law, also known as the law of action and reaction, states that for every action, there is an equal and opposite reaction. Essentially, any force exerted onto a body will create a force of equal magnitude but in the opposite direction on the object that exerted the first force.\n\nRemember, these laws become less accurate when considering speeds near the speed of light (where Einstein's theory of relativity becomes more appropriate) or objects very small or very large. However, for everyday situations, they provide a good model of how things move.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

这显示了 ai-proxy-multi 如何根据 ai-rate-limiting 中针对消费者的速率限制规则来负载均衡流量。

限制最大补全 Token 数​

本示例限制实例可生成的补全 Token 数量。

该示例使用 OpenAI 和 DeepSeek 实例，以便清楚区分所选的上游。

创建路由，并按需替换模型服务提供方相关的值：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini",
              "max_tokens": 50
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat",
              "max_tokens": 100
            }
          }
        ]
      }
    }
  }
EOF

❶ 将 OpenAI 实例的 max_tokens 配置为 50。

❷ 将 DeepSeek 实例的 max_tokens 配置为 100。

发送一个 POST 请求到该路由，请求体中包含系统提示词和一个示例用户问题：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons law" }
    ]
  }'

如果请求被代理到 OpenAI，你应该会看到类似于以下的响应，其中内容根据 50 个 max_tokens 阈值被截断：

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Newton's Laws of Motion are three physical laws that form the bedrock for classical mechanics. They describe the relationship between a body and the forces acting upon it, and the body's motion in response to those forces. \n\n1. Newton's First Law",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 50,
    "total_tokens": 70,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

如果请求被代理到 DeepSeek，你应该会看到类似于以下的响应，其中内容根据 100 个 max_tokens 阈值被截断：

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Newton's Laws of Motion are three fundamental principles that form the foundation of classical mechanics. They describe the relationship between a body and the forces acting upon it, and the body's motion in response to those forces. Here's a brief explanation of each law:\n\n1. **Newton's First Law (Law of Inertia):**\n   - **Statement:** An object will remain at rest or in uniform motion in a straight line unless acted upon by an external force.\n   - **Explanation:** This law"
      },
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 100,
    "total_tokens": 110,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "prompt_cache_hit_tokens": 0,
    "prompt_cache_miss_tokens": 10
  },
  "system_fingerprint": "fp_3a5770e1b4_prod0225"
}

代理到嵌入模型​

本示例在 OpenAI 和 Azure OpenAI 之间分发 Embeddings 请求。Azure 实例使用 openai-compatible 模型服务提供方，因此部署名称会保留在 Azure OpenAI v1 API 所需的请求体中。Responses 和 Embeddings 请求都包含 input；由于此路由不以 /v1/responses 结尾，插件会将其识别为 Embeddings 请求。

创建路由，并按需替换模型服务提供方相关的值：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "text-embedding-3-small"
            }
          },
          {
            "name": "az-openai-instance",
            "provider": "openai-compatible",
            "weight": 0,
            "auth": {
              "header": {
                "api-key": "$AZ_OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "<AZURE_OPENAI_EMBEDDING_DEPLOYMENT>"
            },
            "override": {
              "endpoint": "https://<AZURE_OPENAI_RESOURCE>.openai.azure.com/openai/v1/embeddings"
            }
          }
        ]
      }
    }
  }
EOF

❶ 指定 OpenAI 向量嵌入模型。openai 提供方会将检测到的 Embeddings 请求发送到 https://api.openai.com/v1/embeddings。

❷ 指定 Azure OpenAI 嵌入部署名称。

❸ 指定 Azure OpenAI v1 Embeddings 端点。

向该路由发送一个包含输入字符串的 POST 请求：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "input": "hello world"
  }'

你应该会收到类似以下的响应：

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [
        -0.0067144386,
        -0.039197803,
        0.034177095,
        0.028763203,
        -0.024785956,
        -0.04201061,
        ...
      ],
    }
  ],
  "model": "text-embedding-3-small",
  "usage": {
    "prompt_tokens": 2,
    "total_tokens": 2
  }
}

启用主动健康检查​

本示例在多个模型之间对请求进行负载均衡，并启用主动健康检查。可以在一个或多个实例上启用健康检查。

创建路由，并按需替换模型服务提供方相关的值和健康检查设置：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "llm-instance-1",
            "provider": "openai-compatible",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $YOUR_LLM_API_KEY"
              }
            },
            "options": {
              "model": "$YOUR_LLM_MODEL"
            }
          },
          {
            "name": "llm-instance-2",
            "provider": "openai-compatible",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $YOUR_LLM_API_KEY"
              }
            },
            "options": {
              "model": "$YOUR_LLM_MODEL"
            },
            "checks": {
              "active": {
                "type": "https",
                "host": "yourhost.com",
                "http_path": "/your/probe/path",
                "healthy": {
                  "interval": 2,
                  "successes": 1
                },
                "unhealthy": {
                  "interval": 1,
                  "http_failures": 3
                }
              }
            }
          }
        ]
      }
    }
  }
EOF

❶ 更新主动健康检查的类型。

❷ 如果可用，更新主机。

❸ 更新探测路径。

❹ 配置定期检查健康节点的时间间隔（秒）。

❺ 配置判定上游节点健康的成功次数阈值。

❻ 配置定期检查不健康节点的时间间隔（秒）。

❼ 配置判定上游节点不健康的超时次数阈值。

验证时，行为应与主动健康检查中的验证一致。

自 API7 企业版 3.9.20 和 3.10.7 起，每个实例的健康状态也会通过数据面 Control API 上报。GET /v1/healthcheck 会为每个声明了 checks 的实例列出一个条目，其中 plugin 为 ai-proxy-multi，meta.instance 给出实例名。GET /v1/healthcheck/{src_type}/{src_id}/checkers 会返回该路由或服务拥有的全部健康检查器。

在访问日志中包含 LLM 信息​

本示例将 LLM 请求信息添加到网关访问日志，以便进行分析和审计。除 NGINX 变量外，还可以使用以下变量：

  • apisix_upstream_response_time：网关向上游服务发送请求并接收完整响应所用的时间，单位为毫秒。LLM 上游返回错误响应时，也会以毫秒为单位记录。自 API7 企业版 3.8.8 起可用。
  • request_type：请求类型，取值可以是 traditional_http、ai_chat 或 ai_stream。
  • llm_time_to_first_token：从发送请求到收到 LLM 服务第一个 Token 的时间，单位为毫秒。上游返回错误响应时，该值表示收到错误响应前的时间；如果网关无法连接上游，则该值保持为 0。
  • llm_model：转发到上游 LLM 服务的 LLM 模型名称。
  • request_llm_model：请求中指定的 LLM 模型名称。
  • llm_prompt_tokens：提示词中的 Token 数量。
  • llm_completion_tokens：响应中的补全 Token 数量。
  • llm_content_risk_level：AI 内容审核插件报告的内容风险等级。该变量在 APISIX 3.14.0 中引入。

对于通过 Lua socket transport 发送的请求，网关还会填充标准上游变量 upstream_addr、upstream_status、apisix_upstream_response_time、upstream_scheme、upstream_host、upstream_uri 和 upstream_response_length。这适用于 API7 企业版 3.10.0 及更高版本和 APISIX 3.17.0 及更高版本中的流式与非流式 AI 响应。

这些变量适用于 API7 企业版 3.9.11 及更高版本和 APISIX 3.17.0 及更高版本中的流式与非流式 AI 响应。对于通过 FFI 客户端发送的 APISIX 请求，这些变量也会被填充。

以下其他变量在 API7 企业版 3.9.14 和 APISIX 3.18.0 中引入：

  • llm_total_tokens：使用的 Token 总数，包括提示词和补全 Token。
  • llm_stream：请求是否为流式请求，值为 true 或 false。
  • llm_has_tool_calls：LLM 响应是否包含工具调用，值为 true 或 false。
  • llm_tool_count：请求中提供的工具数量。
  • llm_end_user_id：从请求体提取的终端用户标识符，例如 user、safety_identifier 或 metadata.user_id。
  • llm_cache_read_input_tokens：从服务提供方提示词缓存中读取的提示词 Token 数量。
  • llm_cache_creation_input_tokens：写入服务提供方提示词缓存的提示词 Token 数量。
  • llm_reasoning_tokens：推理模型使用的推理 Token 数量。
提示

访问日志示例使用这些变量，日志插件也可以使用它们。在 API7 企业版 3.9.18 和 3.10.5 以及 APISIX 3.18.0 中，启用 logging.summaries 会将这些变量添加到 llm_summary 对象，并移除 llm_ 前缀。请求中未产生的值会被省略。

若要在访问日志中记录这些值，请将 LLM 变量添加到网关访问日志格式中：

在网关配置文件中添加或更新以下配置：

config.yaml
nginx_config:
  http:
    access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $apisix_upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$request_llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\" \"$llm_total_tokens\" \"$llm_stream\" \"$llm_has_tool_calls\" \"$llm_tool_count\" \"$llm_end_user_id\" \"$llm_cache_read_input_tokens\" \"$llm_cache_creation_input_tokens\" \"$llm_reasoning_tokens\""

重新加载网关，使配置更改生效。

按照在实例之间进行负载均衡创建路由，然后发送请求：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

当 ai-proxy-multi 选择 OpenAI 实例时，插件会将请求中的模型替换为 gpt-4.1-mini。你将收到类似以下内容的响应：

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null,
        "annotations": []
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    ...
  },
  "service_tier": "default",
  "system_fingerprint": null
}

在网关的访问日志中，你应该会看到类似于以下的日志条目：

192.168.215.1 - - [<TIMESTAMP>] 127.0.0.1:9080 "POST /anything HTTP/1.1" 200 808 2.670 "-" "curl/8.6.0" - - 2670 "http://127.0.0.1:9080" "<REQUEST_ID>" "ai_chat" "2670" "gpt-4.1-mini" "gpt-4o-mini" "23" "8" "31" "false" "false" "0" "" "0" "0" "0"

访问日志条目显示，上游响应时间和首个 Token 的响应时间均为 2670 毫秒。请求类型为 ai_chat，请求的模型是 gpt-4o-mini，并被转发到 gpt-4.1-mini。请求使用了 23 个提示词 Token、8 个补全 Token，共计 31 个 Token。

其余值表示这是一个非流式请求，且不包含工具调用、提供的工具、最终用户标识符、提示缓存 Token 或推理 Token。

将 AI 请求日志发送到 Kafka​

以下示例将 AI 请求和响应详情（包括模型、Token 用量和负载）发送到 Kafka。请先按照 kafka-logger 指南完成 broker 和 apisix-logs 主题配置。

创建到你的 LLM 服务的路由，并启用 Kafka 日志记录：

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ],
        "logging": {
          "summaries": true,
          "payloads": true
        }
      },
      "kafka-logger": {
        "brokers": [
          {
            "host": "kafka-server",
            "port": 9092
          }
        ],
        "kafka_topic": "apisix-logs",
        "key": "key1",
        "batch_max_size": 1
      }
    }
  }
EOF

❶ 记录请求的 LLM 模型、持续时间、请求和响应 Token 数。

❷ 记录请求和响应负载。

❸ 更新为你的 Kafka 地址。

❹ 更新为你的 Kafka 主题。

❺ 更新为你的 Kafka 密钥。

❻ 设置为 1 以立即发送日志条目。

向该路由发送一个 POST 请求：

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

如果请求被转发到 OpenAI，你应该会收到类似于以下的响应：

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

在 Kafka 主题中，你还应该看到与该请求对应的日志条目，其中包含 LLM 摘要以及请求和响应负载。转发到 OpenAI 的请求会产生类似以下字段：

{
  "llm_summary": {
    "model": "gpt-4.1-mini",
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31
  },
  "llm_request": {
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ],
    "stream": false
  },
  "llm_response_text": {
    "content": "1+1 equals 2."
  }
}