elasticsearch-logger
elasticsearch-logger 插件将请求和响应日志分批推送到 Elasticsearch 并支持自定义日志格式。启用后，该插件会将请求上下文信息序列化为 Elasticsearch Bulk 格式 并将它们添加到队列中，然后再推送到 Elasticsearch。
示例
以下示例展示了如何在不同场景下配置 elasticsearch-logger 插件。
要跟随示例操作，请启动 Elasticsearch 和 Kibana：
- Docker
- Kubernetes
要跟随示例操作，请在 Docker 中启动一个 Elasticsearch 实例：
docker run -d \
--name elasticsearch \
--network apisix-quickstart-net \
-v elasticsearch_vol:/usr/share/elasticsearch/data/ \
-p 9200:9200 \
-p 9300:9300 \
-e ES_JAVA_OPTS="-Xms512m -Xmx512m" \
-e discovery.type=single-node \
-e xpack.security.enabled=false \
docker.elastic.co/elasticsearch/elasticsearch:7.17.1
在 Docker 中启动一个 Kibana 实例以可视化 Elasticsearch 中的索引数据：
docker run -d \
--name kibana \
--network apisix-quickstart-net \
-p 5601:5601 \
-e ELASTICSEARCH_HOSTS="http://elasticsearch:9200" \
docker.elastic.co/kibana/kibana:7.17.1
为 Elasticsearch Deployment 创建 Kubernetes 清单文件：
elasticsearch-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: aic
name: elasticsearch
spec:
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
env:
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
- name: discovery.type
value: single-node
- name: xpack.security.enabled
value: "false"
ports:
- containerPort: 9200
- containerPort: 9300
为 Elasticsearch Service 创建 Kubernetes 清单文件：
elasticsearch-service.yaml
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: elasticsearch
spec:
selector:
app: elasticsearch
ports:
- name: http
port: 9200
targetPort: 9200
- name: transport
port: 9300
targetPort: 9300
type: ClusterIP
为 Kibana Deployment 创建 Kubernetes 清单文件：
kibana-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: aic
name: kibana
spec:
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.17.1
env:
- name: ELASTICSEARCH_HOSTS
value: "http://elasticsearch.aic.svc:9200"
ports:
- containerPort: 5601
为 Kibana Service 创建 Kubernetes 清单文件：
kibana-service.yaml
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: kibana
spec:
selector:
app: kibana
ports:
- name: http
port: 5601
targetPort: 5601
type: ClusterIP
应用清单：
kubectl apply -f elasticsearch-deployment.yaml -f elasticsearch-service.yaml -f kibana-deployment.yaml -f kibana-service.yaml
要访问 Kibana，请转发 Service 端口：
kubectl port-forward -n aic svc/kibana 5601:5601
如果成功，你应该能在 localhost:5601 看到 Kibana 仪表板。
使用默认日志格式记录日志
以下示例展示了如何在路由上启用 elasticsearch-logger 插件，该插件记录客户端请求和响应，并将日志推送到 Elasticsearch。
创建一个启用 elasticsearch-logger 的路由如下：
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "elasticsearch-logger-route",
"uri": "/anything",
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": ["http://elasticsearch:9200"],
"field": {
"index": "gateway",
"type": "logs"
}
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}'
adc.yaml
services:
- name: httpbin
routes:
- uris:
- /anything
name: elasticsearch-logger-route
plugins:
elasticsearch-logger:
endpoint_addrs:
- "http://elasticsearch:9200"
field:
index: gateway
type: logs
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
将配置同步到网关：
adc sync -f adc.yaml
- Gateway API
- APISIX CRD
elasticsearch-logger-ic.yaml
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: elasticsearch-logger-plugin-config
spec:
plugins:
- name: elasticsearch-logger
config:
endpoint_addrs:
- "http://elasticsearch.aic.svc:9200"
field:
index: gateway
type: logs
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: elasticsearch-logger-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: elasticsearch-logger-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
elasticsearch-logger-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: elasticsearch-logger-route
spec:
ingressClassName: apisix
http:
- name: elasticsearch-logger-route
match:
paths:
- /anything
methods:
- GET
upstreams:
- name: httpbin-external-domain
plugins:
- name: elasticsearch-logger
config:
endpoint_addrs:
- "http://elasticsearch.aic.svc:9200"
field:
index: gateway
type: logs
应用配置：
kubectl apply -f elasticsearch-logger-ic.yaml
❶ 配置 Elasticsearch 的端点地址。
❷ 将 index 字段配置为 gateway。
❸ 将 type 字段配置为 logs。
发送一个请求到该路由以生成一条日志条目：
curl -i "http://127.0.0.1:9080/anything"
你应该收到 HTTP/1.1 200 OK 响应。
导航到 localhost:5601 的 Kibana 仪表板，在 Discover 标签页下，创建一个新的索引模式 gateway 以从 Elasticsearch 获取数据。配置完成后，导航回 Discover 标签页，你应该看到生成的日志，类似如下：
{
"_index": "gateway",
"_type": "logs",
"_id": "CE-JL5QBOkdYRG7kEjTJ",
"_version": 1,
"_score": 1,
"_source": {
"request": {
"headers": {
"host": "127.0.0.1:9080",
"accept": "*/*",
"user-agent": "curl/8.6.0"
},
"size": 85,
"querystring": {},
"method": "GET",
"url": "http://127.0.0.1:9080/anything",
"uri": "/anything"
},
"response": {
"headers": {
"content-type": "application/json",
"access-control-allow-credentials": "true",
"server": "APISIX/3.13.0",
"content-length": "390",
"access-control-allow-origin": "*",
"connection": "close",
"date": "Mon, 13 Jan 2025 10:18:14 GMT"
},
"status": 200,
"size": 618
},
"route_id": "elasticsearch-logger-route",
"latency": 585.00003814697,
"apisix_latency": 18.000038146973,
"upstream_latency": 567,
"upstream": "50.19.58.113:80",
"server": {
"hostname": "0b9a772e68f8",
"version": "3.13.0"
},
"service_id": "",
"client_ip": "192.168.65.1"
},
"fields": {
...
}
}
使用插件元数据自定义日志格式
以下示例展示了如何使用 插件元数据 和 内置变量 自定义日志格式，以记录请求头和响应头中的特定信息。
在 APISIX 中，插件元数据 用于配置同一插件的所有插件实例的通用元数据字段。当一个插件在多个资源中启用并需要统一更新其元数据字段时，这非常有用。
首先，创建一个启用 elasticsearch-logger 的路由如下：
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "elasticsearch-logger-route",
"uri": "/anything",
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": ["http://elasticsearch:9200"],
"field": {
"index": "gateway",
"type": "logs"
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}'
接下来，配置 elasticsearch-logger 的插件元数据：
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/elasticsearch-logger" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"log_format": {
"host": "$host",
"@timestamp": "$time_iso8601",
"client_ip": "$remote_addr",
"env": "$http_env",
"resp_content_type": "$sent_http_Content_Type"
}
}'
adc.yaml
plugin_metadata:
- name: elasticsearch-logger
log_format:
host: "$host"
"@timestamp": "$time_iso8601"
client_ip: "$remote_addr"
env: "$http_env"
resp_content_type: "$sent_http_Content_Type"
将配置同步到网关：
adc sync -f adc.yaml
elasticsearch-logger-metadata.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
namespace: aic
name: apisix-config
spec:
provider:
type: ControlPlane
controlPlane:
service:
name: apisix-admin
port: 9180
auth:
type: AdminKey
adminKey:
value: edd1c9f034335f136f87ad84b625c8f1
pluginMetadata:
elasticsearch-logger:
log_format:
host: "$host"
"@timestamp": "$time_iso8601"
client_ip: "$remote_addr"
env: "$http_env"
resp_content_type: "$sent_http_Content_Type"
应用配置：
kubectl apply -f elasticsearch-logger-metadata.yaml
❶ 记录自定义请求头 env。
❷ 记录响应头 Content-Type。
发送带有 env 头的请求到该路由：
curl -i "http://127.0.0.1:9080/anything" -H "env: dev"
你应该收到 HTTP/1.1 200 OK 响应。
导航到 localhost:5601 的 Kibana 仪表板，在 Discover 标签页下，创建一个新的索引模式 gateway 以从 Elasticsearch 获取数据（如果你尚未创建）。配置完成后，导航回 Discover 标签页，你应该看到生成的日志，类似如下：
{
"_index": "gateway",
"_type": "logs",
"_id": "Ck-WL5QBOkdYRG7kODS0",
"_version": 1,
"_score": 1,
"_source": {
"client_ip": "192.168.65.1",
"route_id": "elasticsearch-logger-route",
"@timestamp": "2025-01-06T10:32:36+00:00",
"host": "127.0.0.1",
"resp_content_type": "application/json"
},
"fields": {
...
}
}
有条件地记录请求体
以下示例展示了如何有条件地记录请求体。
创建一个启用 elasticsearch-logger 的路由如下：
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": ["http://elasticsearch:9200"],
"field": {
"index": "gateway",
"type": "logs"
},
"include_req_body": true,
"include_req_body_expr": [["arg_log_body", "==", "yes"]]
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
},
"uri": "/anything",
"id": "elasticsearch-logger-route"
}'
adc.yaml
services:
- name: httpbin
routes:
- uris:
- /anything
name: elasticsearch-logger-route
plugins:
elasticsearch-logger:
endpoint_addrs:
- "http://elasticsearch:9200"
field:
index: gateway
type: logs
include_req_body: true
include_req_body_expr:
- - arg_log_body
- "=="
- "yes"
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
将配置同步到网关：
adc sync -f adc.yaml
- Gateway API
- APISIX CRD
elasticsearch-logger-ic.yaml
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: elasticsearch-logger-plugin-config
spec:
plugins:
- name: elasticsearch-logger
config:
endpoint_addrs:
- "http://elasticsearch.aic.svc:9200"
field:
index: gateway
type: logs
include_req_body: true
include_req_body_expr:
- - arg_log_body
- "=="
- "yes"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: elasticsearch-logger-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: elasticsearch-logger-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
elasticsearch-logger-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: elasticsearch-logger-route
spec:
ingressClassName: apisix
http:
- name: elasticsearch-logger-route
match:
paths:
- /anything
methods:
- GET
upstreams:
- name: httpbin-external-domain
plugins:
- name: elasticsearch-logger
config:
endpoint_addrs:
- "http://elasticsearch.aic.svc:9200"
field:
index: gateway
type: logs
include_req_body: true
include_req_body_expr:
- - arg_log_body
- "=="
- "yes"
应用配置：
kubectl apply -f elasticsearch-logger-ic.yaml
❶ include_req_body: 设置为 true 以包含请求体。
❷ include_req_body_expr: 仅当 URL 查询字符串 log_body 为 true 时包含请求体。
发送一个满足条件的请求到该路由：
curl -i "http://127.0.0.1:9080/anything?log_body=yes" -X POST -d '{"env": "dev"}'
你应该收到 HTTP/1.1 200 OK 响应。
导航到 localhost:5601 的 Kibana 仪表板，在 Discover 标签页下，创建一个新的索引模式 gateway 以从 Elasticsearch 获取数据（如果你尚未创建）。配置完成后，导航回 Discover 标签页，你应该看到生成的日志，类似如下：
{
"_index": "gateway",
"_type": "logs",
"_id": "Dk-cL5QBOkdYRG7k7DSW",
"_version": 1,
"_score": 1,
"_source": {
"request": {
"headers": {
"user-agent": "curl/8.6.0",
"accept": "*/*",
"content-length": "14",
"host": "127.0.0.1:9080",
"content-type": "application/x-www-form-urlencoded"
},
"size": 182,
"querystring": {
"log_body": "yes"
},
"body": "{\"env\": \"dev\"}",
"method": "POST",
"url": "http://127.0.0.1:9080/anything?log_body=yes",
"uri": "/anything?log_body=yes"
},
"start_time": 1735965595203,
"response": {
"headers": {
"content-type": "application/json",
"server": "APISIX/3.13.0",
"access-control-allow-credentials": "true",
"content-length": "548",
"access-control-allow-origin": "*",
"connection": "close",
"date": "Mon, 13 Jan 2025 11:02:32 GMT"
},
"status": 200,
"size": 776
},
"route_id": "elasticsearch-logger-route",
"latency": 703.9999961853,
"apisix_latency": 34.999996185303,
"upstream_latency": 669,
"upstream": "34.197.122.172:80",
"server": {
"hostname": "0b9a772e68f8",
"version": "3.13.0"
},
"service_id": "",
"client_ip": "192.168.65.1"
},
"fields": {
...
}
}
发送一个不带 URL 查询字符串的请求到该路由：
curl -i "http://127.0.0.1:9080/anything" -X POST -d '{"env": "dev"}'
导航到 Kibana 仪表板的 Discover 标签页，你应该看到生成的日志，但不包含请求体：
{
"_index": "gateway",
"_type": "logs",
"_id": "EU-eL5QBOkdYRG7kUDST",
"_version": 1,
"_score": 1,
"_source": {
"request": {
"headers": {
"content-type": "application/x-www-form-urlencoded",
"accept": "*/*",
"content-length": "14",
"host": "127.0.0.1:9080",
"user-agent": "curl/8.6.0"
},
"size": 169,
"querystring": {},
"method": "POST",
"url": "http://127.0.0.1:9080/anything",
"uri": "/anything"
},
"start_time": 1735965686363,
"response": {
"headers": {
"content-type": "application/json",
"access-control-allow-credentials": "true",
"server": "APISIX/3.13.0",
"content-length": "510",
"access-control-allow-origin": "*",
"connection": "close",
"date": "Mon, 13 Jan 2025 11:15:54 GMT"
},
"status": 200,
"size": 738
},
"route_id": "elasticsearch-logger-route",
"latency": 680.99999427795,
"apisix_latency": 4.9999942779541,
"upstream_latency": 676,
"upstream": "34.197.122.172:80",
"server": {
"hostname": "0b9a772e68f8",
"version": "3.13.0"
},
"service_id": "",
"client_ip": "192.168.65.1"
},
"fields": {
...
}
}
信息
如 果你在设置 include_req_body 或 include_resp_body 为 true 的同时自定义了 log_format，插件将不会在日志中包含这些内容。
作为变通方法，你可以在日志格式中使用 NGINX 变量 $request_body，例如：
{
"elasticsearch-logger": {
...,
"log_format": {"body": "$request_body"}
}
}
在 Elasticsearch 索引中包含请求日期
以下示例展示了如何配置 elasticsearch-logger 插件以在 Elasticsearch 索引中包含请求日期。
创建一个启用 elasticsearch-logger 的路由如下：
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "elasticsearch-logger-route",
"uri": "/anything",
"plugins": {
"elasticsearch-logger": {
"endpoint_addrs": ["http://elasticsearch:9200"],
"field": {
"index": "api7-{%Y.%m.%d}",
"type": "logs"
}
}
},
"upstream": {
"nodes": {
"httpbin.org:80": 1
},
"type": "roundrobin"
}
}'
adc.yaml
services:
- name: httpbin
routes:
- uris:
- /anything
name: elasticsearch-logger-route
plugins:
elasticsearch-logger:
endpoint_addrs:
- "http://elasticsearch:9200"
field:
index: "api7-{%Y.%m.%d}"
type: logs
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
将配置同步到网关：
adc sync -f adc.yaml
- Gateway API
- APISIX CRD
elasticsearch-logger-ic.yaml
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: elasticsearch-logger-plugin-config
spec:
plugins:
- name: elasticsearch-logger
config:
endpoint_addrs:
- "http://elasticsearch.aic.svc:9200"
field:
index: "api7-{%Y.%m.%d}"
type: logs
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: elasticsearch-logger-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: elasticsearch-logger-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
elasticsearch-logger-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: elasticsearch-logger-route
spec:
ingressClassName: apisix
http:
- name: elasticsearch-logger-route
match:
paths:
- /anything
methods:
- GET
upstreams:
- name: httpbin-external-domain
plugins:
- name: elasticsearch-logger
config:
endpoint_addrs:
- "http://elasticsearch.aic.svc:9200"
field:
index: "api7-{%Y.%m.%d}"
type: logs
应用配置：
kubectl apply -f elasticsearch-logger-ic.yaml
❶ 配置 Elasticsearch 的端点地址。
❷ 配置 index 字段以使用当前年、月和日。
❸ 将 type 字段配置为 logs。
发送一个请求到该路由以生成一条日志条目：
curl -i "http://127.0.0.1:9080/anything"
你应该收到 HTTP/1.1 200 OK 响应。
导航到 localhost:5601 的 Kibana 仪表板，在 Discover 标签页下，创建一个新的索引模式 api7* 以从 Elasticsearch 获取数据。配置完成后，导航回 Discover 标签页，你应该看到生成的日志，类似如下：
{
"_index": "api7-2025.3.10",
"_type": "logs",
"_id": "CE-KL5QB0kdYRG7dEiTJ",
"_version": 1,
"_score": 1,
"_source": {
"request": {
...
},
"response": {
...
},
"status": 200,
"size": 618
},
...
}