最近看到一篇公众号讲了nginx-ingress-controller的应用。下面有人评论如何做日志持久化，刚好工作上遇到该问题，整理一个方案，仅供参考。
nginx-ingress-controller的日志
nginx-ingress-controller的日志包括三个部分：
controller日志： 输出到stdout，通过启动参数中的–log_dir可已配置输出到文件，重定向到文件后会自动轮转，但不会自动清理 accesslog：输出到stdout，通过nginx-configuration中的字段可以配置输出到哪个文件。输出到文件后不会自动轮转或清理 errorlog：输出到stderr，配置方式与accesslog类似。给controller日志落盘
给nginx-ingress-controller挂一个hostpath： /data/log/nginx_ingress_controller/ 映射到容器里的/var/log/nginx_ingress_controller/ ， 给nginx-ingress-controller配置log-dir和logtostderr参数，将日志重定向到/var/log/nginx_ingress_controller/中。controller的日志需要做定时清理。由于controller的日志是通过klog(k8s.io/klog)输出的，会进行日志滚动，所以我们通过脚本定时清理一定时间之前的日志文件即可。
给nginx日志落盘
修改configmap： nginx-configuration。配置accesslog和errorlog的输出路径，替换默认的stdout和stderr。输出路径我们可以与controller一致，便于查找。
accesslog和errorlog都只有一个日志文件，我们可以使用logrotate进行日志轮转，将输出到宿主机上的日志进行轮转和清理。配置如：
?
1
2
3
4
5
6
7
8
9
10
$ cat /etc/logrotate.d/nginx.log
/data/log/nginx_ingress_controller/access.log {
su root list
rotate 7
daily
maxsize 50M
copytruncate
missingok
create 0644 www-data root
}
官方提供的模板中，nginx-ingress-controller默认都是以33这个用户登录启动容器的，因此挂载hostpath路径时存在权限问题。我们需要手动在机器上执行chown -R 33:33 /data/log/nginx_ingress_controller.
自动化ops
nginx日志落盘中，第2、3两点均需要人工运维，有什么解决办法吗？
问题的关键是：有什么办法可以在nginx-ingress-controller容器启动之前加一个hook，将宿主机的指定目录执行chown呢？
可以用initContainer。initcontainer必须在containers中的容器运行前运行完毕并成功退出。利用这一k8s特性，我们开发一个docker image，里面只执行如下脚本：
?
1
2
3
4
5
#!/bin/bash
logdir=$LOG_DIR
userID=$USER_ID
echo “try to set dir: $logdir s group as $userID”
chown -R $userID:$userID $logdir
脚本读取一些环境变量， 确认需要修改哪个目录，改成怎样的user group。
将脚本打包成dockerimage， 放在nginx-ingress-controller的deploy yaml中，作为initcontainers。 注意要对该initcontainer配置环境变量和volumeMount.
再说第二点，我们注意到nginx-ingress-controller的基础镜像中就自带了logrotate，那么问题就简单了，我们将写好的logrotate配置文件以configmap的形式挂载到容器中就可以了。
一个deploy yaml如下：
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
—
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: kube-system
spec:
type: ClusterIP
ports:
– name: http
port: 80
targetPort: 80
protocol: TCP
– name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app: ingress-nginx
—
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: kube-system
labels:
app: default-http-backend
spec:
ports:
– port: 80
targetPort: 8080
selector:
app: default-http-backend
—
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: default
namespace: kube-system
spec:
backend:
serviceName: default-http-backend
servicePort: 80
—
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: kube-system
labels:
app: ingress-nginx
data:
use-forwarded-headers: “true”
# 此处配置nginx日志的重定向目标
access-log-path: /var/log/nginx_ingress_controller/access.log
error-log-path: /var/log/nginx_ingress_controller/error.log
—
# 创建一个configmap，配置nginx日志的轮转策略，对应的是nginx日志在容器内的日志文件
apiVersion: v1
data:
nginx.log: |
{{ user_nginx_log.host_path }}/access.log {
rotate {{ user_nginx_log.rotate_count }}
daily
maxsize {{ user_nginx_log.rotate_size }}
minsize 10M
copytruncate
missingok
create 0644 root root
}
{{ user_nginx_log.host_path }}/error.log {
rotate {{ user_nginx_log.rotate_count }}
daily
maxsize {{ user_nginx_log.rotate_size }}
minsize 10M
copytruncate
missingok
create 0644 root root
}
kind: ConfigMap
metadata:
name: nginx-ingress-logrotate
namespace: kube-system
—
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: kube-system
—
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: kube-system
—
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: kube-system
—
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
rules:
– apiGroups:
– “”
resources:
– configmaps
– endpoints
– nodes
– pods
– secrets
verbs:
– list
– watch
– apiGroups:
– “”
resources:
– nodes
verbs:
– get
– apiGroups:
– “”
resources:
– services
verbs:
– get
– list
– watch
– apiGroups:
– “extensions”
resources:
– ingresses
verbs:
– get
– list
– watch
– apiGroups:
– “”
resources:
– events
verbs:
– create
– patch
– apiGroups:
– “extensions”
resources:
– ingresses/status
verbs:
– update
—
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: kube-system
rules:
– apiGroups:
– “”
resources:
– configmaps
– pods
– secrets
– namespaces
verbs:
– get
– apiGroups:
– “”
resources:
– configmaps
resourceNames:
# Defaults to “<election-id>-<ingress-class>”
# Here: “<ingress-controller-leader>-<nginx>”
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
– “ingress-controller-leader-nginx”
verbs:
– get
– update
– apiGroups:
– “”
resources:
– configmaps
verbs:
– create
– apiGroups:
– “”
resources:
– endpoints
verbs:
– get
—
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
– kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: kube-system
—
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
– kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: kube-system
—
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ingress-nginx
namespace: kube-system
spec:
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: 10254
prometheus.io/scrape: true
spec:
serviceAccountName: nginx-ingress-serviceaccount
tolerations:
– key: dedicated
value: ingress-nginx
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
– matchExpressions:
– key: “system/ingress”
operator: In
values:
– “true”
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
# 配置initcontainer，确保在nginx-ingress-controller容器启动前将日志目录的权限配置好
initContainers:
– name: adddirperm
image: “{{ image_registry.addr }}/{{ image.adddirperm }}”
env:
– name: LOG_DIR
value: /var/log/nginx_ingress_controller
– name: USER_ID
value: “33”
volumeMounts:
– name: logdir
mountPath: /var/log/nginx_ingress_controller
containers:
– name: nginx-ingress-controller
image: “{{ image_registry.addr }}/{{ image.ingress }}”
imagePullPolicy: IfNotPresent
args:
– /nginx-ingress-controller
– –default-backend-service=$(POD_NAMESPACE)/default-http-backend
– –configmap=$(POD_NAMESPACE)/nginx-configuration
– –tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
– –udp-services-configmap=$(POD_NAMESPACE)/udp-services
– –publish-service=$(POD_NAMESPACE)/ingress-nginx
– –annotations-prefix=nginx.ingress.kubernetes.io
# 设置controller日志的输出路径和方式
– –log_dir=/var/log/nginx_ingress_controller
– –logtostderr=false
securityContext:
capabilities:
drop:
– ALL
add:
– NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
– name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
– name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
– name: http
containerPort: 80
– name: https
containerPort: 443
resources:
requests:
cpu: 100m
memory: 256Mi
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
# 配置挂载容器中控制器组件和nginx的日志输出路径
– name: logdir
mountPath: /var/log/nginx_ingress_controller
# 配置nginx日志的logrotate配置挂载路径
– name: logrotateconf
mountPath: /etc/logrotate.d/nginx.log
subPath: nginx.log
volumes:
# 控制器组件和nginx的日志输出路径为宿主机的hostpath
– name: logdir
hostPath:
path: {{ user_nginx_log.host_path }}
type: “”
# nginx日志的轮转配置文件来自于configmap
– name: logrotateconf
configMap:
name: nginx-ingress-logrotate
items:
– key: nginx.log
path: nginx.log
—
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: default-http-backend
namespace: kube-system
labels:
app: default-http-backend
spec:
selector:
matchLabels:
app: default-http-backend
template:
metadata:
labels:
app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
tolerations:
– key: dedicated
value: ingress-nginx
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
– matchExpressions:
– key: “system/ingress”
operator: In
values:
– “true”
containers:
– name: default-http-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: “{{ image_registry.addr }}/{{ image.http_backend }}”
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
– containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
—
最后，有的人建议将initcontainer去掉，改为基于原有的nginx-ingress-controller镜像加一层layer，将配置路径权限的脚本放在该层执行。 个人认为这种方法既不美观，也不方便。唯一的好处仅在于deploy yaml仍然简洁（但少不了volumeMount之类的配置）。不过还是看个人使用感受吧~
到此这篇关于nginx-ingress-controller日志持久化方案的解决的文章就介绍到这了,更多相关nginx ingress controller日志持久化内容请搜索服务器之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持服务器之家！
原文链接：https://segmentfault.com/a/1190000022200374
声明： 猿站网有关资源均来自网络搜集与网友提供，任何涉及商业盈利目的的均不得使用，否则产生的一切后果将由您自己承担！ 本平台资源仅供个人学习交流、测试使用 所有内容请在下载后24小时内删除，制止非法恶意传播，不对任何下载或转载者造成的危害负任何法律责任！也请大家支持、购置正版！ 。本站一律禁止以任何方式发布或转载任何违法的相关信息访客发现请向站长举报，会员发帖仅代表会员个人观点，并不代表本站赞同其观点和对其真实性负责。本网站的资源部分来源于网络，如有侵权烦请发送邮件至：2697268773@qq.com进行处理。