网站建设 代表联系群众,石材石料网站搭建教程,设计公司起名字寓意好的字,网站建设要做什么会计科目写在前面 生产环境中的 k8s 集群安全不可忽略#xff0c;即使是内网环境容器化的应用部署虽然本质上没有变化#xff0c;始终是机器上的一个进程但是提高了安全问题的处理的复杂性分享一个开源的 k8s 集群安全合规检查/漏洞扫描 工具 kubescape博文内容涉及#xff1a; kube…写在前面 生产环境中的 k8s 集群安全不可忽略即使是内网环境容器化的应用部署虽然本质上没有变化始终是机器上的一个进程但是提高了安全问题的处理的复杂性分享一个开源的 k8s 集群安全合规检查/漏洞扫描 工具 kubescape博文内容涉及 kubescape 简介介绍kubescape 命令行工具安装扫描运行的集群kubescape 在集群下安装通过 kubescape Clound 可视化查看扫描信息 理解不足小伙伴帮忙指正需要有科学上网环境 对每个人而言真正的职责只有一个找到自我。然后在心中坚守其一生全心全意永不停息。所有其它的路都是不完整的是人的逃避方式是对大众理想的懦弱回归是随波逐流是对内心的恐惧 ——赫尔曼·黑塞《德米安》 简单介绍
k8s 安全问题不可忽略
镜像的不可变性使我们能够方便地部署、测试和发布镜像到其他环境这是一个很大的优势但也带来了潜在的风险镜像及其依赖在过时或者被弃用时无法自动更新或打新的安全补丁尤其一些基础镜像重新做镜像需要依赖一些厂商或者开源项目 team 来维护。
RedHat 在 2022 年 Kubernetes 安全报告中对 300 多名 DevOps、工程和安全专业人士进行了调查发现对 容器安全威胁的担忧以及对容器安全投资的缺乏 是 31% 的受访者对容器策略最普遍的担忧。
支持这些担忧的是 93% 的受访者在过去 12 个月内在其 Kubernetes 环境中经历过至少一次安全事件这些事件有时会导致收入或客户流失。超过一半的受访者 (55%) 在过去一年中还因为安全问题而不得不推迟应用程序的推出。
尽管媒体广泛关注网络攻击但该报告强调实际上是错误配置让 IT 专业人员彻夜难眠。Kubernetes 是高度可定制的具有可以影响应用程序安全状况的各种配置选项。因此受访者最担心的是容器和 Kubernetes 环境中的错误配置导致的风险暴露 (46%) —— 几乎是对攻击的担忧程度 (16%) 的三倍。尽可能自动化配置管理有助于缓解这些问题因此安全工具 —— 而不是人类 —— 提供了帮助开发人员和 DevOps 团队更安全地配置容器和 Kubernetes 的护栏。
以上内容来自Redhat Blog(The State of Kubernetes Security in 2022)
Kubescape 简单介绍
Kubescape 是一个开源的 Kubernetes 安全平台。它的功能包括 风险分析、安全合规性 和 错误配置扫描。针对 DevSecOps 从业者或平台工程师提供易于使用的 CLI 界面、灵活的输出格式和自动扫描功能。同时对于小集群提供了免费的 在线 面板工具它为 Kubernetes 用户和管理员节省了宝贵的时间、精力和资源。
Kubescape 可以扫描运行的集群、静态 YAML 文件和 本地 Helm Charts。它根据多个框架包括 NSA-CISA、MITRE ATTCK®和CIS Benchmark检测错误配置。
Kubescape 由 ARMO 创建是一个 Cloud Native Computing Foundation (CNCF) 沙盒项目。 Kubescape 如果小伙伴觉得重的话 kubectl 有一个类似的插件个人十分推荐叫做 kube-score ,很轻量也可以做一些简单的合规性扫描。但是只有扫描合规性的提示没有规范出处。
Kubescape 安装
当前的集群信息
┌──[rootvms100.liruilongs.github.io]-[~]
└─$kubectl get nodes
NAME STATUS ROLES AGE VERSION
vms100.liruilongs.github.io Ready control-plane 8d v1.25.1
vms101.liruilongs.github.io Ready control-plane 8d v1.25.1
vms102.liruilongs.github.io Ready control-plane 8d v1.25.1
vms103.liruilongs.github.io Ready none 8d v1.25.1
vms105.liruilongs.github.io Ready none 8d v1.25.1
vms106.liruilongs.github.io Ready none 8d v1.25.1
vms107.liruilongs.github.io Ready none 8d v1.25.1
vms108.liruilongs.github.io Ready none 8d v1.25.1
┌──[rootvms100.liruilongs.github.io]-[~]
└─$集群节点信息
┌──[rootvms100.liruilongs.github.io]-[~]
└─$hostnamectlStatic hostname: vms100.liruilongs.github.ioIcon name: computer-vmChassis: vmMachine ID: e93ae3f6cb354f3ba509eeb73568087eBoot ID: a1150b6d97dc4afbb81dae58f131a487Virtualization: vmwareOperating System: CentOS Linux 7 (Core)CPE OS Name: cpe:/o:centos:centos:7Kernel: Linux 5.4.230-1.el7.elrepo.x86_64Architecture: x86-64
┌──[rootvms100.liruilongs.github.io]-[~]
└─$Kubescape 命令行工具安装
Kubescape CLI 安装
通过下面的方式自动安装
curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash如果没有科学上网找一台可以访问的集群下载 install.sh 文件按照下面方式修改获取 curl 命令自行下载。
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$vim install.sh
.......
# curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUTecho curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT
exit 1
...运行 shell获取下载命令找有网的机器下载
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$sh install.sh
Installing Kubescape...curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape上传到指定位置
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$ls
install.sh kubescape-ubuntu-latest
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$mv kubescape-ubuntu-latest /root/.kubescape/kubescape
mv是否覆盖/root/.kubescape/kubescape y
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$修改脚本再次运行。
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$vim install.sh
# exit 1
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$sh install.sh
Installing Kubescape...curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescapeFinished Installation.Your current version is: v2.0.183 [git enabled in build: true]Usage: $ kubescape scan --enable-host-scan到这里 kubescape 命令行工具即安装成功扫描当前运行的集群可以运行如下命令。
┌──[rootvms100.liruilongs.github.io]-[~/ansible]
└─$kubescape scan --enable-host-scan --format html --output results.html --verbose
[info] Kubescape scanner starting
[info] Installing host scanner
[info] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[info] Accessing Kubernetes objects
[success] Accessed to Kubernetes objects
[info] Requesting images vulnerabilities results
[success] Requested images vulnerabilities results
[info] Requesting Host scanner data
[info] Host scanner version : v1.0.39
◑[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-xshr6; error: the server could not find the requested resource (get pods http:host-scanner-xshr6:7888)
◒[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-4tgnp; error: the server could not find the requested resource (get pods http:host-scanner-4tgnp:7888)
.......
[success] Done scanning. Cluster: kubernetes-admin-kubernetes^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^################################################################################
ApiVersion: hostdata.kubescape.cloud/v1beta0
Kind: KubeletInfo
Name: vms105.liruilongs.github.ioControls: 21 (Failed: 6, Excluded: 0)------------------------------------------------------------------------------------------------------
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
------------------------------------------------------------------------------------------------------
| High | CIS-4.1.7 Ensure that the | https://hub.armosec.io/docs/c-0168 | |
| | certificate authorities file | | |
| | permissions are set to 600 or | | |
| | more restrictive | | |--------------------------------------------------------------------------------------------
| | CIS-4.1.9 If the kubelet | https://hub.armosec.io/docs/c-0170 | |
| | config.yaml configuration | | |
| | file is being used validate | | |
| | permissions set to 600 or more | | |
| | restrictive | | |
------------------------------------------------------------------------------------------------------
........
------------------------------------------------------------------------------------------------------
| Low | CIS-4.2.6 Ensure that the | https://hub.armosec.io/docs/c-0177 | protectKernelDefaults |
| | --protect-kernel-defaults | | |
| | argument is set to true | | |--------------------------------------------------------------------------------------------
| | CIS-4.2.7 Ensure that the | https://hub.armosec.io/docs/c-0178 | makeIPTablesUtilChains |
| | --make-iptables-util-chains | | |
| | argument is set to true | | |
------------------------------------------------------------------------------------------------------################################################################################
ApiVersion: v1
Kind: Namespace
Name: kubescapeControls: 18 (Failed: 17, Excluded: 0)----------------------------------------------------------------------------------------------------------------------------------------------
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
----------------------------------------------------------------------------------------------------------------------------------------------
| High | CIS-5.2.11 Minimize the | https://hub.armosec.io/docs/c-0202 | metadata.labels[pod-security.kubernetes.io/enforce]baseline |
| | admission of Windows | | |
| | HostProcess Containers | | |--------------------------------------------------------------------
| | CIS-5.2.2 Minimize the | https://hub.armosec.io/docs/c-0193 | |
| | admission of privileged | | |
| | containers | | |
----------------------------------------------------------------------------------------------------------------------------------------------
| Medium | CIS-5.2.1 Ensure that | https://hub.armosec.io/docs/c-0192 | metadata.labels[pod-security.kubernetes.io/enforce]YOUR_VALUE |
| | the cluster has at least | | |
| | one active policy control | | |
| | mechanism in place | | |------------------------------------------------------------------------------------------------------------------------------------
.......
..........
################################################################################
ApiVersion: hostdata.kubescape.cloud/v1beta0
Kind: ControlPlaneInfo
Name: vms102.liruilongs.github.ioControls: 25 (Failed: 1, Excluded: 0)-----------------------------------------------------------------------------------------------------
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
-----------------------------------------------------------------------------------------------------
| High | CIS-1.1.20 Ensure that the | https://hub.armosec.io/docs/c-0111 | |
| | Kubernetes PKI certificate | | |
| | file permissions are set to | | |
| | 600 or more restrictive | | |
-----------------------------------------------------------------------------------------------------################################################################################
ApiVersion: apps/v1
Kind: Deployment
Name: local-path-provisioner
Namespace: local-path-storageControls: 35 (Failed: 18, Excluded: 0)
...
........
......输出的合规信息和漏洞信息。 异常问题
我在 kubescape 多次次扫描中集群因部分节点因为端口问题无法发生调度节点上的 kubescape 对于的 pod 无法自行删除如果上一次扫描的 pod 或则 ns 没有删除那么下一次的扫描无法进行之前创建的 pod 和 ns 状态一直为 Terminating, 解决办法需要 对命名空间进行彻底删除。
这里执行完命令会进入阻塞状态。
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubevirt]
└─$kubectl delete ns kubescape-host-scanner
namespace kubescape-host-scanner deleted运行脚本删除
┌──[rootvms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript]
└─$cat delete_namespace.sh
#!/bin/bashcoproc kubectl proxy --port30990 if [ $# -eq 0 ] ; thenecho 后面加上你所要删除的ns.exit 1
fikubectl get namespace $1 -o json logging.json
sed -i /finalizers/{n;d} logging.json
curl -k -H Content-Type: application/json -X PUT --data-binary logging.json http://127.0.0.1:30990/api/v1/namespaces/${1}/finalizekill %1
┌──[rootvms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript]
└─$sh delete_namespace.sh kubescape-host-scanner也可以离线运行Kubescape时间网络原因这里不做分享有需要的小伙伴可以到 github 上的项目地址查看详细信息。
一些其他的用法
扫描正在运行的 Kubernetes 集群
kubescape scan --enable-host-scan --verbose使用替代的 kubeconfig 文件
kubescape scan --kubeconfig cluster.conf扫描特定的命名空间
kubescape scan --include-namespaces development,staging,production排除某些命名空间
kubescape scan --exclude-namespaces kube-system,kube-public部署前扫描本地 YAML/JSON 文件
kubescape scan *.yaml┌──[rootvms100.liruilongs.github.io]-[~/ansible/helm]
└─$kubescape scan --enable-host-scan kube-prometheus-stack.yaml --format html --output resout.html
[info] Kubescape scanner starting
[warning] in setContextMetadata. case: git-url; error: repository host gitee.com not supported
[info] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[info] Accessing local objects
[success] Done accessing local objects
[info] Scanning GitLocal
[success] Done scanning GitLocal^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Controls: 55 (Failed: 31, Excluded: 0, Skipped: 0)
Failed Resources by Severity: Critical — 0, High — 34, Medium — 151, Low — 24-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| SEVERITY | CONTROL NAME | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| High | Resources memory limit and request | 7 | 0 | 7 | 100% |
| High | Resource limits | 7 | 0 | 7 | 100% |
| High | List Kubernetes secrets | 4 | 0 | 7 | 57% |
。。。。。。。。。。。。。。。。。。。。。
| Medium | Allow privilege escalation | 6 | 0 | 7 | 86% |
| Medium | Ingress and Egress blocked | 7 | 0 | 7 | 100% |
。。。。。。。。。。。。。。。。。。。。。
| Medium | CIS-5.4.1 Prefer using secrets as files over secrets as environment variables | 1 | 0 | 7 | 14% |
| Medium | CIS-5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions | 7 | 0 | 7 | 100% |
| Medium | CIS-5.7.4 The default namespace should not be used | 56 | 0 | 61 | 92% |
| Low | Immutable container filesystem | 6 | 0 | 7 | 86% |
| Low | Configured readiness probe | 5 | 0 | 7 | 71% |
| Low | Malicious admission controller (validating) | 1 | 0 | 1 | 100% |
| Low | Pods in default namespace | 7 | 0 | 7 | 100% |
| Low | Naked PODs | 1 | 0 | 1 | 100% |
| Low | Label usage for resources | 3 | 0 | 7 | 43% |
| Low | K8s common labels usage | 1 | 0 | 7 | 14% |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| | RESOURCE SUMMARY | 66 | 0 | 72 | 43.26% |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
FRAMEWORKS: ArmoBest (risk: 33.51), cis-v1.23-t1.0.1 (risk: 62.75), DevOpsBest (risk: 61.72), AllControls (risk: 36.46), MITRE (risk: 11.79), NSA (risk: 33.74)[success] Scan results saved. filename: resout.html~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scan results have not been submitted: run kubescape with the --account flag
For more details: https://hub.armosec.io/docs/installing-kubescape?utm_campaignSubmitutm_mediumCLIutm_sourceGitHub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Run with --verbose/-v flag for detailed resources view┌──[rootvms100.liruilongs.github.io]-[~/ansible/helm]
└─$l从 Git 存储库扫描 Kubernetes 清单文件
kubescape scan https://github.com/kubescape/kubescape扫描 Helm 图表,kubescape 将加载默认的 VALUES 文件。
kubescape scan /path/to/directory扫描 Kustomize 目录,Kubescape 将使用 kustomize 文件生成 Kubernetes YAML 对象并扫描它们以确保安全。
kubescape scan /path/to/directory使用 NSA 框架扫描正在运行的 Kubernetes 集群
kubescape scan framework nsa使用 MITRE ATTCK® 框架扫描正在运行的 Kubernetes 集群
kubescape scan framework mitre使用控件名称或控件 ID 扫描特定控件。请参阅控件列表。
kubescape scan control Privileged container指定报告输出格式
JSON
kubescape scan --format json --format-version v2 --output results.jsonXML
kubescape scan --format junit --output results.xmlPDF:
kubescape scan --format pdf --output results.pdf普罗米修斯指标
kubescape scan --format prometheusHTML
kubescape scan --format html --output results.html显示所有扫描到的资源包括通过的资源
kubescape scan --verboseKubescape 在集群下安装
Kubescape 也可以在集群下安装通过集群安装可以在 cloud 里的 UI 界面查看具体扫描信息集群中安装 Kubescape先决条件
确保您拥有 Kubescape Cloud 帐户——如果没有请在此处注册您需要拥有对集群的安装权限您应该能够创建 Deployments、CronJobs、ConfigMaps 和 Secrets你必须有 Kubectl 和 Helm
集群要求
Kubescape 运算符组件至少需要 400Mib RAM 和 400m CPU
具体的安装可以参考下的教程 需要注册 Kubescape Cloud并且 只有的工作节点小于 10 个的时候才免费。
https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
注册登录 这里登录完会弹出一个安装部署安装安装部署安装即可
添加 Helm 源并更新
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm repo add kubescape https://kubescape.github.io/helm-charts/
kubescape has been added to your repositories
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the rancher-stable chart repository
...Successfully got an update from the botkube chart repository
...Successfully got an update from the awx-operator chart repository
...Successfully got an update from the kubescape chart repository
Update Complete. ⎈Happy Helming!⎈运行 charts
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm upgrade --install kubescape kubescape/kubescape-cloud-operator -n kubescape --create-namespace --set account97f09924-0c06-42e4-bdad-5b333321af77 --set clusterNamekubectl config current-context
Release kubescape does not exist. Installing it now.
.........
......................
NAME: kubescape
LAST DEPLOYED: Sat Feb 4 10:03:36 2023
NAMESPACE: kubescape
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing kubescape-cloud-operator version 1.9.5.In a few minutes your scan results will be available in the following link:
https://cloud.armosec.io/config-scanning/kubernetes-admin-kubernetesYou can see and change the values of yours recurring configurations daily scan in the following link:
https://cloud.armosec.io/settings/assets/clusters/scheduled-scans?clusterkubernetes-admin-kuberneteskubectl -n kubescape get cj kubescape-scheduler -ojsonpath{.metadata.name}{\t}{.spec.schedule}{\n}You can see and change the values of yours recurring images daily scan in the following link:
https://cloud.armosec.io/settings/assets/imageskubectl -n kubescape get cj kubevuln-scheduler -ojsonpath{.metadata.name}{\t}{.spec.schedule}{\n}See you!!!
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$运行完成安装提示操作点击刚才注册生成的页面。 验证 Kubescape 在集群中运行状态
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$kubectl -n kubescape get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
gateway 1/1 1 1 19m
kubescape 1/1 1 1 19m
kubevuln 1/1 1 1 19m
operator 1/1 1 1 19mkubescape 会定期扫描
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$kubectl get cronjobs.batch -n kubescape
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
kubescape-scheduler 21 3 * * * False 0 none 20m
kubevuln-scheduler 13 22 * * * False 0 none 20m
┌──[rootvms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$在登录的主页中可以看到集群和第一次扫描结果 安全合规性扫描信息 可以通过不同的维度来看
合规性的维度查看
集群 Pod/Deploy 的维度查看
具体的 合规配置信息查看 漏洞扫描 关于 kubescape 和小伙伴分享到这里时间关系没有深入太多。偶尔听大佬谈到所以研究一下有需要的小伙伴快去尝试吧。
博文部分内容参考
文中涉及参考链接内容版权归原作者所有如有侵权请告知 https://github.com/kubescape/kubescape
https://www.redhat.com/en/blog/state-kubernetes-security-2022-1
https://betterprogramming.pub/image-vulnerability-scanning-for-optimal-kubernetes-security-c3ba933190ef
https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster © 2018-2023 liruilongergmail.comAll rights reserved. 保持署名-非商用-自由转载-相同方式共享(创意共享 3.0 许可证)
