网站建设中英语,青岛网站建设 上流,个人网站备案 淘宝客,在线一键扒站源码php文章目录 环境在namespace中安装Operator先决条件使用Web console从OperatorHub安装删除 使用CLI从OperatorHub安装从已安装的Operator创建应用使用Operator创建etcd集群报错从web console debug从命令行debug分析 参考 环境
RHEL 9.3Red Hat OpenShift Local 2.32
在namespa… 文章目录 环境在namespace中安装Operator先决条件使用Web console从OperatorHub安装删除 使用CLI从OperatorHub安装从已安装的Operator创建应用使用Operator创建etcd集群报错从web console debug从命令行debug分析 参考 环境
RHEL 9.3Red Hat OpenShift Local 2.32
在namespace中安装Operator
先决条件
打开web console
$ crc console
Opening the OpenShift Web Console in the default browser...会自动打开浏览访问web console 注如果不想自动打开浏览器则可以加上 --url 选项
$ crc console --url
https://console-openshift-console.apps-crc.testing然后手工复制URL并从浏览器里访问。
查看用户名密码
$ crc console --credentials
To login as a regular user, run oc login -u developer -p developer https://api.crc.testing:6443.
To login as an admin, run oc login -u kubeadmin -p cWwas-FvXBW-rTjsi-eECwX https://api.crc.testing:6443使用 developer 登录web console 切换到“Administrator”视角可见在“Operators”下只有“Installed Operators”当前没有安装Operator。
这是因为 developer 用户没有安装Operator的权限需要赋权才行。
简略起见就不赋权了。退出登录然后使用 kubeadmin 帐号登录 可见在“Operators”下多了“OperatorHub”子菜单并且“Installed Operators”里显示了“Package Server”。
使用Web console从OperatorHub安装
点击“OperatorHub”搜索“etcd” 点击搜索结果里的“etcd”弹出对话框如下 点击左上角“Install”按钮如下
Update channel默认为 singlenamespace-alphaVersion默认为 0.9.4Installation mode默认为 A specific namespace on the clusterInstalled namespace选择 my-etcd 这是自己创建的projectnamespace若还没有创建则创建一下Update approval默认为 Automatic 最后点击“Install”按钮安装Operator。
大约一两分钟就安装好了 回到“Installed Operators”可见在 my-etec project下出现了etcd Operator 点击可以查看详情 注在命令行可查看相应的CSV
$ oc get csv -n my-etcd
NAME DISPLAY VERSION REPLACES PHASE
etcdoperator.v0.9.4 etcd 0.9.4 etcdoperator.v0.9.2 Succeeded删除
点击右侧三个点选择“Uninstall Operator” 使用CLI从OperatorHub安装
查看当前身份
$ oc whoami
kubeadmin查看OperatorHub中集群可用的Operator列表
$ oc get packagemanifests -n openshift-marketplace
NAME CATALOG AGE
forklift-operator Community Operators 34d
debezium-operator Community Operators 34d
pcc-operator Certified Operators 34d
......
etcd Community Operators 34d
......检查所需Operator以验证其支持的安装模式和可用频道
$ oc describe packagemanifests etcd -n openshift-marketplace
Name: etcd
Namespace: openshift-marketplace
Labels: catalogcommunity-operatorscatalog-namespaceopenshift-marketplaceoperatorframework.io/arch.amd64supportedoperatorframework.io/os.linuxsupportedproviderCNCFprovider-url
Annotations: none
API Version: packages.operators.coreos.com/v1
Kind: PackageManifest
Metadata:Creation Timestamp: 2024-01-10T10:34:13Z
Spec:
Status:Catalog Source: community-operatorsCatalog Source Display Name: Community OperatorsCatalog Source Namespace: openshift-marketplaceCatalog Source Publisher: Red HatChannels:Current CSV: etcdoperator.v0.6.1Current CSV Desc:Annotations:Capabilities: Full LifecycleDescription: etcd is a distributed key value store providing a reliable way to store data across a cluster of machines.Tectonic - Visibility: ocsApiservicedefinitions:Customresourcedefinitions:Owned:Description: Represents a cluster of etcd nodes.Display Name: etcd ClusterKind: EtcdClusterName: etcdclusters.etcd.database.coreos.comVersion: v1beta2Description: etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines. It’s open-source and available on GitHub. etcd gracefully handles leader elections during network partitions and will tolerate machine failure, including the leader. Your applications can read and write data into etcd.
A simple use-case is to store database connection details or feature flags within etcd as key value pairs. These values can be watched, allowing your app to reconfigure itself when they change. Advanced uses take advantage of the consistency guarantees to implement database leader elections or do distributed locking across a cluster of workers._The etcd Open Cloud Service is Public Alpha. The goal before Beta is to fully implement backup features._### Reading and writing to etcdCommunicate with etcd though its command line utility etcdctl or with the API using the automatically generated Kubernetes Service.[Read the complete guide to using the etcd Open Cloud Service](https://coreos.com/tectonic/docs/latest/alm/etcd-ocs.html)### Supported Features
**High availability**
Multiple instances of etcd are networked together and secured. Individual failures or networking issues are transparently handled to keep your cluster up and running.
**Automated updates**
Rolling out a new etcd version works like all Kubernetes rolling updates. Simply declare the desired version, and the etcd service starts a safe rolling update to the new version automatically.
**Backups included**
Coming soon, the ability to schedule backups to happen on or off cluster.Display Name: etcdInstall Modes:Supported: trueType: OwnNamespaceSupported: trueType: SingleNamespaceSupported: falseType: MultiNamespaceSupported: trueType: AllNamespacesKeywords:etcdkey valuedatabasecoreosopen sourceLinks:Name: BlogURL: https://coreos.com/etcdName: DocumentationURL: https://coreos.com/operators/etcd/docs/latest/Name: etcd Operator Source CodeURL: https://github.com/coreos/etcd-operatorMaintainers:Email: supportcoreos.comName: CoreOS, IncMaturity: alphaProvider:Name: CoreOS, IncRelated Images:quay.io/coreos/etcd-operatorsha256:bd944a211eaf8f31da5e6d69e8541e7cada8f16a9f7a5a570b22478997819943Version: 0.6.1Entries:Name: etcdoperator.v0.6.1Version: 0.6.1Name: alphaCurrent CSV: etcdoperator.v0.9.4-clusterwideCurrent CSV Desc:Annotations:Alm - Examples: [{apiVersion: etcd.database.coreos.com/v1beta2,kind: EtcdCluster,metadata: {name: example,annotations: {etcd.database.coreos.com/scope: clusterwide}},spec: {size: 3,version: 3.2.13}},{apiVersion: etcd.database.coreos.com/v1beta2,kind: EtcdRestore,metadata: {name: example-etcd-cluster-restore},spec: {etcdCluster: {name: example-etcd-cluster},backupStorageType: S3,s3: {path: full-s3-path,awsSecret: aws-secret}}},{apiVersion: etcd.database.coreos.com/v1beta2,kind: EtcdBackup,metadata: {name: example-etcd-cluster-backup},spec: {etcdEndpoints: [etcd-cluster-endpoints],storageType:S3,s3: {path: full-s3-path,awsSecret: aws-secret}}}
]Capabilities: Full LifecycleCategories: DatabaseContainer Image: quay.io/coreos/etcd-operatorsha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9bCreated At: 2019-02-28 01:03:00Description: Create and maintain highly-available etcd clusters on KubernetesRepository: https://github.com/coreos/etcd-operatorTectonic - Visibility: ocsApiservicedefinitions:Customresourcedefinitions:Owned:Description: Represents a cluster of etcd nodes.Display Name: etcd ClusterKind: EtcdClusterName: etcdclusters.etcd.database.coreos.comVersion: v1beta2Description: Represents the intent to backup an etcd cluster.Display Name: etcd BackupKind: EtcdBackupName: etcdbackups.etcd.database.coreos.comVersion: v1beta2Description: Represents the intent to restore an etcd cluster from a backup.Display Name: etcd RestoreKind: EtcdRestoreName: etcdrestores.etcd.database.coreos.comVersion: v1beta2Description: The etcd Operater creates and maintains highly-available etcd clusters on Kubernetes, allowing engineers to easily deploy and manage etcd clusters for their applications.etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines. It’s open-source and available on GitHub. etcd gracefully handles leader elections during network partitions and will tolerate machine failure, including the leader.### Reading and writing to etcdCommunicate with etcd though its command line utility etcdctl via port forwarding:$ kubectl --namespace default port-forward service/example-client 2379:2379$ etcdctl --endpoints http://127.0.0.1:2379 get /Or directly to the API using the automatically generated Kubernetes Service:$ etcdctl --endpoints http://example-client.default.svc:2379 get /Be sure to secure your etcd cluster (see Common Configurations) before exposing it outside of the namespace or cluster.### Supported Features* **High availability** - Multiple instances of etcd are networked together and secured. Individual failures or networking issues are transparently handled to keep your cluster up and running.* **Automated updates** - Rolling out a new etcd version works like all Kubernetes rolling updates. Simply declare the desired version, and the etcd service starts a safe rolling update to the new version automatically.* **Backups included** - Create etcd backups and restore them through the etcd Operator.### Common Configurations* **Configure TLS** - Specify [static TLS certs](https://github.com/coreos/etcd-operator/blob/master/doc/user/cluster_tls.md) as Kubernetes secrets.* **Set Node Selector and Affinity** - [Spread your etcd Pods](https://github.com/coreos/etcd-operator/blob/master/doc/user/spec_examples.md#three-member-cluster-with-node-selector-and-anti-affinity-across-nodes) across Nodes and availability zones.* **Set Resource Limits** - [Set the Kubernetes limit and request](https://github.com/coreos/etcd-operator/blob/master/doc/user/spec_examples.md#three-member-cluster-with-resource-requirement) values for your etcd Pods.* **Customize Storage** - [Set a custom StorageClass](https://github.com/coreos/etcd-operator/blob/master/doc/user/spec_examples.md#custom-persistentvolumeclaim-definition) that you would like to use.Display Name: etcdInstall Modes:Supported: trueType: OwnNamespaceSupported: falseType: SingleNamespaceSupported: falseType: MultiNamespaceSupported: trueType: AllNamespacesKeywords:etcdkey valuedatabasecoreosopen sourceLinks:Name: BlogURL: https://coreos.com/etcdName: DocumentationURL: https://coreos.com/operators/etcd/docs/latest/Name: etcd Operator Source CodeURL: https://github.com/coreos/etcd-operatorMaintainers:Email: etcd-devgooglegroups.comName: etcd CommunityMaturity: alphaProvider:Name: CNCFRelated Images:quay.io/coreos/etcd-operatorsha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9bVersion: 0.9.4-clusterwideEntries:Name: etcdoperator.v0.9.4-clusterwideVersion: 0.9.4-clusterwideName: etcdoperator.v0.9.2-clusterwideVersion: 0.9.2-clusterwideName: etcdoperator.v0.9.0Version: 0.9.0Name: clusterwide-alphaCurrent CSV: etcdoperator.v0.9.4Current CSV Desc:Annotations:Alm - Examples: [{apiVersion: etcd.database.coreos.com/v1beta2,kind: EtcdCluster,metadata: {name: example},spec: {size: 3,version: 3.2.13}},{apiVersion: etcd.database.coreos.com/v1beta2,kind: EtcdRestore,metadata: {name: example-etcd-cluster-restore},spec: {etcdCluster: {name: example-etcd-cluster},backupStorageType: S3,s3: {path: full-s3-path,awsSecret: aws-secret}}},{apiVersion: etcd.database.coreos.com/v1beta2,kind: EtcdBackup,metadata: {name: example-etcd-cluster-backup},spec: {etcdEndpoints: [etcd-cluster-endpoints],storageType:S3,s3: {path: full-s3-path,awsSecret: aws-secret}}}
]Capabilities: Full LifecycleCategories: DatabaseContainer Image: quay.io/coreos/etcd-operatorsha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9bCreated At: 2019-02-28 01:03:00Description: Create and maintain highly-available etcd clusters on KubernetesRepository: https://github.com/coreos/etcd-operatorTectonic - Visibility: ocsApiservicedefinitions:Customresourcedefinitions:Owned:Description: Represents a cluster of etcd nodes.Display Name: etcd ClusterKind: EtcdClusterName: etcdclusters.etcd.database.coreos.comVersion: v1beta2Description: Represents the intent to backup an etcd cluster.Display Name: etcd BackupKind: EtcdBackupName: etcdbackups.etcd.database.coreos.comVersion: v1beta2Description: Represents the intent to restore an etcd cluster from a backup.Display Name: etcd RestoreKind: EtcdRestoreName: etcdrestores.etcd.database.coreos.comVersion: v1beta2Description: The etcd Operater creates and maintains highly-available etcd clusters on Kubernetes, allowing engineers to easily deploy and manage etcd clusters for their applications.etcd is a distributed key value store that provides a reliable way to store data across a cluster of machines. It’s open-source and available on GitHub. etcd gracefully handles leader elections during network partitions and will tolerate machine failure, including the leader.### Reading and writing to etcdCommunicate with etcd though its command line utility etcdctl via port forwarding:$ kubectl --namespace default port-forward service/example-client 2379:2379$ etcdctl --endpoints http://127.0.0.1:2379 get /Or directly to the API using the automatically generated Kubernetes Service:$ etcdctl --endpoints http://example-client.default.svc:2379 get /Be sure to secure your etcd cluster (see Common Configurations) before exposing it outside of the namespace or cluster.### Supported Features* **High availability** - Multiple instances of etcd are networked together and secured. Individual failures or networking issues are transparently handled to keep your cluster up and running.* **Automated updates** - Rolling out a new etcd version works like all Kubernetes rolling updates. Simply declare the desired version, and the etcd service starts a safe rolling update to the new version automatically.* **Backups included** - Create etcd backups and restore them through the etcd Operator.### Common Configurations* **Configure TLS** - Specify [static TLS certs](https://github.com/coreos/etcd-operator/blob/master/doc/user/cluster_tls.md) as Kubernetes secrets.* **Set Node Selector and Affinity** - [Spread your etcd Pods](https://github.com/coreos/etcd-operator/blob/master/doc/user/spec_examples.md#three-member-cluster-with-node-selector-and-anti-affinity-across-nodes) across Nodes and availability zones.* **Set Resource Limits** - [Set the Kubernetes limit and request](https://github.com/coreos/etcd-operator/blob/master/doc/user/spec_examples.md#three-member-cluster-with-resource-requirement) values for your etcd Pods.* **Customize Storage** - [Set a custom StorageClass](https://github.com/coreos/etcd-operator/blob/master/doc/user/spec_examples.md#custom-persistentvolumeclaim-definition) that you would like to use.Display Name: etcdInstall Modes:Supported: trueType: OwnNamespaceSupported: trueType: SingleNamespaceSupported: falseType: MultiNamespaceSupported: falseType: AllNamespacesKeywords:etcdkey valuedatabasecoreosopen sourceLinks:Name: BlogURL: https://coreos.com/etcdName: DocumentationURL: https://coreos.com/operators/etcd/docs/latest/Name: etcd Operator Source CodeURL: https://github.com/coreos/etcd-operatorMaintainers:Email: etcd-devgooglegroups.comName: etcd CommunityMaturity: alphaProvider:Name: CNCFRelated Images:quay.io/coreos/etcd-operatorsha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9bVersion: 0.9.4Entries:Name: etcdoperator.v0.9.4Version: 0.9.4Name: etcdoperator.v0.9.2Version: 0.9.2Name: etcdoperator.v0.9.0Version: 0.9.0Name: singlenamespace-alphaDefault Channel: singlenamespace-alphaPackage Name: etcdProvider:Name: CNCF
Events: none由 OperatorGroup 对象定义的Operator组选择目标namespace在其中为同一namespace中的所有Operator生成所需的RBAC访问权限。
订阅Operator的namespace必须具有与Operator的安装模式 AllNamespaces 或 SingleNamespace 模式相匹配的Operator组。如果要使用 AllNamespaces 安装Operator则 openshift-operators namespace已有适当的Operator组。
不过如果采用 SingleNamespace 模式而还没有适当的Operator组则必须创建一个。
注意前面在使用web console时选择 SingleNamespace 模式则在后台自动创建了 OperatorGroup 和 Subscription 对象。
创建 OperatorGroup 对象YAML文件比如 operatorgroup.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:name: operatorgroup_namenamespace: namespace
spec:targetNamespaces:- namespaceoperatorgroup_name my-operatorgroupnamespace my-etcd
注意OLM为每个Operator组创建以下集群角色
operatorgroup_name-adminoperatorgroup_name-editoperatorgroup_name-view
当手动创建Operator组时必须指定一个唯一名称该名称不能和现有集群角色或其它Operator组冲突。
创建 OperatorGroup 对象
oc apply -f operatorgroup.yaml创建一个 Subscription 对象YAML文件为Operator订阅一个namespace比如 sub.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:name: subscription_name# namespace: openshift-operators # 1namespace: my-etcd # 1
spec:channel: channel_name # 2name: operator_name # 3source: redhat-operators # 4sourceNamespace: openshift-marketplace # 5config:env: # 6- name: ARGSvalue: -v10envFrom: # 7- secretRef:name: license-secretvolumes: # 8- name: volume_nameconfigMap:name: configmap_namevolumeMounts: # 9- mountPath: directory_namename: volume_nametolerations: # 10- operator: Existsresources: # 11requests:memory: 64Micpu: 250mlimits:memory: 128Micpu: 500mnodeSelector: # 12foo: bar对于默认的 AllNamespaces 安装模式用法需指定 openshift-operators namespace。也可以指定一个自定义的全局namespace如果创建了。否则需为 SingleNamespace 安装模式使用指定关联的单个namespace。
本例中指定了 my-etcd namespace。
subscription_name etcdchannel_name singlenamespace-alphaoperator_name etcdvolume_name my-volumeconfigmap_name my-configmapdirectory_name my-directory
如果集群为STS模式在 Subscription 对象中包含以下字段
kind: Subscription
# ...
spec:installPlanApproval: Manual # 1config:env:- name: ROLEARNvalue: role_arn # 2创建 Subscription 对象
oc apply -f sub.yaml此时OLM已能感知到所选的Operator。Operator的CSV应该已经出现在目标namespace中由Operator所提供的API应该已经可用于创建。
注文档说的不是很清楚本例应该用不到ConfigMap、PVC、PV这可能只是一个模板。我没有实际实验。
从已安装的Operator创建应用
使用Operator创建etcd集群
前面安装好etcd Operator后在web console上点击etcd查看详情。
在 Provided APIs 下可见该Operator提供了三类新资源
etcd Clusteretcd Backupetcd Restore
这些对象的工作方式与内建的原生Kubernetes对象比如 Deployment 或 ReplicaSet 相似但包含管理etcd所特有的逻辑。 点击“etcd Cluster”下方的“Create instance”如下 点击左下角“Create”按钮如下 报错
点击“EC example”EC表示EtcdCluster然后查看Resources页签 可以看到pod处于pending状态。
注上图是一个clusterwide的Operator因为我重新安装过Operater不过问题都是一样的。
从web console debug
点击“example-bcqztbd6l6”pod然后查看“Logs”页签 可以看到“etcd”容器没有任何log因为它处于“waiting”状态。切换到“check-dns”容器可以看到其log 可以看到在不断的产生重复的log
......
nslookup: cant resolve example-bcqztbd6l6.example.my-etcd.svc
Server: 10.217.4.10
Address 1: 10.217.4.10 dns-default.openshift-dns.svc.cluster.local
......从命令行debug
在命令行查看pod
$ oc describe pod example-bcqztbd6l6 -n my-etcd
Name: example-bcqztbd6l6
Namespace: my-etcd
......
Init Containers:check-dns:Container ID: cri-o://8e4f03cfea06f682d877e6122ebd84f4b6f8ae75f87ba0fd3ebae1fabd36ebbeImage: busybox:1.28.0-glibcImage ID: docker.io/library/busyboxsha256:0b55a30394294ab23b9afd58fab94e61a923f5834fba7ddbae7f8e0c11ba85e6Port: noneHost Port: noneCommand:/bin/sh-cTIMEOUT_READY0while ( ! nslookup example-bcqztbd6l6.example.my-etcd.svc )do# If TIMEOUT_READY is 0 we should never time out and exit TIMEOUT_READY$(( TIMEOUT_READY-1 ))if [ $TIMEOUT_READY -eq 0 ];thenecho Timed out waiting for DNS entryexit 1fisleep 1doneState: RunningStarted: Wed, 14 Feb 2024 17:49:40 0800Ready: FalseRestart Count: 0Environment: noneMounts: none
Containers:etcd:Container ID: Image: quay.io/coreos/etcd:v3.2.13Image ID: Ports: 2380/TCP, 2379/TCPHost Ports: 0/TCP, 0/TCPCommand:/usr/local/bin/etcd--data-dir/var/etcd/data--nameexample-bcqztbd6l6--initial-advertise-peer-urlshttp://example-bcqztbd6l6.example.my-etcd.svc:2380--listen-peer-urlshttp://0.0.0.0:2380--listen-client-urlshttp://0.0.0.0:2379--advertise-client-urlshttp://example-bcqztbd6l6.example.my-etcd.svc:2379--initial-clusterexample-bcqztbd6l6http://example-bcqztbd6l6.example.my-etcd.svc:2380--initial-cluster-statenew--initial-cluster-tokena8f3fa08-f114-4c12-95b7-60e14eea400cState: WaitingReason: PodInitializingReady: FalseRestart Count: 0Liveness: exec [/bin/sh -ec ETCDCTL_API3 etcdctl endpoint status] delay10s timeout10s period60s #success1 #failure3Readiness: exec [/bin/sh -ec ETCDCTL_API3 etcdctl endpoint status] delay1s timeout5s period5s #success1 #failure3Environment: noneMounts:/var/etcd from etcd-data (rw)
Conditions:Type StatusInitialized False Ready False ContainersReady False PodScheduled True
......可见在 Init Containers: 处其 Command 是一段脚本包含了一个死循环只要 nslookup example-bcqztbd6l6.example.my-etcd.svc 有问题返回值不为0就会一直循环下去。
在命令行查看log
$ oc logs example-bcqztbd6l6 -n my-etcd
Defaulted container etcd out of: etcd, check-dns (init)
Error from server (BadRequest): container etcd in pod example-bcqztbd6l6 is waiting to start: PodInitializing说明pod里的 etcd 容器在等待pod初始化。
查看 etcd 容器的log
$ oc logs example-bcqztbd6l6 -c etcd -n my-etcd
Error from server (BadRequest): container etcd in pod example-bcqztbd6l6 is waiting to start: PodInitializing只是说在等待pod初始化。从以上log看不出来pod卡在那里了只能通过 describe pod 看到有段 Init Containers 逻辑得知 check-dns 容器然后查看其log
$ oc logs example-bcqztbd6l6 -c check-dns -n my-etcd
......
nslookup: cant resolve example-bcqztbd6l6.example.my-etcd.svc
Server: 10.217.4.10
Address 1: 10.217.4.10 dns-default.openshift-dns.svc.cluster.local
......可见和从web console看到的log是一致的。
分析
为什么会报这个错误呢我在网上百度了一下貌似这是 busybox 的一个bug。
参见 https://stackoverflow.com/questions/52109039/nslookup-cant-resolve-kubernetes-default 里面提到 You have encountered a bug in the latest versions of the busybox docker image. Use the tag busybox:1.28 instead of latest. 原issue https://github.com/docker-library/busybox/issues/48
不过也可能是因为我使用的是Red Hat OpenShift Local。没有深究。
参考
https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/operators/index#user-tasks
