Evaluators Guide: CyberArk Enterprise Issuer
cert-manager CyberArk Enterprise Issuer CyberArk Certificate Manager
Warning
This document is currently a working draft and assumes some knowledge of Kubernetes/OpenShift and there my some inaccuracies and errors. It is intended to provide Information Security and platform teams with a quick overview for integrating CyberArk's Workload Identity Issuer with 'cert-managerandIstio` service-mesh. It is NOT intended to be used for production purposes.
Please also not that the original Venafi product names are currently transitioning to CyberArk therefore the document uses both Venafi and CyberArk interchangeably.
Objective
- Install CyberArk's Enterprise Issuer in a Kubernetes or OpenShift cluster using
helm - Install CyberArk's Enterprise
cert-managerin a Kubernetes or OpenShift cluster usinghelm - Configure CyberArk Enterprise Issuer to connect to CyberArk's SaaS or Self Hosted control planes
- Configure
cert-managerto use CyberArk Enterprise Issuer for requesting certificates via CyberArk's SaaS or Self Hosted control planes - Create a new certificate request resource and observe it creating a new certificate request
- View the new certificate in the Kubernetes secret store
Overview
This evaluation guide is intended to to provide a single point of reference for installing and configuring CyberArk Enterprise Issuer (EI) with cert-manager. It provides the evaluator with a structured guided step by step list of tasks with explanations along the way.
It will include the following components:
- Enterprise Issuer for CyberArk Certificate Manager: Enterprise Issuer for CyberArk Certificate Manager (formerly as Venafi Enhanced Issuer) is a cert-manager issuer that can be either cluster-wide or per namespace. This component enables your clusters to issue certificates from Venafi Control Plane.
- Venafi Connection: Venafi Connection is used to configure the connection and authentication between Venafi Control Plane and your Kubernetes cluster.
- Enterprise cert-manager: Enterprise cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using certificates. Venafi provides Long-Term Support (LTS) releases of cert-manager to help organizations with lifecycle planning. Each LTS release is maintained for a minimum of two years.
The components are stored in either a public or private repository.
- Public Registries: These are deployed in all available Venafi Control Plane regions and serve from a single location: registry.venafi.cloud. Public registries hold publicly available artifacts and do not require authentication.
- Private Registries: These registries store enterprise Venafi software and are available in region-specific locations. US region - private-registry.venafi.cloud EU region - private-registry.venafi.eu Users in the US, Canada, Australia, and Singapore regions should use the US registry. Users in the EU and UK should use the EU registry. Access to these registries requires a subscription to the Venafi Control Plane. For your convenience, the private registries also contain the public artifacts hosted by the public registries.
There are two ways to acquire credentials to access the registry:
- Using the Venafi Control Plane UI
- Using the CLI tool for CyberArk Certificate Manager
Users in the US, Canada, Australia, and Singapore regions should use the US registry. Users in the EU and UK should use the EU registry:
- US: oci://private-registry.venafi.cloud/
- EU: oci://private-registry.venafi.eu/
Prerequisites
To complete this quick-start you will need to ensure that the following utilities are installed:
venctlA CLI tool that enables interaction with CyberArk Certificate Manager from the command line. You can install it from here: https://docs.venafi.cloud/vaas/venctl/t-venctl-install/imgpkgA CLI tool that allows users to store a set of arbitrary files as an OCI image. You can install it from here: https://github.com/carvel-dev/imgpkgjqA lightweight CLI tool and flexible command-line JSON processor. You can install it from here: https://jqlang.orgcmctlA command line tool that can help you manage cert-manager and its resources inside your cluster. You can install it from here: https://cert-manager.io/docs/reference/cmctl/#installation
Step 1. Enable access to the private CyberArk OCI registry
In this step we will enable access to the private CyberArk OCI (Open Container Initiative) registry which stores the enterprise container images, Helm charts, and other artifacts. To save some time we'll use the venctl command line to get the credential.
A note about CyberArk OCI registries
There are two types of registries: public and private.
- Public Registries: These are deployed in all available Venafi Control Plane regions and serve from a single location: registry.venafi.cloud. Public registries hold publicly available artifacts and do not require authentication.
-
Private Registries: These registries store enterprise Venafi software and are available in region-specific locations. US region - private-registry.venafi.cloud EU region - private-registry.venafi.eu Users in the US, Canada, Australia, and Singapore regions should use the US registry. Users in the EU and UK should use the EU registry. Access to these registries requires a subscription to the Venafi Control Plane. For your convenience, the private registries also contain the public artifacts hosted by the public registries. There are two ways to acquire credentials to access the registry:
-
Using the Venafi Control Plane UI
- Using the CLI tool for CyberArk Certificate Manager
This step also requires access to a system where jq and kubectl are installed.
Users in the US, Canada, Australia, and Singapore regions should use the US registry. Users in the EU and UK should use the EU registry:
- US: oci://private-registry.venafi.cloud/
- EU: oci://private-registry.venafi.eu/
a. Create an "image pull" credential
venctl iam service-accounts registry create --name "Enterprise Image Pull Secret" \ # (1)
--scopes cert-manager-components,enterprise-venafi-issuer,enterprise-approver-policy,openshift-routes \
--output dockerconfig \
--output-file venafi_registry_docker_config.json \ # (2)
--validity 365 \
--api-key {API_KEY} # (3)
- This is the display name for the credential listed in the SaaS control plane under "Service accounts"
- This is the file name that will be used to store the new credential"
- This is API key used to authenticate with the SaaS control plane under "Account, Preferences"
We now have three options for pulling images from the private repository. We can configure Kubernetes (option 1) to use the "image pull" credential so that it can directly pull images from the private repository. However, some orgnizations require helm charts and images to be pulled down and added "mirrored" (option 2) to their own private repositories. Alternatively, the charts and images can be downloaded (option 3) manually.
This option will create a new "image pull" secret in Kubernetes that will enable Kubernetes to pull the CyberArk Enterprise components directly from the private repo
a. Create a new secret from the credential
kubectl create namespace venafi
kubectl create secret docker-registry venafi-image-pull-secret --namespace venafi --from-file .dockerconfigjson=venafi_registry_docker_config.json
b. Inspect the secret
To set up Docker mirroring, follow the specific process for your mirroring tool, like Artifactory.
The username and password can be extracted from the "image pull" credential from above using the following command:
We'll use this option to download the helm charts and images for cert-manager and the Enterprise issuer using the imgpkg CLI utility.
A note about imgpkg
imgpkg is a tool that allows users to store a set of arbitrary files as an OCI image. One of the driving use cases is to store Kubernetes configuration (plain YAML, ytt templates, Helm templates, etc.) in OCI registry as an image.
You can install it from here: https://github.com/carvel-dev/imgpkg
a. Extract the username and password
cat venafi_registry_docker_config.json \
jq '.. | select(.username?) | "username: \(.username)\npassword: \(.auth)"' -r
b. Login to the private repo using the username/password extracted from the above command,
docker login private-registry.venafi.cloud \
--username $(cat venafi_registry_docker_config.json | jq '.. | select(.username?).username' -r) \
--password $(cat venafi_registry_docker_config.json | jq '.. | select(.username?).auth | @base64d' -r | cut -d: -f2)
c. Download the Enterprise Issuer image and helm chart
imgpkg copy --image private-registry.venafi.cloud/venafi-issuer/venafi-enhanced-issuer:v0.16.0 --to-tar venafi-enhanced-issuer-v0.16.0.tar
imgpkg copy --image private-registry.venafi.cloud/charts/venafi-enhanced-issuer:v0.16.0 --to-tar venafi-enhanced-issuer-helm-v0.16.0.tar
d. Download cert-manger images and helm chart
imgpkg copy --image private-registry.venafi.cloud/cert-manager/cert-manager-controller:v1.18.2 --to-tar cert-manager-controller-v1.18.2.tar
imgpkg copy --image private-registry.venafi.cloud/cert-manager/cert-manager-acmesolver:v1.18.2 --to-tar cert-manager-acmesolver-v1.18.2.tar
imgpkg copy --image private-registry.venafi.cloud/cert-manager/cert-manager-cainjector:v1.18.2 --to-tar cert-manager-cainjector-v1.18.2.tar
imgpkg copy --image private-registry.venafi.cloud/cert-manager/cert-manager-webhook:v1.18.2 --to-tar cert-manager-webhook-v1.18.2.tar
imgpkg copy --image private-registry.venafi.cloud/cert-manager/cert-manager-startupapicheck:v1.18.2 --to-tar cert-manager-startupapicheck-v1.18.2.tar
imgpkg copy --image private-registry.venafi.cloud/charts/cert-manager:v1.18.2 --to-tar cert-manager-helm-v1.18.2.tar
imgpkg copy --tar venafi-enhanced-issuer-v0.16.0.tar --to-repo myrepo/venafi-enhanced-issuer
imgpkg copy --tar venafi-enhanced-issuer-helm-v0.16.0.tar --to-repo myrepo/venafi-enhanced-issuer
imgpkg copy --tar cert-manager-controller-v1.18.2.tar --to-repo myrepo/cert-manager-controller
imgpkg copy --tar cert-manager-acmesolver-v1.18.2.tar --to-repo myrepo/cert-manager-acmesolver
imgpkg copy --tar cert-manager-cainjector-v1.18.2.tar --to-repo myrepo/cert-manager-cainjector
imgpkg copy --tar cert-manager-webhook-v1.18.2.tar --to-repo myrepo/cert-manager-webhook
imgpkg copy --tar cert-manager-startupapicheck-v1.18.2.tar --to-repo myrepo/cert-manager-startupapicheck
imgpkg copy --tar cert-manager-helm-v1.18.2.tar --to-repo myrepo/cert-manager
Step 2. Install and configure cert-manager
Use the following helm command to install cert-manager from the CyberArk OCI image registry.
helm upgrade cert-manager oci://registry.venafi.cloud/charts/cert-manager \ # (1)
--install \
--wait \
--create-namespace \
--namespace venafi \
--values cert-manager.values.yaml \ # (2)
--version v1.18.2
- Replace with your own private repo if required.
- Use the
cert-manager.values.yamlprovided below.
Use the following YAML to provide the values to helm.
global:
imagePullSecrets:
- name: venafi-image-pull-secret # (1)
crds:
enabled: true
image:
repository: private-registry.venafi.cloud/cert-manager/cert-manager-controller
acmesolver:
image:
repository: private-registry.venafi.cloud/cert-manager/cert-manager-acmesolver
webhook:
image:
repository: private-registry.venafi.cloud/cert-manager/cert-manager-webhook
cainjector:
image:
repository: private-registry.venafi.cloud/cert-manager/cert-manager-cainjector
startupapicheck:
image:
repository: private-registry.venafi.cloud/cert-manager/cert-manager-startupapicheck
- This is the name of the Kubernetes secret you created in step 1 above.
Step 3. Install Enterprise issuer
Use the following helm command to install enterprise issuer from the CyberArk OCI image registry.
helm upgrade venafi-enhanced-issuer oci://registry.venafi.cloud/charts/venafi-enhanced-issuer \
--install \
--wait \
--namespace venafi \
--values venafi-enhanced-issuer.values.yaml \ # (1)
--version v0.16.0
- Use the
venafi-enhanced-issuer.values.yamlprovided below.
global:
imagePullSecrets:
- name: venafi-image-pull-secret # (1)
venafiConnection:
include: true # set to `false` if Venafi Connection CRDs & RBAC are already installed
venafiEnhancedIssuer:
manager:
image:
repository: private-registry.venafi.cloud/venafi-issuer/venafi-enhanced-issuer
- This is the name of the Kubernetes secret you created in step 1 above.
Note
The enterprise issuer will automatically install the `Venafi Connection" component. Venafi Connection is used to configure the connection and authentication between Venafi Control Plane and your Kubernetes cluster.
Venafi Connection is a "namespaced" resource. Any service account or secret referenced by a Venafi Connection must be in the same namespace as the Venafi Connection resource or in a namespace that matches the allowReferencesFrom field, as described in cross namespace Venafi Connection references.
You can add the custom resources definitions for Venafi Connections to your Kubernetes API server when you install one of the projects that use the Venafi Connection resources as configuration.
Step 4. Venafi Connections
The custom resources definitions for VenafiIssuer and VenafiClusterIssuer are added to your Kubernetes API server when you install Enterprise Issuer.
Select one of the following:
VenafiClusterIssuer is a cluster-scoped resource and doesn't have a namespace. Any Venafi Connection custom resource referenced by a VenafiClusterIssuer MUST be in the venafi namespace, or which ever namespace you installed Enterprise Issuer in.
Todo
create a secret
#title="venafi-cloud-credentials secret"
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: venafi-cloud-credentials
namespace: venafi
stringData:
venafi-cloud-key: $(API_KEY)
EOF
create a SaaS connection
#title="Create VenafiConnection"
# Create a Venafi Connection that uses the above credential
kubectl apply -f - <<EOF
apiVersion: jetstack.io/v1alpha1
kind: VenafiConnection
metadata:
name: venafi-saas-connection
namespace: venafi
spec:
vaas:
url: https://api.venafi.cloud
apiKey:
- secret:
name: venafi-cloud-credentials
fields: ["venafi-cloud-key"]
EOF
Inspect the connection
Create a cluster role
# create role that allows creating sa tokens for 'sandbox'
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: read-creds-secret-role-for-venafi-connection
rules:
- apiGroups: [""]
resources:
- secrets
verbs:
- get
resourceNames: [ "venafi-cloud-credentials" ]
---
# link the controller's service account to the 'create-tokens-for-vault-sa' role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: read-creds-secret-role-for-venafi-connection
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: read-creds-secret-role-for-venafi-connection
subjects:
- kind: ServiceAccount
name: venafi-connection
namespace: venafi
EOF
VenafiIssuer is a "namespaced" resource. Any Venafi Connection custom resource referenced by a VenafiIssuer MUST be in the same namespace as the VenafiIssuer resource or MUST be in a namespace explicitly allowed in the Venafi Connection CR definition.
Todo
get an access token with VCert
vcert getcred -u "https://tpp.emea.venafidemo.com" --username carl --password Tdoxwg7HysJOACvT
create a TPP secret
#title="venafi-cloud-credentials secret"
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: venafi-tpp-credentials
namespace: venafi
stringData:
access-token: tuGwRpdwS580UvoTSOfORA==
EOF
create a TPP connection
kubectl apply -f - <<EOF
apiVersion: jetstack.io/v1alpha1
kind: VenafiConnection
metadata:
name: venafi-tpp-connection
namespace: venafi
spec:
tpp:
url: https://tpp.emea.venafidemo.com
accessToken:
- secret:
name: venafi-tpp-credentials
fields: ["access-token"]
EOF
Inspect the connection
Create a cluster role
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: read-creds-secret-role-for-venafi-tpp-connection
rules:
- apiGroups: [""]
resources:
- secrets
verbs:
- get
resourceNames: [ "venafi-tpp-credentials" ]
---
# link the controller's service account to the 'create-tokens-for-vault-sa' role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: read-creds-secret-role-for-venafi-tpp-connection
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: read-creds-secret-role-for-venafi-tpp-connection
subjects:
- kind: ServiceAccount
name: venafi-connection
namespace: venafi
EOF
Step 5. Creating a cluster issuers
Enterprise Issuer for CyberArk Certificate Manager (formally Enterprise Issuer) has two custom resources: VenafiIssuer and VenafiClusterIssuer.
The custom resources definitions for VenafiIssuer and VenafiClusterIssuer are added to your Kubernetes API server when you install Enterprise Issuer.
Select one of the following:
VenafiClusterIssuer is a cluster-scoped resource and doesn't have a namespace. Any Venafi Connection custom resource referenced by a VenafiClusterIssuer MUST be in the venafi namespace, or which ever namespace you installed Enterprise Issuer in.
The example below assumes that the following Venafi Connection resource exists:
VenafiIssuer is a "namespaced" resource. Any Venafi Connection custom resource referenced by a VenafiIssuer MUST be in the same namespace as the VenafiIssuer resource or MUST be in a namespace explicitly allowed in the Venafi Connection CR definition.
The example below assumes that the following Venafi Connection resource exists:
Step 6. Configure an RBAC for cert-manager auto-approve
If you are using Approver Policy or Enterprise Approver Policy for CyberArk Certificate Manager, no further action is required. If not, you must let cert-manager auto-approve the certificate requests that reference the VenafiClusterIssuer and VenafiIssuer types with the following RBAC
# Configure cert-manager approval
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cert-manager-controller-approve:venafi-enhanced-issuer
rules:
- apiGroups: ["cert-manager.io"]
resources: ["signers"]
verbs: ["approve"]
resourceNames: ["venafiissuers.jetstack.io/*", "venaficlusterissuers.jetstack.io/*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cert-manager-controller-approve:venafi-enhanced-issuer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cert-manager-controller-approve:venafi-enhanced-issuer
subjects:
- name: cert-manager
namespace: venafi
kind: ServiceAccount
EOF
Check the cluster issuer
Step 7. Creating a testing certificate resources
Create a new Certificate
kubectl apply -f - <<EOF
kind: Certificate
apiVersion: cert-manager.io/v1
metadata:
name: venafi-tpp-test-1
namespace: venafi
spec:
secretName: venafi-tpp-test-1
commonName: srvc2.acme.com
issuerRef:
name: venafi-tpp-cluster-issuer
kind: "VenafiClusterIssuer"
group: "jetstack.io"
privateKey:
rotationPolicy: Always
size: 2048
dnsNames:
- srvc3.acme.com
#uris:
#- spiffe://cluster.local/ns/sandbox/sa/srvc1
EOF
Step 8. Securing an NGINX Ingress
There are two primary ways to secure an NGINX ingress resource: using annotations on the ingress with ingress-shim or directly creating a certificate resource.
In this example, we will add annotations to the ingress, and take advantage of ingress-shim to have it create the certificate resource on our behalf.
Now lets secure an NGINX ingress. For this demo we'll use the example kuard application.
First we'll create a deployment
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: kuard
spec:
selector:
matchLabels:
app: kuard
replicas: 1
template:
metadata:
labels:
app: kuard
spec:
containers:
- image: gcr.io/kuar-demo/kuard-amd64:1
imagePullPolicy: Always
name: kuard
ports:
- containerPort: 8080
EOF
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: kuard
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
selector:
app: kuard
EOF
Finally lets create an ingress that uses the venafi-saas-cluster-issuer to get a certificate.
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kuard
annotations:
cert-manager.io/issuer: venafi-saas-cluster-issuer
cert-manager.io/issuer-kind: VenafiClusterIssuer
cert-manager.io/issuer-group: jetstack.io
cert-manager.io/common-name: example.example.com
spec:
ingressClassName: nginx
tls:
- hosts:
- example.example.com
secretName: example.example.com-tls
rules:
- host: example.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kuard
port:
number: 80
EOF
Inspect the certificate
Step 9. Access to out of tree issuers
This section covers configuring an "out-of-tree" VenafiIssuer that can be access from another namespace.
a. Create Environment Variables
The a only for testing the out-of-tree VenafiIssuer.
export CERT_SUFFIX="$(date +%S%H%M%d%m)"
export CERT_NAME=cert-${CERT_SUFFIX}.svc.cluster.local
export CERT_ZONE="cert-manager\\demo"
export CCM_APIKEY="REPLACE_WITH_API_KEY"
export CCM_NAMESPACE="venafi"
export CERT_NAMESPACE="foo"
b. Create a new VenafiConnection resource
# create venafi connection resource
kubectl apply -f - <<EOF
apiVersion: jetstack.io/v1alpha1
kind: VenafiConnection
metadata:
name: venafi-saas-connection-for-nginx
namespace: ${CCM_NAMESPACE}
spec:
allowReferencesFrom:
matchLabels:
kubernetes.io/metadata.name: ${CERT_NAMESPACE}
vaas:
url: https://api.venafi.cloud
apiKey:
- secret:
name: venafi-cloud-credentials
fields: ["venafi-cloud-key"]
EOF
c. Check the VenafiConnection
d. Create cluster role and binding
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: read-ccm-credentials
namespace: ${CCM_NAMESPACE}
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["venafi-cloud-credentials"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-ccm-credentials
namespace: ${CCM_NAMESPACE}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: read-ccm-credentials
subjects:
- kind: ServiceAccount
name: venafi-connection
namespace: ${CCM_NAMESPACE}
EOF
e. Create a test namespace
f. Create a VenafiIssuer in the test namespace
kubectl apply -f - <<EOF
apiVersion: jetstack.io/v1alpha1
kind: VenafiIssuer
metadata:
name: venafi-saas-issuer-for-nginx
namespace: ${CERT_NAMESPACE}
spec:
venafiConnectionName: venafi-saas-connection-for-nginx
venafiConnectionNamespace: ${CCM_NAMESPACE}
zone: ${CERT_ZONE}
EOF
g. Create a Certificate resource
This is a test certificate only used to test the out-of-tree VenafiConnection
kubectl apply -f - <<EOF
kind: Certificate
apiVersion: cert-manager.io/v1
metadata:
name: ${CERT_NAME}
namespace: ${CERT_NAMESPACE}
spec:
secretName: ${CERT_NAME}
commonName: ${CERT_NAME}
duration: 24h
renewBefore: 8h
issuerRef:
name: venafi-saas-issuer-for-nginx
kind: "VenafiIssuer"
group: "jetstack.io"
privateKey:
rotationPolicy: Always
size: 2048
dnsNames:
- ${CERT_NAME}
#uris:
#- spiffe://cluster.local/ns/sandbox/sa/srvc1
EOF
h. Validate the certificate is ready
Use the following command to check that the certificate reaches the ready state.
echo "Waiting up to 30s for Certificate ${CERT_NAME} in ns ${CERT_NAMESPACE} to be Ready..."
for i in {1..30}; do
READY=$(kubectl -n "${CERT_NAMESPACE}" get certificate "${CERT_NAME}" \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)
if [[ "${READY}" == "True" ]]; then
echo "#################################################"
echo "Certificate ${CERT_NAME} is Ready"
echo "#################################################"
exit 0
fi
sleep 1
done
echo "Certificate ${CERT_NAME} not Ready after 30s"
echo "Troubleshoot with:"
echo " kubectl -n ${CERT_NAMESPACE} describe certificate ${CERT_NAME}"
echo " kubectl -n ${CERT_NAMESPACE} describe certificaterequest -l cert-manager.io/certificate-name=${CERT_NAME}"
i. Update the VirtualServer config to include