[URGENT] Libvirt Certificates Expired

[fnavidan]

Hi Guys

I'm running CS 4.20.2.0 in a single node where management and agent is running on the same machine. Libvirt certificates have been expired and I am not sure how to properly regenerate them manually as I cannot provision host keys neither using UK nor using CMK. Following certs have expiresd:

/etc/pki/libvirt/servercert.pem
/etc/pki/libvirt/clientcert.pem
/etc/cloudstack/agent/cloud.crt

As libvirt and agent are both failed, management is unable to provision host keys.

I have also tried following cmk command:

cmk provision certificate hostid=97cd673c-a651-4580-8a2c-f302265906b6 reconnect=true forced=true

Also note that no explicit ca plugin is specified in management or agent properties

Please guide me through proper way of host keys and agent cert recovery so that trust chain won't break.

[fnavidan]

I was able to recover the KVM host successfully using following steps.

1. Change default certificate validity period to 10 years:
   Global Settings -> ca.framework.cert.validity.period: Set to 3650 (~10 years)

2. Generate CSR:
   On KVM host run:

    /usr/share/cloudstack-common/scripts/util/keystore-setup \
    /etc/cloudstack/agent/agent.properties \
    /etc/cloudstack/agent/cloud.jks \
    <KEYSTORE_PASSWORD> \
    3650 \
    /etc/cloudstack/agent/cloud.csr
   

3. Issue certificate using cmk

   • On management server run:

     cmk issue certificate
     domain=
     csr="$(cat /etc/cloudstack/agent/cloud.csr)"
     duration=3650

   • Save "certificates" value returned in json format into /etc/cloudstack/agent/cloud.crt on KVM host.

   • Replace literal "\n" in json value with new line using:
     printf "%b" "$(cat /etc/cloudstack/agent/cloud.crt)" > /etc/cloudstack/agent/cloud.crt

4. Import certificate to keystore.
   On KVM host run:

   /usr/share/cloudstack-common/scripts/util/keystore-cert-import
   /etc/cloudstack/agent/agent.properties
   "<KEYSTORE_PASSWORD>"
   /etc/cloudstack/agent/cloud.jks
   ssh
   /etc/cloudstack/agent/cloud.crt
   "$(cat /etc/cloudstack/agent/cloud.crt | sed ':a;N;$!ba;s/\n/^/g;s/ //g')"
   /etc/cloudstack/agent/cloud.ca.crt
   "$(cat /etc/cloudstack/agent/cloud.ca.crt | sed ':a;N;$!ba;s/\n/^/g;s/ //g')"
   /etc/cloudstack/agent/cloud.key
   ""

5. Restart libvirtd and cloudstack-agent

   systemctl restart libvirtd
   systemctl restart cloudstack-agent

[herrickaccuwebhosting]

Great writeup, this will save people a ton of time! One thing worth adding for anyone else hitting this: expired libvirt/agent certs are usually a symptom of the cert validity period being left at the CloudStack default (90 days in older setups), so bumping ca.framework.cert.validity.period like you did is the right long term fix, but it's also worth setting a calendar reminder or monitoring check on cert expiry dates going forward since KVM hosts going dark from expired certs with no warning is a rough way to find out.
We manage a good number of KVM/CloudStack hosts on the AccuWeb.Cloud side and cert expiry silently taking down agent connectivity is exactly the kind of thing that turns into a 2am page, so proactive expiry monitoring has saved us a few times.