CNE: Checking and Troubleshooting Environment Licensing

The DXP operator reports an environment’s automatic licensing status on the environment’s LiferayEnvironment resource with the short name lenv. The operator checks the license every ten minutes by default, and each check is a heartbeat. Read the status with kubectl, then look up each failing condition in Troubleshooting.

The examples below use the namespace liferay and the LiferayEnvironment name liferay-default. Replace them with your own, including in Secret names such as liferay-default-activation.

Checking License Status

Three views show the environment’s status:

The kubectl get lenv columns summarize it.
kubectl describe lenv lists each condition with its reason and message.
The status.apps list shows each Marketplace add-on’s download state.

Columns and Phases

Run kubectl get lenv to see the environment’s license status:

kubectl --namespace liferay get lenv

The output has these columns:

Column	Description
`WORKLOAD`	The environment’s StatefulSet
`ACTIVATED`	The status of the `Activated` condition
`ADD-ONS`	The status of the `AddOnsReady` condition
`MAX`	The license’s node limit
`VALID-UNTIL`	The license expiration
`DESIRED`	The `liferay-default` chart’s `replicaCount`, empty when autoscaling is enabled
`CEILING`	The maximum number of replicas: the node limit, or fewer when the license is invalid or expired, or after the provisioning grace period ends
`EFFECTIVE`	The replica count the operator set on the StatefulSet

Add --output wide to show PHASE, ENVIRONMENT-ID, ACTIVATED-AT, and UNREACHABLE-SINCE as well. PHASE has one of these values:

Phase	Description
`Pending`	The environment is waiting for its activation code or bundle.
`Degraded`	Activation failed, the license is expired or invalid, or the provisioning service is unreachable.
`Ready`	Neither `Pending` nor `Degraded` applies. `Ready` doesn’t mean every condition is healthy.

Conditions

To see each condition with its reason and message, describe the environment:

kubectl --namespace liferay describe lenv liferay-default

The environment reports these conditions:

Condition	Description
`Activated`	Whether the environment is activated
`AddOnsReady`	Whether every entitled Marketplace add-on is downloaded
`LicenseValid`	Whether the current license passes validation
`ProvisioningGracePeriodExpired`	Whether the provisioning service stayed unreachable longer than the provisioning grace period (`provisioningGracePeriod` on the `liferay-dxp-operator` chart, `168h` by default), shown only when `True`
`ProvisioningReachable`	Whether the most recent online heartbeat got the license from the provisioning service
`ReplicasCountValid`	Whether the requested replica count fits within the replica ceiling

Add-On Status

To see the state of each add-on, read the environment’s status.apps list:

kubectl --namespace liferay get lenv liferay-default --output jsonpath='{.status.apps}'

The list has one entry for each entitled add-on. For an online environment, it also has one Orphaned entry for each add-on whose entitlement ended. An entry can have these fields. The last three appear only in an online environment’s entry after its download fails, and message and nextRetry only while the state is Failed.

Field	Description
`name`	The add-on’s name
`virtualEntryId`	The add-on’s ID and the name of its file, `[virtualEntryId].lpkg`, in the environment’s namespace folder on the marketplace volume
`checksum`	The add-on’s SHA-256 checksum, from the entitlement
`state`	`Downloaded`, `Downloading`, `Failed`, or `Orphaned`
`consecutiveFailures`	The number of failed download attempts in a row
`message`	The last download error
`nextRetry`	When the operator next tries the download

Troubleshooting

Find the condition that reports a problem, then the reason kubectl describe lenv shows for it. If the activation workflow fails, find its failed step instead. If an add-on is missing from Liferay DXP while every condition is healthy, see An Add-On Is Downloaded but Not Installed.

`Activated`

The Activated condition is False for one of these reasons.

`AwaitingActivationCode`

The operator can’t read the activation code. This is also why the ACTIVATED column can stay empty after you create the activation Secret (liferay-default-activation).

If the Secret is missing from the environment’s namespace, create it. If it has no code under the expected key (activationCode by default), replace it with kubectl apply, because kubectl create fails on a Secret that already exists:

kubectl --namespace liferay create secret generic liferay-default-activation \
    --from-literal=activationCode="[activation code]" \
    --dry-run=client --output yaml | kubectl apply --filename -

`ActivationRejected`

The condition’s message gives the cause:

activation: unexpected status [code]: The service returned an error. Wait for the next retry, and contact Liferay Support if the error persists.
provisioning: activation rejected: The provisioning service rejected the code. Get a new code from One Liferay and replace the code in the activation Secret.
Any other message, such as a connection error: The operator could not reach the service. Restore outbound HTTPS access to the provisioning service (https://api.one.liferay.com by default).

The operator retries with increasing delays. To retry immediately, set the licensing.liferay.com/activation-requested-at annotation to the current time:

kubectl --namespace liferay annotate lenv liferay-default \
    licensing.liferay.com/activation-requested-at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
    --overwrite

`AwaitingOfflineActivationBundle`

The operator finds no bundle file, for one of these reasons:

licensing.offlineActivationBundle is not set. Set it with the other offline mode values.
No file with that name exists in the folder named after the environment’s namespace on the marketplace volume. Copy the bundle there.
marketplace.enabled on the liferay-dxp-operator chart is false, so the operator has no marketplace volume. Turn on the marketplace volume.

`OfflineActivationBundleInvalid`

The bundle can’t be read as a ZIP file, or it lacks manifest.json. Replace the bundle.

The Activation Workflow Fails

Find the failed step of the activation workflow:

read-environment: Nothing changed. Either the environment is already activated, or no LiferayEnvironment has the name in the liferay-environment-name parameter. In the second case, submit the workflow again with the right name.
wait: The operator did not activate the environment within 300 seconds. Read the Activated condition’s reason with kubectl describe lenv and your own credentials, not the login token. Fix the cause, and submit the workflow again.
Any other step: Read the failed step’s log, fix the error, and submit the workflow again.

`AddOnsReady`

AddOnsReady keeps its last value while Activated, LicenseValid, or ProvisioningReachable is False. It also keeps its last value while ReplicasCountValid is False with the reason WorkloadUpdateRejected or AutoscalerUpdateRejected. Fix those conditions first.

The AddOnsReady condition is False for one of these reasons.

`DownloadsFailing`

The condition’s message, Unable to download [n] of [total] entitled add-ons: [names]., names the add-ons the operator can’t download.

For an online environment, check that the marketplace volume is turned on. If it is, read the message of each Failed entry in status.apps for the cause:

add-on download: unexpected status [code]: The add-on’s download request returned an HTTP error.
addon cache: checksum mismatch for virtual entry [id]: got [sum], want [sum]: The newly downloaded .lpkg file doesn’t match the checksum the entitlement supplies.
Post "": unsupported protocol scheme "": The entitlement has no download link for the add-on.
Any other message: The download request failed, for example with a connection error, or the operator can’t write to the marketplace volume. If the message shows a permission error on the volume, make the volume’s root writable by the operator.

For an online environment, the operator keeps retrying each failed add-on, and the entry’s nextRetry shows when the next attempt runs. A failing add-on doesn’t affect the license or the other add-ons. Contact Liferay Support if the error persists.

For an offline environment, the operator’s log gives the cause on an Unable to extract add-on line:

kubectl --namespace dxp-operator-system logs deployment/dxp-operator

If the bundle lacks the add-on or the add-on’s file is damaged, replace the bundle. If the operator can’t write to the marketplace volume, make the volume’s root writable by the operator.

`Downloading`

Downloads are in progress. Wait for them to finish, then check AddOnsReady again.

An Add-On Is Downloaded but Not Installed

The ADD-ONS column shows True and the add-on’s status.apps entry has the state Downloaded, but the add-on is missing from Liferay DXP. Check a Liferay DXP pod, such as liferay-default-0:

Check that the pod has the sync sidecar, liferay-marketplace-sync:
```
kubectl --namespace liferay get pod liferay-default-0 --output jsonpath='{.spec.initContainers[*].name}'
```
If the output doesn’t list liferay-marketplace-sync, turn on add-on delivery. If the liferay-default chart’s marketplace.enabled is already true, check whether the StatefulSet has replaced the pod:
```
kubectl --namespace liferay rollout status statefulset/liferay-default
```
If the rollout doesn’t finish, list the pods with kubectl --namespace liferay get pods. If one stays Pending, read its events with kubectl --namespace liferay describe pod [pod name]. An event naming the claim liferay-default-marketplace means the environment’s claim doesn’t exist, so create it. If the rollout finishes and the pod still lacks liferay-marketplace-sync, the value didn’t reach the chart, so check where you set it.
Read the sync sidecar’s log:
```
kubectl --namespace liferay logs liferay-default-0 --container liferay-marketplace-sync
```
For each add-on it copies, the log has a line like the one in Checking That an Add-On Arrived, naming [virtualEntryId].lpkg. The sync sidecar checks for new files every marketplace.syncInterval seconds (600 by default). If the line is there, restart the pods to complete the installation.

If the log says bash: /liferay-init-scripts/deploy-sync.sh: No such file or directory, the operator doesn’t write the script the sync sidecar runs. Upgrade the operator to chart 0.2.0 or later.
If a full interval passes with no line for the add-on, list the files the sync sidecar sees:
```
kubectl --namespace liferay exec liferay-default-0 --container liferay-marketplace-sync -- ls /marketplace
```
If [virtualEntryId].lpkg appears, the sync sidecar copied it before the pod last restarted, so Liferay DXP already tried to install it. Search the Liferay DXP container’s log for a deployment error:
```
kubectl --namespace liferay logs liferay-default-0 --container liferay-default | grep "Unable to deploy LPKG file"
```
Contact Liferay Support, and include any error the search finds.

If [virtualEntryId].lpkg doesn’t appear, the environment’s claim, liferay-default-marketplace, doesn’t point to the same storage directory as the operator’s claim in dxp-operator-system. To fix it, see Turning On Add-On Delivery.

`ProvisioningReachable`

The ProvisioningReachable condition is False with the reason EntitlementsFetchFailed: the operator could not get the license from One Liferay’s provisioning service. The environment keeps running on its last license until the provisioning grace period expires. The condition’s message gives the cause:

A message that starts with entitlements: decode or entitlements: unexpected status: The service returned an error or a response the operator can’t read. Wait for the next retry, and contact Liferay Support if the error persists.
Any other message, such as a connection error: The operator could not reach the service. Restore outbound HTTPS access to the provisioning service (https://api.one.liferay.com by default).

`ProvisioningGracePeriodExpired`

The ProvisioningGracePeriodExpired condition is True with the reason ProvisioningUnreachable. The operator kept failing to get the license (EntitlementsFetchFailed) for longer than the provisioning grace period, so it scaled the StatefulSet down to one replica. Fix the EntitlementsFetchFailed cause.

When the operator gets the license again, it scales the StatefulSet back to replicaCount, up to the node limit. With autoscaling, the StatefulSet stays at one replica until the autoscaler scales it.

`LicenseValid`

The LicenseValid condition is False for one of these reasons.

`Expired`

After the license expires, the operator caps the StatefulSet at one replica until the expiration grace period ends (expirationGracePeriod on the liferay-dxp-operator chart, 2160h by default). Then it scales the StatefulSet to zero. Renew the subscription. An online environment requests its license on every heartbeat, so it gets the renewed license on the next one. An offline environment needs a new bundle.

`Invalid`

The operator can’t read the license’s owner or expiration date, so it scales the StatefulSet to zero. For an online environment, contact Liferay Support. For an offline environment, replace the bundle.

When the environment receives a license the operator can read, it scales the StatefulSet back as it does after an EnvironmentMismatch.

`EnvironmentMismatch`

The license was issued for a different environment, and the operator scales the StatefulSet to zero. kubectl get lenv shows 0 in the MAX column.

For an online environment, contact Liferay Support. For an offline environment, request a new bundle with this environment’s current offline request.

When the environment receives a license issued for it, the operator scales the StatefulSet back to replicaCount, up to the node limit. If the StatefulSet stays at zero replicas, scale it up yourself, for example with kubectl scale.

`ReplicasCountValid`

The ReplicasCountValid condition is False or Unknown for one of these reasons.

`ExceedsLicensedMaximum`

The requested replica count exceeds the replica ceiling in the CEILING column, so the operator runs only up to the ceiling. Scaling past the ceiling also fails with replicas [count] exceeds the replica ceiling [ceiling] for StatefulSet "[name]".

If the node limit in the MAX column is 0, fix EnvironmentMismatch. If LicenseValid reports Invalid, fix Invalid, even when the MAX column is empty. If the ceiling is below the node limit, fix ProvisioningGracePeriodExpired or Expired, whichever is reported. Otherwise, lower the requested count to fit the node limit.

`WorkloadUpdateRejected`

Kubernetes rejected the operator’s change to the StatefulSet’s replica count. Fix the cause given in the condition’s message. The operator retries.

`WorkloadNotFound`

ReplicasCountValid is Unknown: no StatefulSet in the namespace has the name in spec.workloadRef.name, so the operator enforces no node limit. Correct workloadRef on the LiferayEnvironment you wrote.

Capability: