How to replicate a production environment into a non-production one?

Issue

In order to test new developments, we want to set up a test environment that is a copy of the production environment to confirm that we will not have problems in the subsequent deployment in production.

How can we approach this operation? Do you have any official guide for this or any suggestions?

Environment

Any Liferay version

Resolution

There is no official Liferay guide to creating non-production environments from production environments.

The most similar is Creating a Backup & Disaster Recovery Plan for Liferay Digital Experience Platforms, article, which mentions the different aspects of the environment you have to consider.

Steps to follow

In order to perform the copy, you would have to follow the following steps:

Make a production backup (Liferay installation, DB data, document library, and Elasticsearch).
Create a new environment in which to restore the production backup.
Adjust the configuration of the new environment before startup to avoid problems during startup or to prevent the new environment from connecting to production machines
Start the new environment

1. Backup

Perform standard production backup, storing:

Liferay installation, including developments, portal-ext.properties, and other application server configurations.
The database information.
The Elasticsearch index information.
The Document Library.

If you need more information on how to perform the backup, check out the following resources:

2. Preparing the new environment

When preparing the new environment, it will be necessary to restore all information from the production backup.

You will have to install a new DB and a new Elasticsearch server to import all the backup data from the production environment.

Regarding the installation of Liferay, you can choose one of two options:

Option 1: Restore the full copy of the Liferay installation.
Option 2: Install a clean Liferay bundle and apply all the developments, customizations, and configurations from the backup copy

Option 2 is similar to the one carried out when upgrading to a new liferay Update (see https://learn.liferay.com/w/dxp/installation-and-upgrades/maintaining-a-liferay-installation/updating-liferay )

It would also be necessary to configure the web servers in front of the Liferay installation.

3. Reconfigure the new environment

You have to reconfigure the new environment before booting to avoid startup problems or to prevent the new environment from connecting to production machines.

For more information, see the section “Configuration of the new non-production environment”

4. Start up the new environment

Start the new environment and verify that no ERRORs or WARNs occur during startup, check for any errors that may appear.

Important: do not start the new non-production environment until you have reconfigured it to avoid using the production configuration, for more information see the next section.

Configuration of the new non-production environment

Before the first boot, it is necessary to reconfigure the new environment to avoid problems during the startup or to prevent the new environment from connecting to the production machines.

So, it is necessary to check the following points:

Portal-ext.properties

From the production portal-ext.properties, in the new environment, it is necessary:

Modify the DB configuration to point to the DB of the new environment.
Check the rest of the file and replace any reference to IPs or domains of the production environment
(for example: up to 7.3 in case you use redirect.url.security.mode=ip, you have to update the redirect.url.ips.allowed property with the IPs of the new environment).

System Settings Configuration

It is necessary to replace the System Settings configuration that is stored in the Liferay DB and can be overwritten using the [LIFERAY_HOME]/osgi/configs folder.

In this way, in order to take into account configurations for the new environment on the first startup, these .config files can be applied in [LIFERAY_HOME]/osgi/configs

These two documentation articles may be useful:

The second article explains how to export the production configuration and import it into the copy environment.

Roughly speaking, the procedure is as follows:

Access the System Settings of the source production environment.
From the 3-dot menu at the top right, you can export all settings (Export All Settings), as a series of .config files.
Extract the zip file.
Edit the exported configuration files to adapt them to the data of the new environment: go through each of them and replace any reference to production servers.
Copy the .config configuration files to the $LIFERAY_HOME/osgi/configs folder of the new environment. (don’t copy the .zip itself)

The .config information will be read, processed, and saved in the database at the first startup.

Instance Settings configuration

It would be necessary to perform the same procedure for the system settings of each of the system instances.

Follow the same steps:

Access the Instance Settings of the source production environment.
From the 3-dot menu at the top right, you can export all settings (Export All Settings), as a series of .config files.
Extract the zip file.
Edit the exported configuration files to adapt them to the data of the new environment: review each of them and replace any reference to production servers.
Copy the .config configuration files to the $LIFERAY_HOME/osgi/configs folder of the new environment. (don’t copy the .zip itself)

ElasticSearch configuration:

If the new environment is started without further ado, it will connect to the production ElasticSearch, which can be a problem since the new environment will start indexing information in the production index.
Therefore it is important to replace in System Settings the Elasticsearch configuration that if you follow the steps above will be exported in the file com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration.config.
More information:
- https://learn.liferay.com/en/w/dxp/using-search/installing-and-upgrading-a-search-engine/elasticsearch/connecting-to-elasticsearch
- https://help.liferay.com/hc/articles/360029031671-Configuring-the-Liferay-Elasticsearch-Connector

LDAP configuration:

If the new environment is started without further ado, it will connect to the production LDAP, which can be a problem.
Therefore it is important to replace in Instance Settings the LDAP configuration that if you follow the steps above will be exported in the file com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration.config
com.liferay.portal.security.ldap.configuration.LDAPServerConfiguration-<<UUID>>.config

Domain names:

Replace the domain names in all portal-ext.properties and System Settings.
If you have configured your domain name in the instance or site settings, you also have to replace it in both places.
Check the virtualhost table of your DB and replace the domain names there.

Users’ emails:

To avoid emails being sent to end users during the testing environment, it is important to modify the emails in the Users_ table, replacing all of them with a dummy account.
It is also advisable to apply some kind of anonymization of the users’ data. For more information about tables that store sensitive data, see: https://help.liferay.com/hc/articles/360018136072

Network connectivity: Best practices

In order to avoid problems, a good practice when duplicating a production environment is to configure the network name resolution and routing so that the resources of the source environment cannot be accessed from the copy machine, so that if, for example, from that Liferay test machine you try to access the production DB or Elasticsearch, it is not possible because it does not connect through the network.
This can be achieved for example by using docker containers, since it would isolate the connectivity with the production DB or Elasticsearch by using its own private network.
It is also important to avoid being able to connect to the production LDAP.

Liferay license

A last point to consider would be to deploy a valid license for the new environment because if you start with the production license you will get an error that the license is not valid. You will have to deploy a specific license for that machine.