title	platform	product	category	subcategory	date
Data Center App Performance Toolkit User Guide For Confluence	platform	marketplace	devguide	build	2021-06-16

Data Center App Performance Toolkit User Guide For Confluence

This document walks you through the process of testing your app on Confluence using the Data Center App Performance Toolkit. These instructions focus on producing the required performance and scale benchmarks for your Data Center app.

In this document, we cover the use of the Data Center App Performance Toolkit on two types of environments:

Development environment: Confluence Data Center environment for a test run of Data Center App Performance Toolkit and development of app-specific actions. We recommend you use the AWS Quick Start for Confluence Data Center with the parameters prescribed here.

Set up a development environment Confluence Data Center on AWS.
Create a dataset for the development environment.
Run toolkit on the development environment locally.
Develop and test app-specific actions locally.

Enterprise-scale environment: Confluence Data Center environment used to generate Data Center App Performance Toolkit test results for the Marketplace approval process. Preferably, use the AWS Quick Start for Confluence Data Center with the parameters prescribed below. These parameters provision larger, more powerful infrastructure for your Confluence Data Center.

Set up an enterprise-scale environment Confluence Data Center on AWS.
Load an enterprise-scale dataset on your Confluence Data Center deployment.
Set up an execution environment for the toolkit.
Running the test scenarios from execution environment against enterprise-scale Confluence Data Center.

Development environment

Running the tests in a development environment helps familiarize you with the toolkit. It'll also provide you with a lightweight and less expensive environment for developing. Once you're ready to generate test results for the Marketplace Data Center Apps Approval process, run the toolkit in an enterprise-scale environment.

1. Setting up Confluence Data Center development environment

We recommend that you set up development environment using the AWS Quick Start for Confluence Data Center (How to deploy tab). All the instructions on this page are optimized for AWS. If you already have an existing Confluence Data Center environment, you can also use that too (if so, skip to Create a dataset for the development environment).

Using the AWS Quick Start for Confluence

If you are a new user, perform an end-to-end deployment. This involves deploying Confluence into a new ASI:

Navigate to AWS Quick Start for Confluence Data Center > How to deploy tab > Deploy into a new ASI link.

If you have already deployed the ASI separately by using the ASI Quick StartASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, or Confluence Data Center development environment) with ASI, deploy Confluence into your existing ASI:

Navigate to AWS Quick Start for Confluence Data Center > How to deploy tab > Deploy into your existing ASI link.

{{% note %}} You are responsible for the cost of AWS services used while running this Quick Start reference deployment. This Quick Start doesn't have any additional prices. See Amazon EC2 pricing for more detail. {{% /note %}}

To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.

AWS cost estimation for the development environment

AWS Confluence Data Center development environment infrastructure costs about 20 - 40$ per working week depending on such factors like region, instance type, deployment type of DB, and other.

Quick Start parameters for development environment

All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.

Confluence setup

Parameter	Recommended value
Collaborative editing mode	synchrony-local
Confluence Version	The Data Center App Performance Toolkit officially supports `7.4.9` (Long Term Support release) and `7.0.5` Platform Release

Cluster nodes

Parameter	Recommended value
Cluster node instance type	t3.medium (we recommend this instance type for its good balance between price and performance in testing environments)
Maximum number of cluster nodes	1
Minimum number of cluster nodes	1
Cluster node instance volume size	50

Database

Parameter	Recommended value
The database engine	PostgresSQL
The database engine version to use	10
Database instance class	db.t3.medium
RDS Provisioned IOPS	1000
Master (admin) password	Password1!
Enable RDS Multi-AZ deployment	false
Application user database password	Password1!
Database storage	200

Networking (for new ASI)

Parameter	Recommended value
Trusted IP range	0.0.0.0/0 (for public access) or your own trusted IP range
Availability Zones	Select two availability zones in your region
Permitted IP range	0.0.0.0/0 (for public access) or your own trusted IP range
Make instance internet facing	True
Key Name	The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info.

Networking (for existing ASI)

Parameter	Recommended value
Make instance internet facing	True
Permitted IP range	0.0.0.0/0 (for public access) or your own trusted IP range
Key Name	The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info.

Running the setup wizard

After successfully deploying Confluence Data Center in AWS, you'll need to configure it:

In the AWS console, go to Services > CloudFormation > Stack > Stack details > Select your stack.
On the Outputs tab, copy the value of the LoadBalancerURL key.
Open LoadBalancerURL in your browser. This will take you to the Confluence setup wizard.
On the Get apps page, do not select addition apps, just click Next.
On the next page, populate the Your License Key field by either:
- Using your existing license, or
- Generating an evaluation license, or
- Contacting Atlassian to be provided two time-bomb licenses for testing. Ask for it in your DCHELP ticket. Click Next.
On the Load Content page, click on the Empty Site.
On the Configure User Management page, click on the Manage users and groups within Confluence.
On the Configure System Administrator Account page, populate the following fields:
- Username: admin (recommended)
- Name: admin (recommended)
- Email Address: email address of the admin user
- Password: admin (recommended)
- Confirm Password: admin (recommended) Click Next.
On the Setup Successful page, click on the Start.
After going through the welcome setup, enter any Space name to create an initial space and click Continue.
Enter the first page title and click Publish.

2. Generate dataset for development environment

After creating the development environment Confluence Data Center, generate test dataset to run Data Center App Performance Toolkit:

1 space with 1-5 pages and 1-5 blog posts.

3. Run toolkit on the development environment locally

{{% warning %}} Make sure English language is selected as a default language on the > General configuration > Languages page. Other languages are not supported by the toolkit. {{% /warning %}}

Clone Data Center App Performance Toolkit locally.
Follow the README.md instructions to set up toolkit locally.
Navigate to dc-app-performance-toolkit/app folder.
Open the confluence.yml file and fill in the following variables:
- application_hostname: your_dc_confluence_instance_hostname without protocol.
- application_protocol: http or https.
- application_port: for HTTP - 80, for HTTPS - 443, 8080, 1990 or your instance-specific port.
- secure: True or False. Default value is True. Set False to allow insecure connections, e.g. when using self-signed SSL certificate.
- application_postfix: it is empty by default; e.g., /confluence for url like this http://localhost:1990/confluence.
- admin_login: admin user username.
- admin_password: admin user password.
- load_executor: executor for load tests. Valid options are jmeter (default) or locust.
- concurrency: 2 - number of concurrent JMeter/Locust users.
- test_duration: 5m - duration of the performance run.
- ramp-up: 5s - amount of time it will take JMeter or Locust to add all test users to test execution.
- total_actions_per_hour: 2000 - number of total JMeter/Locust actions per hour.
- WEBDRIVER_VISIBLE: visibility of Chrome browser during selenium execution (False is by default).
Run bzt.
```
bzt confluence.yml
```
Review the resulting table in the console log. All JMeter/Locust and Selenium actions should have 95+% success rate.
In case some actions does not have 95+% success rate refer to the following logs in dc-app-performance-toolkit/app/results/confluence/YY-MM-DD-hh-mm-ss folder:
- results_summary.log: detailed run summary
- results.csv: aggregated .csv file with all actions and timings
- bzt.log: logs of the Taurus tool execution
- jmeter.*: logs of the JMeter tool execution
- locust.*: logs of the Locust tool execution (in case you use Locust as load_executor in confluence.yml)
- pytest.*: logs of Pytest-Selenium execution

{{% warning %}} Do not proceed with the next step until you have all actions 95+% success rate. Ask support if above logs analysis did not help. {{% /warning %}}

4. Develop and test app-specific action locally

Data Center App Performance Toolkit has its own set of default test actions for Confluence Data Center: JMeter/Locust and Selenium for load and UI tests respectively.

App-specific action - action (performance test) you have to develop to cover main use cases of your application. Performance test should focus on the common usage of your application and not to cover all possible functionality of your app. For example, application setup screen or other one-time use cases are out of scope of performance testing.

Define main use case of your app. Usually it is one or two main app use cases.
Your app adds new UI elements in Confluence Data Center - Selenium app-specific action has to be developed.
Your app introduces new endpoint or extensively calls existing Confluence Data Center API - JMeter/Locust app-specific actions has to be developed.
JMeter and Locust actions are interchangeable, so you could select the tool you prefer:

JMeter - UI-based performance tool.
Locust - code-based (Python requests library) performance tool.

{{% note %}} We strongly recommend developing your app-specific actions on the development environment to reduce AWS infrastructure costs. {{% /note %}}

Custom dataset

You can filter your own app-specific pages/blog posts for your app-specific actions.

Create app-specific pages/blog posts that have specific anchor in title, e.g. AppPage anchor and pages titles like AppPage1, AppPage2, AppPage3.
Go to the search page of your Confluence Data Center - CONFLUENCE_URL/dosearchsite.action?queryString= (Confluence versions 6.X and below) or just click to search field in UI (Confluence versions 7.X and higher).
Write CQL that filter just your pages or blog posts from step 1, e.g. title ~ 'AppPage*'.
Edit Confluence configuration file dc-app-performance-toolkit/app/confluence.yml:
- custom_dataset_query: CQL from step 3.

Next time when you run toolkit, custom dataset pages will be stored to the dc-app-performance-toolkit/app/datasets/confluence/custom_pages.csv with columns: page_id, space_key.

Example of app-specific Selenium action development with custom dataset

You develop an app that adds additional UI elements to Confluence pages or blog posts. In this case, you should develop Selenium app-specific action:

Create app-specific Confluence pages with AppPagee anchor in title: AppPage1, AppPage2, *AppPage3, etc.
Go to the search page of your Confluence Data Center - CONFLUENCE_URL/dosearchsite.action?queryString= (Confluence versions 6.X and below) or just click to search field in UI (Confluence versions 7.X and higher) and check if CQL is correct: title ~ 'AppPage*'.
Edit dc-app-performance-toolkit/app/confluence.yml configuration file and set custom_dataset_query: "title ~ 'AppPage*'".
Extend example of app-specific action in dc-app-performance-toolkit/app/extension/confluence/extension_ui.py.
Code example. So, our test has to open page or blog post with app-specific UI element and measure time to load of this app-specific page or blog post.
If you need to run app_speicifc_action as specific user uncomment app_specific_user_login function in code example. Note, that in this case test_1_selenium_custom_action should follow just before test_2_selenium_z_log_out action.
In dc-app-performance-toolkit/app/selenium_ui/confluence_ui.py, review and uncomment the following block of code to make newly created app-specific actions executed:

# def test_1_selenium_custom_action(confluence_webdriver, confluence_datasets, confluence_screen_shots):
#     extension_ui.app_specific_action(confluence_webdriver, confluence_datasets)

Run toolkit with bzt confluence.yml command to ensure that all Selenium actions including app_specific_action are successful.

Example of app-specific Locust/JMeter action development

You develop an app that introduces new GET and POST endpoints in Confluence Data Center. In this case, you should develop Locust or JMeter app-specific action.

Locust app-specific action development example

Extend example of app-specific action in dc-app-performance-toolkit/app/extension/confluence/extension_locust.py, so that test will call the endpoint with GET request, parse response use these data to call another endpoint with POST request and measure response time.
Code example.
In dc-app-performance-toolkit/app/confluence.yml set load_executor: locust to make locust as load executor.
Set desired execution percentage for standalone_extension. Default value is 0, which means that standalone_extension action will not be executed. Locust uses actions percentage as relative weights, so if some_action: 10 and standalone_extension: 20 that means that standalone_extension will be called twice more.
Set standalone_extension weight in accordance with the expected frequency of your app use case compared with other base actions.
App-specific tests could be run (if needed) as a specific user. Use @run_as_specific_user(username='specific_user_username', password='specific_user_password') decorator for that.
Run toolkit with bzt confluence.yml command to ensure that all Locust actions including app_specific_action are successful.

JMeter app-specific action development example

Check that confluence.yml file has correct settings of application_hostname, application_protocol, application_port, application_postfix, etc.
Set desired execution percentage for standalone_extension. Default value is 0, which means that standalone_extension action will not be executed. For example, for app-specific action development you could set percentage of standalone_extension to 100 and for all other actions to 0 - this way only login_and_view_dashboard and standalone_extension actions would be executed.
Navigate to dc-app-performance-toolkit/app folder and run from virtualenv(as described in dc-app-performance-toolkit/README.md):

python util/jmeter/start_jmeter_ui.py --app confluence
Open Confluence thread group > actions per login and navigate to standalone_extension
Add GET HTTP Request: right-click to standalone_extension > Add > Sampler HTTP Request, chose method GET and set endpoint in Path.
Add Regular Expression Extractor: right-click to to newly created HTTP Request > Add > Post processor > Regular Expression Extractor
Add Response Assertion: right-click to newly created HTTP Request > Add > Assertions > Response Assertion and add assertion with Contains, Matches, Equals, etc types.
Add POST HTTP Request: right-click to standalone_extension > Add > Sampler HTTP Request, chose method POST, set endpoint in Path and add Parameters or Body Data if needed.
Right-click on View Results Tree and enable this controller.
Click Start button and make sure that login_and_view_dashboard and standalone_extension are successful.
Right-click on View Results Tree and disable this controller. It is important to disable View Results Tree controller before full-scale results generation.
Click Save button.
To make standalone_extension executable during toolkit run edit dc-app-performance-toolkit/app/confluence.yml and set execution percentage of standalone_extension accordingly to your use case frequency.
App-specific tests could be run (if needed) as a specific user. In the standalone_extension uncomment login_as_specific_user controller. Navigate to the username:password config element and update values for app_specific_username and app_specific_password names with your specific user credentials. Also make sure that you located your app-specific tests between login_as_specific_user and login_as_default_user_if_specific_user_was_loggedin controllers.
Run toolkit to ensure that all JMeter actions including standalone_extension are successful.

Using JMeter variables from the base script

Use or access the following variables of the extension script from the base script. They can also be inherited.

${blog_id} - blog post id being viewed or modified (e.g. 23766699)
${blog_space_key} - blog space key (e.g. PFSEK)
${page_id} - page if being viewed or modified (e.g. 360451)
${space_key} - page space key (e.g. TEST)
${file_path} - path of file to upload (e.g. datasets/confluence/static-content/upload/test5.jpg)
${file_type} - type of the file (e.g. image/jpeg)
${file_name} - name of the file (e.g. test5.jpg)
${username} - the logged in username (e.g. admin)

{{% warning %}} App-specific actions are required. Do not proceed with the next step until you have completed app-specific actions development and got successful results from toolkit run. {{% /warning %}}

Enterprise-scale environment

After adding your custom app-specific actions, you should now be ready to run the required tests for the Marketplace Data Center Apps Approval process. To do this, you'll need an enterprise-scale environment.

5. Setting up Confluence Data Center enterprise-scale environment

We recommend that you use the AWS Quick Start for Confluence Data Center (How to deploy tab) to deploy a Confluence Data Center testing environment. This Quick Start will allow you to deploy Confluence Data Center with a new Atlassian Standard Infrastructure (ASI) or into an existing one.

The ASI is a Virtual Private Cloud (VPC) consisting of subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components required by all Atlassian applications, and then deploys Confluence into this new VPC. Deploying Confluence with a new ASI takes around 50 minutes. With an existing one, it'll take around 30 minutes.

Using the AWS Quick Start for Confluence

If you are a new user, perform an end-to-end deployment. This involves deploying Confluence into a new ASI:

Navigate to AWS Quick Start for Confluence Data Center > How to deploy tab > Deploy into a new ASI link.

If you have already deployed the ASI separately by using the ASI Quick StartASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, or Confluence Data Center development environment) with ASI, deploy Confluence into your existing ASI:

Navigate to AWS Quick Start for Confluence Data Center > How to deploy tab > Deploy into your existing ASI link.

{{% note %}} You are responsible for the cost of the AWS services used while running this Quick Start reference deployment. There is no additional price for using this Quick Start. For more information, go to aws.amazon.com/pricing. {{% /note %}}

To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.

AWS cost estimation

AWS Pricing Calculator provides an estimate of usage charges for AWS services based on certain information you provide. Monthly charges will be based on your actual usage of AWS services, and may vary from the estimates the Calculator has provided.

*The prices below are approximate and may vary depending on factors such as (region, instance type, deployment type of DB, etc.)

Stack	Estimated hourly cost ($)
One Node Confluence DC	0.8 - 1.1
Two Nodes Confluence DC	1.2 - 1.7
Four Nodes Confluence DC	2.0 - 3.0

Stop cluster nodes

To reduce AWS infrastructure costs you could stop cluster nodes when the cluster is standing idle.
Cluster node might be stopped by using Suspending and Resuming Scaling Processes.

To stop one node within the cluster, follow the instructions below:

In the AWS console, go to Services > EC2 > Auto Scaling Groups and open the necessary group to which belongs the node you want to stop.
Click Edit (in case you have New EC2 experience UI mode enabled, press Edit on Advanced configuration) and add HealthCheck to the Suspended Processes. Amazon EC2 Auto Scaling stops marking instances unhealthy as a result of EC2 and Elastic Load Balancing health checks.
Go to EC2 Instances, select instance, click Instance state > Stop instance.

To return node into a working state follow the instructions:

Go to EC2 Instances, select instance, click Instance state > Start instance, wait a few minutes for node to become available.
Go to EC2 Auto Scaling Groups and open the necessary group to which belongs the node you want to start.
Press Edit (in case you have New EC2 experience UI mode enabled, press Edit on Advanced configuration) and remove HealthCheck from Suspended Processes of Auto Scaling Group.

Stop database

To reduce AWS infrastructure costs database could be stopped when the cluster is standing idle. Keep in mind that database would be automatically started in 7 days.

To stop database:

In the AWS console, go to Services > RDS > Databases.
Select cluster database.
Click on Actions > Stop.

To start database:

In the AWS console, go to Services > RDS > Databases.
Select cluster database.
Click on Actions > Start.

Quick Start parameters

All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.

Confluence setup

Parameter	Recommended value
Collaborative editing mode	synchrony-local
Confluence Version	The Data Center App Performance Toolkit officially supports `7.4.9` (Long Term Support release) and `7.0.5` Platform Release

Cluster nodes

Parameter	Recommended Value
Cluster node instance type	m5.2xlarge
Maximum number of cluster nodes	1
Minimum number of cluster nodes	1
Cluster node instance volume size	200

We recommend m5.2xlarge to strike the balance between cost and hardware we see in the field for our enterprise customers. More info could be found in public recommendations.

The Data Center App Performance Toolkit framework is also set up for concurrency we expect on this instance size. As such, underprovisioning will likely show a larger performance impact than expected.

Database

Parameter	Recommended Value
The database engine	PostgresSQL
The database engine version to use	10
Database instance class	db.m5.xlarge
RDS Provisioned IOPS	1000
Master (admin) password	Password1!
Enable RDS Multi-AZ deployment	false
Application user database password	Password1!
Database storage	200

{{% note %}} The Master (admin) password will be used later when restoring the SQL database dataset. If password value is not set to default, you'll need to change DB_PASS value manually in the restore database dump script (later in Preloading your Confluence deployment with an enterprise-scale dataset). {{% /note %}}

Networking (for new ASI)

Parameter	Recommended Value
Trusted IP range	0.0.0.0/0 (for public access) or your own trusted IP range
Availability Zones	Select two availability zones in your region
Permitted IP range	0.0.0.0/0 (for public access) or your own trusted IP range
Make instance internet facing	true
Key Name	The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info.

Networking (for existing ASI)

Parameter	Recommended Value
Make instance internet facing	true
Permitted IP range	0.0.0.0/0 (for public access) or your own trusted IP range
Key Name	The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info.

Application tuning

Parameter	Recommended Value
Catalina options	-Dreindex.thread.count=8 -Dreindex.attachments.thread.count=8 -Dconfluence.reindex.documents.to.pop=600

Running the setup wizard

After successfully deploying Confluence Data Center in AWS, you'll need to configure it:

In the AWS console, go to Services > CloudFormation > Stack > Stack details > Select your stack.
On the Outputs tab, copy the value of the LoadBalancerURL key.
Open LoadBalancerURL in your browser. This will take you to the Confluence setup wizard.
On the Get apps page, do not select addition apps, just click Next.
On the next page, populate the Your License Key field by either:
- Using your existing license, or
- Generating an evaluation license, or
- Contacting Atlassian to be provided two time-bomb licenses for testing. Ask for it in your DCHELP ticket. Click Next.
On the Load Content page, click on the Empty Site.
On the Configure User Management page, click on the Manage users and groups within Confluence.
On the Configure System Administrator Account page, populate the following fields:
- Username: admin (recommended)
- Name: admin (recommended)
- Email Address: email address of the admin user
- Password: admin (recommended)
- Confirm Password: admin (recommended) Click Next.
On the Setup Successful page, click on the Start.
After going through the welcome setup, enter any Space name to create an initial space and click Continue.
Enter the first page title and click Publish.

{{% note %}} After Preloading your Confluence deployment with an enterprise-scale dataset, the admin user will have admin/admin credentials. {{% /note %}}

6. Preloading your Confluence deployment with an enterprise-scale dataset

Data dimensions and values for an enterprise-scale dataset are listed and described in the following table.

Data dimensions	Value for an enterprise-scale dataset
Pages	~900 000
Blogposts	~100 000
Attachments	~2 300 000
Comments	~6 000 000
Spaces	~5 000
Users	~5 000

{{% note %}} All the datasets use the standard admin/admin credentials. {{% /note %}}

Pre-loading the dataset is a three-step process:

Importing the main dataset. To help you out, we provide an enterprise-scale dataset you can import either via the populate_db.sh script.
Restoring attachments. We also provide attachments, which you can pre-load via an upload_attachments.sh script.
Re-indexing Confluence Data Center. For more information, go to Re-indexing Confluence.

The following subsections explain each step in greater detail.

Importing the main dataset

You can load this dataset directly into the database (via a populate_db.sh script).

Loading the dataset via populate_db.sh script (~90 min)

{{% note %}} We recommend doing this via the CLI. {{% /note %}}

To populate the database with SQL:

In the AWS console, go to Services > EC2 > Instances.
On the Description tab, do the following:
- Copy the Public IP of the Bastion instance.
- Copy the Private IP of the Confluence node instance.

Using SSH, connect to the Confluence node via the Bastion instance:

For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):

ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
export SSH_OPTS1='-o ServerAliveInterval=60'
export SSH_OPTS2='-o ServerAliveCountMax=30'
ssh ${SSH_OPTS1} ${SSH_OPTS2} -o "proxycommand ssh -W %h:%p ${SSH_OPTS1} ${SSH_OPTS2} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

For more information, go to Connecting your nodes over SSH.

Download the populate_db.sh script and make it executable:

wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/confluence/populate_db.sh && chmod +x populate_db.sh

Review the following Variables section of the script:

DB_CONFIG="/var/atlassian/application-data/confluence/confluence.cfg.xml"
CONFLUENCE_CURRENT_DIR="/opt/atlassian/confluence/current"
CONFLUENCE_DB_NAME="confluence"
CONFLUENCE_DB_USER="postgres"
CONFLUENCE_DB_PASS="Password1!"
CONFLUENCE_VERSION_FILE="/media/atl/confluence/shared-home/confluence.version"
DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/confluence"

Run the script:

./populate_db.sh 2>&1 | tee -a populate_db.log

{{% note %}} Do not close or interrupt the session. It will take some time to restore SQL database. When SQL restoring is finished, an admin user will have admin/admin credentials.

In case of a failure, check the Variables section and run the script one more time. {{% /note %}}

Restoring attachments (~3 hours)

After Importing the main dataset, you'll now have to pre-load an enterprise-scale set of attachments.

{{% note %}} Populate DB and restore attachments scripts could be run in parallel in separate terminal sessions to save time. {{% /note %}}

Using SSH, connect to the Confluence node via the Bastion instance:

For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):

ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
export SSH_OPTS1='-o ServerAliveInterval=60'
export SSH_OPTS2='-o ServerAliveCountMax=30'
ssh ${SSH_OPTS1} ${SSH_OPTS2} -o "proxycommand ssh -W %h:%p ${SSH_OPTS1} ${SSH_OPTS2} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

For more information, go to Connecting your nodes over SSH.

Download the upload_attachments.sh script and make it executable:

wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/confluence/upload_attachments.sh && chmod +x upload_attachments.sh

Review the following Variables section of the script:

DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/confluence"
ATTACHMENTS_TAR="attachments.tar.gz"
ATTACHMENTS_DIR="attachments"
TMP_DIR="/tmp"
EFS_DIR="/media/atl/confluence/shared-home"

Run the script:

./upload_attachments.sh 2>&1 | tee -a upload_attachments.log

{{% note %}} Do not close or interrupt the session. It will take some time to upload attachments to Elastic File Storage (EFS). {{% /note %}}

Re-indexing Confluence Data Center (~2-3 hours)

For more information, go to Re-indexing Confluence.

{{% note %}} For Confluence 7, populate_db.sh script triggers index process automatically. So no need to start index manually once again, just wait until current index process is finished. {{% /note %}}

For Confluence 6:

Log in as a user with the Confluence System Administrators global permission.
Go to > General Configuration > Content Indexing.
Click Rebuild and wait until re-indexing is completed.

Confluence will be unavailable for some time during the re-indexing process.

Create Index Snapshot (~30 min)

For more information, go to Administer your Data Center search index.

Log in as a user with the Confluence System Administrators global permission.
Create any new page with a random content (without a new page index snapshot job will not be triggered).
Go to > General Configuration > Scheduled Jobs.
Find Clean Journal Entries job and click Run.

Make sure that Confluence index snapshot was created. To do that, use SSH to connect to the Confluence node via Bastion (where NODE_IP is the IP of the node):

ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
export SSH_OPTS1='-o ServerAliveInterval=60'
export SSH_OPTS2='-o ServerAliveCountMax=30'
ssh ${SSH_OPTS1} ${SSH_OPTS2} -o "proxycommand ssh -W %h:%p ${SSH_OPTS1} ${SSH_OPTS2} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

Download the index-snapshot.sh file. Then, make it executable and run it:

wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/confluence/index-snapshot.sh && chmod +x index-snapshot.sh
./index-snapshot.sh 2>&1 | tee -a index-snapshot.log

Index snapshot creation time is about 20-30 minutes. When index snapshot is successfully created, the following will be displayed in console output:

Snapshot was created successfully.

7. Setting up an execution environment

For generating performance results suitable for Marketplace approval process use dedicated execution environment. This is a separate AWS EC2 instance to run the toolkit from. Running the toolkit from a dedicated instance but not from a local machine eliminates network fluctuations and guarantees stable CPU and memory performance.

Go to GitHub and create a fork of dc-app-performance-toolkit.

Clone the fork locally, then edit the confluence.yml configuration file. Set enterprise-scale Confluence Data Center parameters:

    application_hostname: test_confluence_instance.atlassian.com   # Confluence DC hostname without protocol and port e.g. test-confluence.atlassian.com or localhost
    application_protocol: http      # http or https
    application_port: 80            # 80, 443, 8080, 2990, etc
    secure: True                    # Set False to allow insecure connections, e.g. when using self-signed SSL certificate
    application_postfix:            # e.g. /confluence in case of url like http://localhost:2990/confluence
    admin_login: admin
    admin_password: admin
    load_executor: jmeter           # jmeter and locust are supported. jmeter by default.
    concurrency: 200                # number of concurrent virtual users for jmeter or locust scenario
    test_duration: 45m
    ramp-up: 5m                     # time to spin all concurrent users
    total_actions_per_hour: 20000   # number of total JMeter/Locust actions per hour.

Push your changes to the forked repository.
Launch AWS EC2 instance.
- OS: select from Quick Start Ubuntu Server 18.04 LTS.
- Instance type: c5.2xlarge
- Storage size: 30 GiB
Connect to the instance using SSH or the AWS Systems Manager Sessions Manager.
```
ssh -i path_to_pem_file ubuntu@INSTANCE_PUBLIC_IP
```
Install Docker. Setup manage Docker as a non-root user.
Clone forked repository.

{{% note %}} At this stage app-specific actions are not needed yet. Use code from master branch with your confluence.yml changes. {{% /note %}}

You'll need to run the toolkit for each test scenario in the next section.

8. Running the test scenarios from execution environment against enterprise-scale Confluence Data Center

Using the Data Center App Performance Toolkit for Performance and scale testing your Data Center app involves two test scenarios:

Performance regression
Scalability testing

Each scenario will involve multiple test runs. The following subsections explain both in greater detail.

Scenario 1: Performance regression

This scenario helps to identify basic performance issues without a need to spin up a multi-node Confluence DC. Make sure the app does not have any performance impact when it is not exercised.

Run 1 (~50 min)

To receive performance baseline results without an app installed:

Use SSH to connect to execution environment.

Run toolkit with docker from the execution environment instance:

cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt confluence.yml

View the following main results of the run in the dc-app-performance-toolkit/app/results/confluence/YY-MM-DD-hh-mm-ss folder:
- results_summary.log: detailed run summary
- results.csv: aggregated .csv file with all actions and timings
- bzt.log: logs of the Taurus tool execution
- jmeter.*: logs of the JMeter tool execution
- pytest.*: logs of Pytest-Selenium execution

{{% note %}} Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above. {{% /note %}}

Run 2 (~50 min)

To receive performance results with an app installed:

Install the app you want to test.
Setup app license.

Run toolkit with docker from the execution environment instance:

cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt confluence.yml

{{% note %}} Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above. {{% /note %}}

Generating a performance regression report

To generate a performance regression report:

Use SSH to connect to execution environment.
Install and activate the virtualenv as described in dc-app-performance-toolkit/README.md
Allow current user (for execution environment default user is ubuntu) to access Docker generated reports:
```
sudo chown -R ubuntu:ubuntu /home/ubuntu/dc-app-performance-toolkit/app/results
```
Navigate to the dc-app-performance-toolkit/app/reports_generation folder.
Edit the performance_profile.yml file:
- Under runName: "without app", in the fullPath key, insert the full path to results directory of Run 1.
- Under runName: "with app", in the fullPath key, insert the full path to results directory of Run 2.

Run the following command:

python csv_chart_generator.py performance_profile.yml

In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss folder, view the .csv file (with consolidated scenario results), the .png chart file and performance scenario summary report.

Analyzing report

Use scp command to copy report artifacts from execution env to local drive:

From local machine terminal (Git bash terminal for Windows) run command:

export EXEC_ENV_PUBLIC_IP=execution_environment_ec2_instance_public_ip
scp -r -i path_to_exec_env_pem ubuntu@$EXEC_ENV_PUBLIC_IP:/home/ubuntu/dc-app-performance-toolkit/app/results/reports ./reports

Once completed, in the ./reports folder you will be able to review the action timings with and without your app to see its impact on the performance of the instance. If you see an impact (>20%) on any action timing, we recommend taking a look into the app implementation to understand the root cause of this delta.

Scenario 2: Scalability testing

The purpose of scalability testing is to reflect the impact on the customer experience when operating across multiple nodes. For this, you have to run scale testing on your app.

For many apps and extensions to Atlassian products, there should not be a significant performance difference between operating on a single node or across many nodes in Confluence DC deployment. To demonstrate performance impacts of operating your app at scale, we recommend testing your Confluence DC app in a cluster.

Run 3 (~50 min)

To receive scalability benchmark results for one-node Confluence DC with app-specific actions:

Apply app-specific code changes to a new branch of forked repo.
Use SSH to connect to execution environment.
Pull cloned fork repo branch with app-specific actions.

Run toolkit with docker from the execution environment instance:

cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt confluence.yml

{{% note %}} Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above. {{% /note %}}

Run 4 (~50 min)

{{% note %}} Before scaling your DC make sure that AWS vCPU limit is not lower than needed number. Use vCPU limits calculator to see current limit. The same article has instructions on how to increase limit if needed. {{% /note %}}

To receive scalability benchmark results for two-node Confluence DC with app-specific actions:

In the AWS console, go to CloudFormation > Stack details > Select your stack.
On the Update tab, select Use current template, and then click Next.
Enter 2 in the Maximum number of cluster nodes and the Minimum number of cluster nodes fields.
Click Next > Next > Update stack and wait until stack is updated.
Confirm new node is appeared on the > General Configuration > Clustering page.

Make sure that Confluence index successfully synchronized to the second node. To do that, use SSH to connect to the second node via Bastion (where NODE_IP is the IP of the second node):

ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
export SSH_OPTS1='-o ServerAliveInterval=60'
export SSH_OPTS2='-o ServerAliveCountMax=30'
ssh ${SSH_OPTS1} ${SSH_OPTS2} -o "proxycommand ssh -W %h:%p ${SSH_OPTS1} ${SSH_OPTS2} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

Once you're in the second node, download the index-sync.sh file. Then, make it executable and run it:

wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/confluence/index-sync.sh && chmod +x index-sync.sh
./index-sync.sh 2>&1 | tee -a index-sync.log

Index synchronizing time is about 10-30 minutes. When index synchronizing is successfully completed, the following lines will be displayed in console output:

Log file: /var/atlassian/application-data/confluence/logs/atlassian-confluence.log
Index recovery is required for main index, starting now
main index recovered from shared home directory

Run toolkit with docker from the execution environment instance:

cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt confluence.yml

{{% note %}} Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above. {{% /note %}}

Run 5 (~50 min)

{{% note %}} Before scaling your DC make sure that AWS vCPU limit is not lower than needed number. Use vCPU limits calculator to see current limit. The same article has instructions on how to increase limit if needed. {{% /note %}}

To receive scalability benchmark results for four-node Confluence DC with app-specific actions:

Scale your Confluence Data Center deployment to 3 nodes as described in Run 4.
Check Index is synchronized to the new node #3 the same way as in Run 4.
Scale your Confluence Data Center deployment to 4 nodes as described in Run 4.
Check Index is synchronized to the new node #4 the same way as in Run 4.

Run toolkit with docker from the execution environment instance:

cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt confluence.yml

{{% note %}} Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above. {{% /note %}}

Generating a report for scalability scenario

To generate a scalability report:

Use SSH to connect to execution environment.
Allow current user (for execution environment default user is ubuntu) to access Docker generated reports:
```
sudo chown -R ubuntu:ubuntu /home/ubuntu/dc-app-performance-toolkit/app/results
```
Navigate to the dc-app-performance-toolkit/app/reports_generation folder.
Edit the scale_profile.yml file:
- For runName: "1 Node", in the fullPath key, insert the full path to results directory of Run 3.
- For runName: "2 Nodes", in the fullPath key, insert the full path to results directory of Run 4.
- For runName: "4 Nodes", in the fullPath key, insert the full path to results directory of Run 5.
Run the following command from the virtualenv (as described in dc-app-performance-toolkit/README.md):
```
python csv_chart_generator.py scale_profile.yml
```
In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss folder, view the .csv file (with consolidated scenario results), the .png chart file and summary report.

Analyzing report

Use scp command to copy report artifacts from execution env to local drive:

From local terminal (Git bash terminal for Windows) run command:

export EXEC_ENV_PUBLIC_IP=execution_environment_ec2_instance_public_ip
scp -r -i path_to_exec_env_pem ubuntu@$EXEC_ENV_PUBLIC_IP:/home/ubuntu/dc-app-performance-toolkit/app/results/reports ./reports

Once completed, in the ./reports folder, you will be able to review action timings on Confluence Data Center with different numbers of nodes. If you see a significant variation in any action timings between configurations, we recommend taking a look into the app implementation to understand the root cause of this delta.

{{% warning %}} After completing all your tests, delete your Confluence Data Center stacks. {{% /warning %}}

Attaching testing results to DCHELP ticket

{{% warning %}} Do not forget to attach performance testing results to your DCHELP ticket. {{% /warning %}}

Make sure you have two reports folders: one with performance profile and second with scale profile results. Each folder should have profile.csv, profile.png, profile_summary.log and profile run result archives.
Attach two reports folders to your DCHELP ticket.

Support

In case of technical questions, issues or problems with DC Apps Performance Toolkit, contact us for support in the community Slack #data-center-app-performance-toolkit channel.

Files

dc-apps-performance-toolkit-user-guide-confluence.md

Latest commit

History

dc-apps-performance-toolkit-user-guide-confluence.md

File metadata and controls

Data Center App Performance Toolkit User Guide For Confluence

Development environment

1. Setting up Confluence Data Center development environment

Using the AWS Quick Start for Confluence

AWS cost estimation for the development environment

Quick Start parameters for development environment

Running the setup wizard

2. Generate dataset for development environment

3. Run toolkit on the development environment locally

4. Develop and test app-specific action locally

Custom dataset

Example of app-specific Selenium action development with custom dataset

Example of app-specific Locust/JMeter action development

Using JMeter variables from the base script

Enterprise-scale environment

5. Setting up Confluence Data Center enterprise-scale environment

Using the AWS Quick Start for Confluence

AWS cost estimation

Stop cluster nodes

Stop database

Quick Start parameters

Running the setup wizard

6. Preloading your Confluence deployment with an enterprise-scale dataset

Importing the main dataset

Loading the dataset via populate_db.sh script (~90 min)

Restoring attachments (~3 hours)

Re-indexing Confluence Data Center (~2-3 hours)

Create Index Snapshot (~30 min)

7. Setting up an execution environment

8. Running the test scenarios from execution environment against enterprise-scale Confluence Data Center

Scenario 1: Performance regression

Run 1 (~50 min)

Run 2 (~50 min)

Generating a performance regression report

Analyzing report

Scenario 2: Scalability testing

Run 3 (~50 min)

Run 4 (~50 min)

Run 5 (~50 min)

Generating a report for scalability scenario

Analyzing report

Attaching testing results to DCHELP ticket

Support