About Microservices

Describes how to start microservices.

You can manage microservices in CentOS environments or in supported Ubuntu environments. You can also manage the microservices using the Logi Composer CLI.

See the following sections:

• Enable Microservices
• Start Microservices
• Restart Microservices
• Stop Microservices
• Disable Microservices
• Microservice Startup Order

Enable Microservices

You can also enable microservices using the CLI. See Manage Self-Service Analytics Microservices Using the Command Line Utility.

To enable all microservices, enter the following command:

sudo systemctl enable $(systemctl list-unit-files | grep zoomdata | grep edc | awk '{print $1}')

To enable a specific microservice, enter the following command:

sudo systemctl enable <service>

Replace <service> with the name of the microservice. See Self-Service Analytics Microservice Name Reference.

Start Microservices

The list of microservices can be found in Self-Service Analytics Microservice Name Reference. The order in which microservices should be started is described in Microservice Startup Order.

You can also start microservices using the CLI. See Manage Self-Service Analytics Microservices Using the Command Line Utility.

To start all microservices, enter the following command:

sudo systemctl start $(systemctl list-unit-files | grep zoomdata | grep edc | awk '{print $1}')

To start a specific microservice, enter the following command:

sudo systemctl start <service>

Replace <service> with the name of the microservice. See Self-Service Analytics Microservice Name Reference.

Restart Microservices

You can also restart microservices using the CLI. See Manage Self-Service Analytics Microservices Using the Command Line Utility.

To restart all microservices, enter the following command:

sudo systemctl restart $(systemctl list-unit-files | grep zoomdata | grep edc | awk '{print $1}')

To restart a specific microservice, enter the following command:

sudo systemctl restart <service>

Replace <service> with the name of the microservice. See Self-Service Analytics Microservice Name Reference.

Stop Microservices

You can also stop microservices using the CLI. See Manage Self-Service Analytics Microservices Using the Command Line Utility.

To stop all microservices, enter the following command:

sudo systemctl stop $(systemctl list-unit-files | grep zoomdata | awk '{print $1}')

To stop a specific microservice, enter the following command:

sudo systemctl stop <service>

Replace <service> with the name of the microservice. See Self-Service Analytics Microservice Name Reference.

Disable Microservices

You can also disable microservices using the CLI. See Manage Self-Service Analytics Microservices Using the Command Line Utility.

To disable all microservices, enter the following command:

sudo systemctl disable $(systemctl list-unit-files | grep zoomdata | awk '{print $1}')

To disable a specific microservice, enter the following command:

sudo systemctl disable <service>

Replace <service> with the name of the microservice. See Self-Service Analytics Microservice Name Reference.

Microservice Startup Order

The only microservices that must be started first are the Postgres and Service Discovery microservices. The other microservices are more tolerant and can be started in any order.

insightsoftware recommends starting microservices in the following order. You can start microservices using CentOS or Ubuntu commands or using the CLI. See Start Microservices and Manage Self-Service Analytics Microservices Using the Command Line Utility. A full list of Self-Service Analytics microservices can be found in Self-Service Analytics Microservice Name Reference.

1. The Postgres microservice (postgresql-<version>) used for metadata storage (where <version> is the version of Postgres you have installed.

2. The Service Discovery microservice (zoomdata-consul) used for microservice discovery.

3. The connector microservices (in the format zoomdata-edc-<connector_name>) used to connect to different data stores. The following CentOS example starts all available connector microservices, rather than starting them individually:

   sudo systemctl start $(systemctl list-unit-files | grep zoomdata-edc | awk ‘{print $1}‘)
   

4. The query engine microservice (zoomdata-query-engine) used for query engine processing.

5. The zoomdata-data-writer-postgresql microservice.

6. The following optional microservices, if they are installed.

   • zoomdata-admin-server
   • zoomdata-screenshot-service

7. The main Self-Service Analytics microservice (zoomdata).

Scaling Self-Service Analytics Microservices

Self-Service Analytics microservices allow you to scale your Self-Service Analytics installation, giving you expanded performance gains. As you roll out your microservices changes, you will need to:

• Estimate User and Microservice Loads
• Add or Remove Nodes in a High Availability Environment
• Configure Throughput for Microservices
• Load Monitor Microservices
• Scale Microservices Up
• Scale Microservices Down

All microservices, except composer web, support horizontal scaling. Composer web supports vertical scaling only.

Estimate User and Microservice Loads

The load used by each microservice depends on the type of interactions your users have while using Self-Service Analytics. Some actions load only the composer web component, while others may load multiple components.

Some examples of actions that load microservices are included in the table below.

Action	Microservices Loaded
Open the home page	composer web
Open a dashboard	composer web, connector, query engine
Open the visual Gallery	composer web
Create a source	composer web, connector, query engine
Create an uploaded source	composer web, connector, data writer, query engine
Upload new data via API	composer web, data writer
Execute scheduled dashboard report	composer web, connector, screenshot service, query engine
Live mode	composer web, connector, query engine

Horizontal scaling up of a microservice does not provide doubled performance gains due to sharing of PostreSQL resources and overlapping of microservices.

Start your scale planning based on an approximate number of services and expected number of users. See Server Size Guidelines for more information on services and user estimation.

Add or Remove Nodes in a High Availability Environment

To scale a microservice up or down, you may need to Add Nodes to an Existing High Availability Installation or Remove Nodes from a High Availability Environment. Before you add or remove nodes, you should understand the actual load on your microservices to make the decision of scaling up or down. See Self-Service Analytics System Metrics.

Configure Throughput for Microservices

All microservices have a configuration property, server.jetty.max-threads, you can use to limit throughput. Once configured, you can more accurately monitor the load on each microservice.

Set your estimated concurrent users for each service at +20%, based on your calculations made using Server Size Guidelines. To manage configuration properties of server.jetty.max-threads, see Configure and Start the Configuration Microservice.

For example, if you have three instances of the query engine microservice and you expect 150 concurrent users, each instance is expected to handle 50 concurrent users. To accommodate this demand, set server.jetty.max-threads to (60 = 150 / 3 & 120%) for each instance of query engine.

Load Monitor Microservices

With throughput defined for your microservices, you can measure their use to decide what services to scale up or scale down.

Self-Service Analytics microservices exposes several different metrics to help you understand the current load on each service. The main outputs to monitor include:

• jetty_threads_busy shows a current count of concurrent users.
• jetty_threads_config_max shows the maximum allowed concurrent users. This should match the value you set in server.jetty.max-threads.

These show a moment in time, which can vary depending on use spikes that may not actually require scaling up or down. Smooth out the data by looking at a specific length of time, such as five minutes, and monitor the average load across all instances of the same microservice.

Average over five minutes:

avg by instance_type (jetty_threads_busy) over 5m / avg by instance_type (jetty_threads_config_max)

In Prometheus:

sum by (job)(sum_over_time(jetty_threads_busy[5m]))
/ sum by (job)(count_over_time(jetty_threads_busy[5m]))
/ avg by (job) (jetty_threads_config_max)

Scale Microservices Up

The recommended load threshold for all microservices is 90%. After setting up your thresholds, monitor them manually, or use a metric monitoring tool such as Prometheus to define alerts at that threshold.

sum by (job)(sum_over_time(jetty_threads_busy[5m]))
/ sum by (job)(count_over_time(jetty_threads_busy[5m]))
/ avg by (job) (jetty_threads_config_max)
> 90

For more information on scaling up your environment, see Add Nodes to an Existing High Availability Installation.

Scale Microservices Down

When monitoring microservices for scaling up, you define and set alerts around defined load thresholds for types of services.

In contrast, when you monitor services to scale them down, monitor free resources, the amount of free threads dedicated to the system. The recommended threshold is dedicate two times the resources of a single instance (2 * resources).

Average over five minutes:

sum by instance_type (jetty_threads_config_max) - avg by instance_type (jetty_threads_busy) over 5m

In Prometheus:

(sum by (job) (jetty_threads_config_max) - (sum by (job)(sum_over_time(jetty_threads_busy[5m])) / sum by (job)(count_over_time(jetty_threads_busy[5m])))) / avg by (job) (jetty_threads_config_max)

Self-Service Analytics System Metrics

Self-Service Analytics exposes metrics for discovering and monitoring Self-Service Analytics microservices. Discover system metrics using one of several available tools.

Discovering Metrics

Connect to the composer-webservice actuator endpoints at: http://[host_ip]:[port]/composer/actuator.

Available endpoints include:

"prometheus": {
	"href": "http://127.0.0.1:8080/composer/actuator/prometheus",
	"templated": false
},
	"metrics-requiredMetricName": (
		"href": "http://127.0.0.1:8080/composer/actuator/metrics/{requiredMetricName)",
		"templated": true
	},
	"metrics":{
		"href": "http://127.0.0.:8080/composer/actuator/metrics",
		"templated": false
	},

Connect to all other microservices actuator endpoints at: http://[host_ip]:[port]/actuator.

Available endpoints include:

{
	"_links": {
		"self": {
			"http://127.0.0.1:8105/actuator",
			"templated": false
		},
		"metrics-sorted": (
			"href": "http://127.0.0.1:8105/actuator/metrics-sorted",
			"templated": false
		},
		"health-path": (
			"href": "http://127.0.0.1:8105/actuator/health/{*path}",
			"templated": true
		},

For a list of all metrics exposed by Composer, connect to: http://[host_ip]:[port]/actuator/metrics.

Monitoring Microservices: Prometheus

Self-Service Analytics metrics data is formatted so you can access that data through a Prometheus server. Navigate to the Prometheus composer-web actuator endpoint to see your data: http://[host_ip]:[port]/composer/actuator/prometheus.

Example Prometheus actuator response:

# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter jvm_classes_unloaded_classes_total{service_instance="Zoomdata:ip-127-0-0-0.ec2.internal:8080",} 63.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge process_files_max_files_{service_instance="Zoomdata:ip-127-0-0-0:ec2.internal:8080",} 10240.0
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to be
# TYPE jvm_gc_memory_allocated_bytes_total counter jvm_gc_memory_allocated_bytes_total{service_instance="Zoomdata:ip-127-0-0-0.ec2.internal:8080",} 4.1484812288E10
# HELP websocket_sessions_active
# TYPE websocket_sessions_active gauge websocket_sessions_active{service_instance="Zoomdata:ip-127-0-0-0.ec2.internal.8080",} 2.0
# HELP zoomdata_sessions_count_last_minute
# TYPE zoomdata_sessions_count_last_minute gauge zoomdata_sessions_count_last_minute{service_instance=Zoomdata:ip-127-0-0-0.ec2.internal:8080",} 0.0

Configure Prometheus

Install Prometheus. See https://prometheus.io/docs/prometheus/latest/getting_started/ for more information.

Build your prometheus.yml as shown in the example here to connect to the composer-webservice. Replace [username], [password] , [host_ip] and [port] with your appropriate values.

scrape_configs:
	- job_name: 'composer-web'
		metrics_path: 'composer/actuator/prometheus'
		scrape_interval: 5s
		basic_auth:
			username: [username]
			password: [password]
		static_configs:
		- targets: ['[host_ip]:[port]']

	- job_name: 'query-engine'
		metrics_path: 'actuator/prometheus'
		scrape_interval: 5s
		static_configs:
		- targets: ['[host_ip]:[port]']

Monitoring Microservices: Statsd and Graphite

You can also collect Self-Service Analytics metrics using the network daemon statsd. This network daemon runs on the Node.js platform to listen for statistics sent over UDP or TCP and sends aggregate information to one or more backend services. Unlike Prometheus, statsd does not collect metric data from specific endpoints.

Use in conjunction with Graphite to store numeric time-series data and render data graphs on demand.

Setup and Configure Statsd and Graphite

Install statsd and Graphite to connect to and access system metrics.

Example of a Graphite dashboard with metrics collected by statsd:

Manage Self-Service Analytics Microservices Using the Command Line Utility

You can use the zdmanage CLI command to manage the various microservices that are deployed in the Self-Service Analytics environment. This command line utility is automatically installed when installing your software using the automated installation script. This utility script wraps underlying UNIX commands to help you perform common management operations such as stopping and starting microservices.

Prerequisites

To use the command line utility tool, you need root access.

Using the Command Line Utility Tool

The zdmanage command line utility is located in the following directory:

/opt/zoomdata/bin/zdmanage

The zdmanage CLI command syntax is as follows:

zdmanage services <command> <service_name>

The following commands (<command>) are supported:

Command	Action
list	Displays all the microservices installed in your deployment.
status	Provides a status of all microservices in your Self-Service Analytics environment.
start	Starts microservices. The order in which microservices should be restarted is described in Microservice Startup Order.
stop	Stops microservices.
restart	Restarts microservices.
enable	Enables microservices automatically when the OS starts up.
disable	Shuts down microservices automatically when the OS starts up.
configure	Opens the specified Self-Service Analytics property file.

For <service_name>, specify the microservice name or all (to apply the command to all Self-Service Analytics microservices). For a complete list of microservices, see Self-Service Analytics Microservice Name Reference.

Common Commands

Show all Self-Service Analytics microservices installed:

sudo /opt/zoomdata/bin/zdmanage services list

Provide a status of all installed microservices:

sudo /opt/zoomdata/bin/zdmanage services status all

Show the status of the web microservice:

sudo /opt/zoomdata/bin/zdmanage services status zoomdata

Stop a specific microservice:

sudo /opt/zoomdata/bin/zdmanage services stop [zoomdata-service_name]

Manage the Self-Service Analytics Query Engine

The query engine is a stand-alone microservice within your Self-Service Analytics environment that processes visual queries. If you install or upgrade Self-Service Analytics using the supplied installation script, the query engine microservice is started automatically. If you install or upgrade Self-Service Analytics manually, you must manually enable and start the query engine microservice.

Configure the Query Engine

In your environment, the query engine is called zoomdata-query-engine. To configure and manage the microservices that the query engine executes and runs, you need to make changes to the query-engine.properties file. It contains properties that are specific to how the query engine operates within your environment.

The query engine also has a query-engine.env file and a query-engine.jvm file. You can edit these files to configure the following:

• To define the environment variables that are visible for the microservice, edit the query-engine.env file.
• To configure the JVM options that are used to start up Self-Service Analytics microservices, edit the query-engine.jvm file.

The default memory configurations give the query engine is 4 GB of heap memory. During microservice startup, the query engine consumes at least 4 GB of memory (plus some off heap memory). You can alter this using the following parameters in the query-engine.jvm file, i.e. by increasing to 6Gb.

-Xms6g
-Xmx6g

For information on editing configuration files, see Edit a Configuration File. For information about query engine properties, see Query Engine Properties.