Run Self-Service Analytics in Kubernetes

Use Kubernetes to mange your Self-Service Analytics configuration management tasks. This scalable solution allows you deploy your environment securely and scale all microservices as needed.

Kubernetes is supported for new installations only for version 26.3 and later.

See the following topics:

Helm Chart for Self-Service Analytics

Helm Chart for Self-Service Analytics

The Helm chart provided in the Self-Service Analytics Helm Chart repository is a lightweight way to configure and run Self-Service Analytics in Kubernetes. It includes sub-charts for use with 3rd-party components such as PostgreSQL, Consul , OpenTelemetery Collector, and more. This Helm chart is publicly available in the Self-Service Analytics Helm Chart repository, and includes, by default, a 14 day trial Self-Service Analytics license. Optional components are disabled by default. This topic covers:

Prerequisites
- Required Resources
Working With Helm
Configuring the Chart

Prerequisites

The following prerequisites are required for a successful deployment using Kubernetes and the Helm chart:

A Kubernetes cluster (versions 1.23-1.27)
- Install Kubernetes or have access to a cluster.
- A local copy of kubectl configured to work with your cluster.
Helm (3.8-3.11). See Installing Helm.
Access to Docker Hub.

This Helm chart was developed with portability in mind. It relies only on Vanilla Kubernetes features and doesn’t use any (cloud) vendor-specific features. Testing is performed in an Amazon EKS cluster and Minikube: you should need to make little to no modifications while working with other flavors of Kubernetes such as Azure AKS or Google GKE.

Required Resources

Self-Service Analytics is built using the microservices architecture. Since the list of required microservices (e.g. connectors) depends on your usage scenarios, it’s difficult to estimate the exact resource requirements. However, for a deployment to be useful it must include an instance of Composer Web, a Query Engine instance, a Connector instance, and a Consul service for discovery. The usage of autoscaling will obviously impact the resource requirements. Resource estimates for some common setups:

Minimal, one replica setup	Standard one replica setup (default)	Advanced, tree replica setup
Mandatory services: Web Server, Query Engine, Consul. Optional services: none. Connectors: one arbitrary connector. PostgreSQL metadata store location: external. Footprint: CPU: 4.5 vCPU RAM: 11.1 Gi Storage: 11 Gi	Mandatory services: Web Server, Query Engine, Consul. Optional services: Data Writer. Connectors: PostgreSQL. PostgreSQL metadata store location: internal. Footprint: CPU: 5.3 vCPU RAM: 12.1 Gi Storage: 19 Gi	Mandatory services: Web Server, Query Engine, Consul. Optional: Data Writer, Screenshot Service. Connectors: two arbitrary connectors. PostgreSQL metadata store location: external. Footprint: CPU: 17.7 vCPU RAM: 41.6 Gi Storage: 31 Gi

For more exact estimates, you will need to first determine the set of required components, then sum up their resource limits specified in the chart.

Working With Helm

This section explains the generic usage of Helm and Helm chart lifecycle applied to the Self-Service Analytics Helm Chart. For a generic introduction to Helm, see the Helm Quickstart Guide.

Obtain the Chart

Add the chart repository:

helm repo add composer https://composer-repo.logianalytics.com/helm-charts/stable

List the charts available:

helm repo update composer
helm search repo composer

Install with the Default Config

Run the following command to install the chart with the default config:

helm install <release-name> composer/composer

by running the first command the chart was released. Remember the release name, you will need it to interact with it later. Take a look at the output, it suggests the next steps like retrieving the address for ingress.

insightsoftware recommends you do not use the command helm install --replace.

To keep track of a release’s state, or to re-read the next steps, you can use the helm status:

helm status <release-name>

Our default Ingress resource and Ingress Controller’s configuration might not work for you. Follow this guide to re-configure our Ingress.

Customize the Chart Before Installing

Run the following command to list all the configurable parameters of the Self-Service Analytics chart and their default values:

helm show values composer/composer

Review the list of parameters to decide if anything needs to be changed. Use the guidance provided in Deciding on the Configuration to evaluate the parameters to help you decide if anything has to be changed. Once you have identified the necessary changes, follow this generic guide on applying the changes to your Helm release. Applying changes to your Helm release:

Create the values.yaml file and add YAML sections that you want to override to it. For example, in the following snippet, the PostgreSQL connector logging level is increased to DEBUG.
edc: postgresql: properties: logging.level.com.zoomdata: DEBUG
Install a new release with the override values.
helm upgrade <release-name> composer/composer -f values.yaml
Alternatively, upgrade an existing release.

Upgrade a Release and Recover on Failure

To apply a new configuration to an existing release use helm upgrade:

helm upgrade <release-name> composer/composer -f values.yaml

All affected pods will be automatically restarted.

Some updates to the Helm chart values are incompatible with helm upgrade. If you have such an incompatibility, re-install the chart.

If something went wrong during the upgrade, review the revision history of your release:

helm history <release-name>

Identify the revision you want to rollback to, then run:

helm rollback <release-name> <revision-number>

Uninstall a Release

If a particular release is no longer required, use this command to uninstall:

helm uninstall <release-name> --keep-history

Using --keep-history is optional. If you use it, you can better audit the cluster’s history, and even undelete a release using helm rollback. Running helm uninstall on a release will remove most of the Kubernetes resources from the cluster. Some resources that constitute metadata and re-usable configs are preserved. This allows you to create another Self-Service Analytics release with the same metadata like Sources, Visuals, Dashboards, etc. The preserved resources are PersistentVolumeClaims (PVC) and corresponding PersistentVolumes (PV) for:

Composer PostgreSQL metadata database folder. The PVC will have a name such as data-<release-name>-postgresql-0 and the capacity of 8Gi. It will be Bound to a PV. These resources will be present only if the release was configured to create an internal PostgreSQL instance (the default setting). If you do not intend to reuse the metadata, you can delete this PVC and PV with kubectl delete. See PostgreSQL Metadata Store Configuration for information about reusing this metadata database in a new release.
The volume with connectors’ drivers. The PVC will have the name composer-shared-volume. It will be Bound to the PV with the same name only if you configured Driver Drop-In for your connectors. If you did not configure this, it will be in the Pending state. If you do not intend to install a new release into the current namespace, you can delete the PVC. Delete the composer-shared-volume PV only if you don’t intend to install new Composer charts into this cluster.
The volume with the Consul service metadata. The PVC will have a name such as data-<namespace>-<release-name>-consul-server-0 and the capacity of 10Gi. It will be Bound to a PV. Composer can re-create the service discovery information stored in Consul upon a new release, you can safely delete these resources.

Deep Chart Inspection and Customization

Helm pull provides a way to get all source files of the chart. This is useful when you want to perform a deep inspection of the chart. Use the following commands to download the chart from the repository and unpack it in a local directory:

helm repo update composer
helm pull composer/composer --untar

Once you have the source code of the chart, you’ll be able to modify it.

Resort to this method only if there is no other way to achieve your goal via exposed configuration options.

Once you are done with the required modifications, you’ll be able to either repackage the chart into your custom chart or install it from the local directory:

helm install <release-name> . -f values.yaml

If you require such a chart modification, please submit an enhancement request for the configurability gaps identified.

Debugging the Chart

Perform a dry run before installing a release:

helm install <release-name> composer/composer -f values.yaml --debug --dry-run

Adding the --debug option causes the server to render your templates. If rendering goes fine, then the resulting manifest file is returned. If not, an error with a stack trace is returned. Attach this error output when reaching out to support. Helm also allows you to see what templates are installed on the server for a particular (successful) release:

helm get manifest <release-name>

Configuring the Chart

The Self-Service Analytics platform has lots of configuration options and some optional components. While the out-of-the-box configuration of the Helm chart provides some meaningful defaults, most likely you will need to customize some important aspects like the list of required connectors or the need for horizontal autoscaling.

The Default Configuration

The default configuration installs a release with the following components: Enabled Self-Service Analytics services:

Mandatory: Web Server, Query Engine, Consul.
Connectors: PostgreSQL.
Optional: Data Writer.
PostgreSQL metadata store location: internal.

Advanced capabilities:

Default Ingress rule: ENABLED
Tracing infrastructure: DISABLED
Horizontal autoscaling: DISABLED
Data Gateway: DISABLED

See the next section to learn how to customize it.

Deciding on the Configuration

This list covers the main decisions and recommended actions that you’ll need to perform to determine your configuration:

Do you want to use an external, managed PostgreSQL instance as the Self-Service Analytics metadata store (the recommended setup)? Alternatively, you can let the chart install an internal PostgreSQL instance (the default).
- Change the default PostgreSQL passwords if using an internal instance.
What is the list of data stores that you need to connect to? By default, only PostgreSQL connector is installed. Customize the list of connectors according to your needs.
Do you want to enable horizontal autoscaling?
Are you going to schedule Dashboard Reports that will deliver you dashboard screenshots periodically? If yes, enable the Screenshot Service component. This is also required for sending self service reports if enabled in your environment.
Will you offer self service reports and expanded export options to your users? If yes, enable self service reports and its microservice. Minimum memory request for this service is 6Gi, with a limit of 9Gi and uses 1 CPU core.
Are you going to upload flat files (e.g. CSV, JSON) for further analysis? If not, disable the Data Writer component to save cluster resources.
Do you have special requirements for the Ingress configuration? If yes, reconfigure our default Ingress.
Do you have a license that you want to apply to this deployment? If yes, inject the license during the chart installation.
How do you want to integrate your software into your observability infrastructure?

Once you decide on your target configuration, prepare corresponding override values for the Helm chart and put them into your values.yaml. Override values fall into two categories:

Self-Service Analytics application configuration properties injected into services running inside pods/containers:
- Regular application properties
- Credentials
Configuration for Kubernetes resources governed by Helm.

The next section covers Self-Service Analytics application configuration. See Scaling Configuration and Ingress Configuration to learn more broadly about Kubernetes specifics.

Injecting Self-Service Analytics Configuration Properties

There are two categories of Self-Service Analytics application configuration properties that differ in their sensitivity, hence the way they are specified in the values.yaml:

Regular application configuration properties that don’t contain sensitive data and are not exposed as separate Helm chart parameters.
Sensitive application configuration properties, such as database credentials, that are exposed as Helm chart parameters.

Regular properties for a Self-Service Analytics service are specified as the properties map within the object representing this service in the values.yaml file. Each key-value pair in this map represents a property name and value. For example, the following snippet shows a number of regular properties with names starting with mail. that specify mail server configuration for Zoomdata Web component:

zoomdataWeb:
  properties:
    mail.smtp.host: "email-smtp.us-east-1.amazonaws.com"
    mail.smtp.port: 465
    mail.smtp.auth: true
    mail.smtp.ssl.enable: true
    mail.smtp.ssl.protocols: "TLSv1.2"
    mail.from: "admin@example.com"

Regular properties go to ConfigMaps when Self-Service Analytics is installed in a Kubernetes cluster and are treated as if they are specified in regular properties files. The names of the properties should have the same names as in regular properties files (usually found under /etc/zoomdata or /opt/zoomdata/conf for Linux-based deployments). Each sensitive property for a Self-Service Analytics service is specified as a separate parameter in the object representing this service in the values.yaml l file. For example, parameters zoomdataWeb.mailLogin and zoomdataWeb.mailPassword below are sensitive properties that specify, correspondingly, login and password for the mail server configured for Zoomdata Web component:

zoomdataWeb:
  mailLogin: composer
  mailPassword: ChangeMe12345

It is not possible to override sensitive configuration properties with regular ones. For example, in the snippet below, regular properties mail.loginand mail.password will be ignored by the Zoomdata Web component:

zoomdataWeb:
  mailLogin: composer
  mailPassword: ChangeMe12345
  properties:
    mail.login: ignored
    mail.password: ignored

Sensitive properties go to Secrets when Self-Service Analytics is installed in a Kubernetes cluster. They are usually injected into corresponding services through environment variables. Use a data gateway in your environment to connect to information securely outside of your environment. Use a gateway client to authenticate your connection and make the data available to users. To enable the data gateway, add these values to the values.yaml.

# Data Gateway Service configuration
dataGatewayService:
  # Is Data Gateway Service enabled
  enabled: true

Zoomdata Web

Parameter in Values File	Description	Injected As	Corresponding Application Property
`zoomdataWeb.contextPath`	Web context path	Env variable	`server.servlet.context-path`
`zoomdataWeb.metadataDbUrl`	Metadata database URL	Env variable	`spring.datasource.url`
`zoomdataWeb.metadataDbUsername`	Metadata database username	Secret	`spring.datasource.username`
`zoomdataWeb.metadataDbPassword`	Metadata database password	Secret	`spring.datasource.password`
`zoomdataWeb.uploadDbUrl`	Upload database URL	Env variable	`upload.destination.params.jdbc_url`
`zoomdataWeb.uploadDbUsername`	Upload database username	Secret	`upload.destination.params.user_name`
`zoomdataWeb.uploadDbPassword`	Upload database password	Secret	`upload.destination.params.password`
`zoomdataWeb.keysetDbUrl`	Keyset database URL	Env variable	`keyset.destination.params.jdbc_url`
`zoomdataWeb.keysetDbUsername`	Keyset database username	Secret	`keyset.destination.params.user_name`
`zoomdataWeb.keysetDbPassword`	Keyset database password	Secret	`keyset.destination.params.password`
`zoomdataWeb.userAuditingDbUrl`	User auditing database URL	Env variable	`user-auditing.destination.params.jdbc_url`
`zoomdataWeb.userAuditingDbUsername`	User auditing database username	Secret	`user-auditing.destination.params.user_name`
`zoomdataWeb.userAuditingDbPassword`	User auditing database password	Secret	`user-auditing.destination.params.password`
`zoomdataWeb.mailLogin`	Mail server login	Secret	`mail.login`
`zoomdataWeb.mailPassword`	Mail server password	Secret	`mail.password`
`zoomdataWeb.adminPassword`	Password for the built-in `admin` user. If not set, you will be prompted to set it on the first login. Supported in v23.2 and later only. Setting this value for earlier versions will have no effect.	Secret	`admin.password`
`zoomdataWeb.supervisorPassword`	Password for the built-in `spervisor` user. Defaults to the value of `adminPassword`. Supported in v23.2 and later only. Setting this value for earlier versions will have no effect.	Secret	`supervisor.password`

Query Engine

Parameter in Values File	Description	Injected As	Corresponding Application Property
`queryEngine.dbEnabled`	Use Query Engine database for storing query results cache when `true`
`queryEngine.dbUrl`	Query Engine database URL	Env variable	`spring.qe.datasource.jdbcUrl`
`queryEngine.dbUsername`	Query Engine database username	Secret	`spring.qe.datasource.username`
`queryEngine.dbPassword`	Query Engine database password	Secret	`spring.qe.datasource.password`

JVM Properties

There are two categories of properties available for all services:

Properties to configure Self-Service Analytics services heap size.
A catch-all property that allows passing arbitrary Java options to each service.

The following Helm chart parameters are used to control heap size for Self-Service Analytics Java services:

Property in Values File	Description	Scope	Java Option
`heapSizeMin`	Initial JVM heap size.	Service	`-Xms`
`heapSizeMax`	Maximum JVM heap size.	Service	`-Xmx`

The following snippet shows the default heap size configuration for connectors:

edc:
  common:
    heapSizeMin: "256M"
    heapSizeMax: "512M"

The following snippet shows how to override the default settings for the PostgreSQL connector:

edc:
  postgresql:
    heapSizeMin: "512M"
    heapSizeMax: "1024M"

A catch-all property for passing arbitrary Java options is called additionalJavaOpts and is supported for each Composer service. For example, this is how to enable garbage collector logging for Query Engine:

queryEngine:
  additionalJavaOpts: "-verbose:gc"

Injecting Credentials

Some services might need additional credentials provided in separate files, like Java trust stores and Kerberos keytab files. To inject such credentials, you need to use additional Kerberos Secrets and Volumes. For example, let’s consider how to inject a Java trust store into the Elasticsearch 8 connector:

Create a trust store Secret:

kubectl create secret generic <secret-name> --from-file=truststore.p12

Configure the Elasticsearch 8 connector to mount this Secret as a volume and access the trust store from its file system:

edc:
  elasticsearch-8.0:
    enabled: true
    additionalJavaOpts: "-Djavax.net.ssl.trustStore=/opt/zoomdata/security/truststore.p12 -Djavax.net.ssl.truststoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=<truststore-password>"
    extraVolumeMounts:
      - name: truststore
        mountPath: /opt/zoomdata/security/truststore.p12
        subPath: truststore.p12
        readOnly: true
    extraVolumes:
      - name: truststore
        secret:
          secretName: <secret-name>

Install or upgrade the Helm chart.

Explore more topics about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

Horizontal Pod Autoscaling

Enable the Horizontal Pod Autoscaler in your environment with a new or existing instance of Prometheus and Prometheus Adapter and our custom metrics for Kubernetes to adjust the number of replicas used, scaling up or down depending on the workload. This topic covers:

Overview - About HPA
Self-Service Analytics’s HPA Metrics
HPA Configuration
Caveats of the Default Config
Cluster-Wide Prometheus and Prometheus Adapter

Overview - About HPA

The Horizontal Pod Autoscaler (HPA) is a Kubernetes feature that dynamically adjusts the number of replicas (pods) in a deployment or replica set based on the observed metrics. It helps maintain optimal resource utilization and ensures that the application can handle varying levels of workload efficiently. HPA continuously monitors the specified metrics and automatically scales the number of pods up or down to meet the desired performance targets. The Self-Service Analytics HPA implementation works by utilizing custom metrics provided by Prometheus Adapter and Prometheus. Prometheus collects application metrics from each pod. Prometheus Adapter makes these metrics available to the Horizontal Pod Autoscaler, which uses them to make scaling decisions.

Self-Service Analytics’s HPA Metrics

Three custom metrics are utilized in Self-Service Analytics environments:

cpuAverageUtilization - CPU average utilization measures the average CPU utilization of the application over a specific period. It provides insights into how much CPU capacity the application requires to handle its workload. The default value is 80%.
memoryAverageUtilization - JVM heap average utilization monitors the average utilization of Java Virtual Machine (JVM) heap memory. It measures the proportion of heap memory consumed by the application over a defined interval. HPA utilizes this metric to scale the application based on memory consumption, preventing memory-related issues and optimizing resource allocation. The default value is 90%.
threadsQueueAverageSize - The number of threads in the queue to the application reflects the number of HTTP requests waiting in the application’s processing queue. This metric provides visibility into the application’s ability to handle incoming requests promptly. HPA automatically adjusts the number of replicas to ensure sufficient processing capacity and minimize request queuing. The default value is 400 requests in the queue.

These metrics work in conjunction to ensure optimal performance and resource utilization.

HPA Configuration

HPA is disabled by default. Once you have enabled it, it’s preconfigured with reasonable defaults.

Enable HPA

Before you enable HPA, see Caveats of the Default Config.

Add the following to your values.yaml:
hpa: enabled: true
Install a new helm chart release or upgrade an existing one.
When you enable HPA, 8GB of additional storage for Prometheus is required if you install it using the Helm chart.

Global and Per-Service HPA Configuration

All Self-Service Analytics HPA configuration properties can be specified on the global and per-service levels. The service-level configuration has higher priority and overrides the global one. For example, here are the steps to change the number of replicas on the global level:

Set the value of the minimum and/or maximum number of replicas in yourvalues.yaml:

hpa:
  minReplicas: 2  # default is 1
  maxReplicas: 4  # default is 3

Install a new helm chart release or upgrade an existing one. This configuration will be applied to all Self-Service Analytics services.

The default number of maximum replicas is 3. A higher value requires a corresponding license.

HPA can also be configured on the per-service level, for example:

hpa:
  enabled: true

zoomdataWeb:
  hpa:
    maxReplicas: 5

dataWriter:
  hpa:
    enabled: false

In the above example, HPA is enabled for all services except for the Data Writer. The maximum number of replicas for Zoomdata Web is increased to 5 and defaults to 3 for all other services.

Additional Properties

The full list of configuration properties can be found in the chart values under the hpa object:

helm show values composer/composer

While enabling HPA and customizing the minimum and maximum replicas is a common practice, it’s important to exercise caution when modifying other properties. Many properties are preconfigured for optimal performance. Modifying these properties without a clear understanding of their implications may lead to unexpected issues or degraded performance.

Caveats of the Default Config

By default, when you enable HPA for Self-Service Analytics, Self-Service Analytics’s helm chart will try and install an instance of Prometheus and Prometheus Adapter into the namespace of Composer’s deployment. This is not a best practice, but rather a convenient getting-started setup for those who are new to Kubernetes and who have no other things running in their clusters. The limitation is that only one service can implement Kubernetes Custom Metrics API. In our case it’s Prometheus Adapter. By installing Prometheus Adapter from our helm chart it will come pre-configured with Self-Service Analytics-specific rules and it’ll be difficult to re-use it in other Kubernetes applications. Also, you won’t be able to install another instance of the Composer application or another Prometheus Adapter into the same cluster. The recommended approach is to have a cluster-wide Prometheus and Prometheus Adapter that can serve all the HPAs in the cluster.

Cluster-Wide Prometheus and Prometheus Adapter

Installing an instance of Prometheus and Prometheus Adapter as part of the Self-Service Analytics’s deployment is not a best practice. For a production deployment, we recommend that you deploy a cluster-wide instance of Prometheus and Prometheus Adapter and refer to them from different Composer deployments.

If you already have Prometheus and the Prometheus Adapter installed and configured you’ll only need to add Self-Service Analytics rules to your Prometheus Adapter configuration (step 3) and disable Prometheus and Prometheus Adapter installation in Composer’s helm chart (steps 4 and 5).

Installing and configuring Prometheus and Prometheus Adapter:

Create a monitoring namespace in your Kubernetes cluster:
kubectl create namespace monitoring
Install Prometheus into the monitoring namespace. We don’t require any special configurations for Prometheus, all our pods will be automatically discovered and scraped by Prometheus with the default configuration:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install prometheus prometheus-community/prometheus -n monitoring

Install the Prometheus Adapter into the monitoring namespace. The Prometheus Adapter requires two pieces of configuration:

The URL and port of the Prometheus server. If they are installed in the same namespace (for Self-Service Analytics, the monitoringnamespace), it’ll be simply http://prometheus-server:80.

The rules for converting Self-Service Analytics application metrics coming from Prometheus to custom Kubernetes metrics required by the HPA configuration. Copy the following snippet into a file called prometheus-adapter.yml:

prometheus-adapter:
  prometheus:
    url: http://prometheus-server
    port: 80
  rules:
    custom:
      - seriesQuery: '{__name__=~"jvm_memory_(used|max)_bytes",area="heap"}'
        resources:
          overrides:
            namespace: { resource: "namespace" }
            pod: { resource: "pod" }
        name:
          matches: '^jvm_memory_(used|max)_bytes$'
          as: "jvm_heap_usage_percent"
        metricsQuery: 'round(sum(jvm_memory_used_bytes{area="heap", <<.LabelMatchers>>}) by (<<.GroupBy>>) * 100 / sum(jvm_memory_max_bytes{area="heap", <<.LabelMatchers>>}) by ((<<.GroupBy>>))'
      - seriesQuery: 'system_cpu_usage'
        resources:
        overrides:
          namespace: { resource: "namespace" }
          pod: { resource: "pod" }
        name:
          matches: 'system_cpu_usage'
          as: 'cpu_usage_percent'
        metricsQuery: 'round(sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>) * 100)'
      - seriesQuery: 'jetty_threads_jobs'
        resources:
          overrides:
            namespace: { resource: "namespace" }
            pod: { resource: "pod" }
        name:
          matches: 'jetty_threads_jobs'
          as: 'jetty_threads_jobs'
        metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

Run the following commands to install the Prometheus Adapter into the monitoring namespace:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-adapter prometheus-community/prometheus-adapter -f prometheus-adapter.yml -n monitoring

Update the Self-Service Analytics helm chart configuration file values.yaml:

# Make sure HPA is enabled
hpa:
  enabled: true

# Disable installation of Prometheus
prometheus:
  enabled: false

# Disable installation of Prometheus Adapter
prometheus-adapter:
  enabled: false

Install a new helm chart release or upgrade an existing one.

Find more information about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

Scaling Configuration

When deployed in Kubernetes, Self-Service Analytics supports both vertical (adding resources) and horizontal (adding pods) scaling. This topic covers:

Horizontal Autoscaling
Manual Scaling
- Manual Horizontal Scaling
- Manual Vertical Scaling

Horizontal Autoscaling

By default, the Helm chart comes with horizontal autoscaling disabled. Follow this guide to learn more about Horizontal Pod Autoscaling and how to enable it.

Manual Scaling

If you decide not to enable autoscaling you can still scale the resources/pods manually.

Manual Horizontal Scaling

To set the number of replicas for a Self-Service Analytics service such as zoomdataWeb do the following:

Add the desired number of replicas to the values.yaml:
zoomdataWeb: replicaCount: 2 # default is 1
Install a new helm chart release or upgrade an existing one.
Don’t use kubectl to change the number of replicas for Self-Service Analytics pods. These changes will be overwritten on the next usage of Helm.

Manual Vertical Scaling

Configure each pod’s resource manually in the corresponding block of the values.yaml file, but it’s always recommended to prefer horizontal over vertical scaling. Improper change of the resources may lead to application failure on startup or instability during the operation. Since Self-Service Analytics consists of a number of Java-based microservices, apart from generic pod resource types (i.e. CPU and memory), Self-Service Analytics services also expose properties for configuring application JVM memory: heapSizeMin and heapSizeMax. For example, here are the resource-related defaults for the zoomdataWebservice:

zoomdataWeb: # Initial JVM heap size heapSizeMin: "4G" # Maximum JVM heap size heapSizeMax: "4G" resources: limits: memory: "6Gi" requests: cpu: "3.5" memory: "6Gi"

If you decide to customize these properties, please follow these general recommendations:

Memory request and limit should be the same.
Memory request should be at least 25% bigger than JVM’s maximum heap size to account for non-heap memory consumed by Java applications.
CPU limit is empty.

Find more information about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

Ingress Configuration

By default, the chart creates an Ingress resource with the name default-ingress. Also, it relies on the ingress-nginx sub-chart to install an Ingress Controller with the default name nginx. This might not work with your existing Ingress Controller. To resolve this issue, override the config for the Self-Service Analytics Ingress in your values.yaml:

ingress:
  enabled: true
  installController: false
  className: <your-ingress-controller-name>
  ingressName: default-ingress

This config disables installation of the nginx Ingress Controller and points Self-Service Analytics’s Ingress to your controller. If such a config is still not versatile enough for your needs, disable both the Ingress and Ingress Controller and create your own Ingress.

Find more information about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

Data Writer Configuration

The Data Writer component is an optional component that is enabled by default. It’s required by the following Self-Service Analytics features:

Uploading flat files (e.g. CSV, JSON) for further analysis.
Landing streaming data via Upload API.
Performing multipass and multisource filtering of your data with the help of Keysets.
Tracking end users’ access to data via User Auditing.

If none of these features are required, add the following configuration to your values.yaml to disable it:

dataWriter:
  enabled: false

Then install a new helm chart release or upgrade an existing one.

Find more information about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

Screenshot Service Configuration

The Screenshot Service component is an optional component, disabled by default. Enable if you are going to schedule Dashboard Reports. To enable, add the following configuration to your values.yaml:

screenshotService:
  enabled: true

Find more information about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

Apply Licenses

By default, Self-Service Analytics comes with a trial license valid for 14 days. The first start up of activates this license and generates a unique installation id for your deployment. Once you are done with the trial, you’ll need to request a new license key using the generated installation id and then apply the key to your deployment. There are multiple ways to apply a license key: via UI, using the licensing API, or by setting Helm override values. This article describes setting Helm override values.

Replace an Existing License
Apply an OEM License

Caveats of Different License Configuration Options

If you apply your license using Self-Service Analytics’s user interface or the licensing API, you can only scale the Web Server service to more than three replicas, if allowed by the license’s restrictions. The replica count for all other services is limited to three replicas. If you apply your license using Helm values, you should use Helm to update the license; it will not be possible to update the license via UI or API. See Manage License Keys Using Configuration Properties or Environment Variables.

Replace an Existing License

Obtain a new license key for your existing deployment.

Add the following properties to your values.yaml:

licenseKey: "your-license-key"
licenseInstallationId: "your-installation-id"

Upgrade your existing Helm release. The license is applied to all Self-Service Analytics services.

Apply an OEM License

Regular license keys are tied to the installation id that is unique for your deployment. The ability to have an arbitrary number of deployments can be provided by an OEM license. An OEM license allows you to have an unattended installation because you won’t need to request a license key for each unique installation id. If you have an OEM license key, set only the licenseKey property in your values.yaml. Install a new Helm chart release or upgrade an existing one.

Find more information about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

PostgreSQL Metadata Store Configuration

Internal PostgreSQL Metadata Store
- Configuring Credentials
- Reusing Existing Internal Metadata Database for New Release
External PostgreSQL Metadata Store

Self-Service Analytics stores its metadata in a PostgreSQL database. The Helm chart lets you customize the location of the metadata store database. You have two options:

Default: The chart installs an internal instance of PostgreSQL.
Recommended: Point Self-Service Analytics to an external, managed instance of PostgreSQL.

Internal PostgreSQL Metadata Store

Configuring Credentials

The default chart config is supplied with the hardcoded database username and password to simplify the initial installation of the chart:

defaultDbUsername: "zoomdata"
defaultDbPassword: "ChangeMe12345"

Change the username and password for your production release if you choose to use the internal PostgreSQL instance.

Reusing Existing Internal Metadata Database for New Release

The usage of an internal instance of PostgreSQL will create a PersistentVolumeClaim (PVC) and corresponding PervistentVolume (PV) with the PostgreSQL data dir folder. Uninstalling such a chart will keep these PVC and PV so that you can re-use the metadata for another release. The usual name of the PVC is a name such as data-<release-name>-postgresql-0. To reuse the metadata:

Add the following config to your values.yaml to point a new release to the existing PVC:

postgresql:
  primary:
    persistence:
      existingClaim=data-<release-name>-postgresql-0

Install or upgrade the Helm chart.

External PostgreSQL Metadata Store

To configure the chart to use an external PostgreSQL-compatible database system:

Disable the creation of the built-in PostgreSQL instance by adding this config to your values.yaml.
postgresql: enabled: false
Create the next databases in your database system.
zoomdata zoomdata-upload zoomdata-keyset zoomdata-user-auditing zoomdata-qe
We recommend that you also create a separate PostgreSQL user for Self-Service Analytics databases. See Create the Metadata Store User & Stores.

Define the following Helm chart parameters in your values.yaml and set them according to your database setup.

zoomdataWeb:
  metadataDbUrl: ""
  metadataDbUsername: ""
  metadataDbPassword: ""
  uploadDbUrl: ""
  uploadDbUsername: ""
  uploadDbPassword: ""
  keysetDbUrl: ""
  keysetDbUsername: ""
  keysetDbPassword: ""
  userAuditingDbUrl: ""
  userAuditingDbUsername: ""
  userAuditingDbPassword: ""

queryEngine:
  dbUrl: ""
  dbUsername: ""
  dbPassword: ""

If you only define a single user, then you can skip individual *DbUsername and *DbPassword parameters and set only the following two defaults.

defaultDbUsername: ""
defaultDbPassword: ""

Install or upgrade the Helm chart.

Find more information about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

Self Service Reports Configuration

Self service reports are an optional feature you can offer your users that is disabled by default. You can enable it during installation, or post installation by enabling the appropriate server-level variable. You must enable this service to offer the following features:

Self service report creation.
Expanded export options that include conditional formatting and content formatting for reports and table visuals.

To enable, add the following configuration to your values.yaml to disable it:

report-service:
  enabled: true

Then install a new helm chart release or upgrade an existing one.

Find more information about Self-Service Analytics and Kubernetes here: Run Self-Service Analytics in Kubernetes.

​Helm Chart for Self-Service Analytics

​Prerequisites

​Required Resources

​Working With Helm

​Obtain the Chart

​Install with the Default Config

​Customize the Chart Before Installing

​Upgrade a Release and Recover on Failure

​Uninstall a Release

​Deep Chart Inspection and Customization

​Debugging the Chart

​Configuring the Chart

​The Default Configuration

​Deciding on the Configuration

​Injecting Self-Service Analytics Configuration Properties

Zoomdata Web

Query Engine

​JVM Properties

​Injecting Credentials

​Horizontal Pod Autoscaling

​Overview - About HPA

​Self-Service Analytics’s HPA Metrics

​HPA Configuration

​Enable HPA

​Global and Per-Service HPA Configuration

​Additional Properties

​Caveats of the Default Config

​Cluster-Wide Prometheus and Prometheus Adapter

​Scaling Configuration

​Horizontal Autoscaling

​Manual Scaling

​Manual Horizontal Scaling

​Manual Vertical Scaling

​Ingress Configuration

​Data Writer Configuration

​Screenshot Service Configuration

​Apply Licenses

​Caveats of Different License Configuration Options

​Replace an Existing License

​Apply an OEM License

​PostgreSQL Metadata Store Configuration

​Internal PostgreSQL Metadata Store

​Configuring Credentials

​Reusing Existing Internal Metadata Database for New Release

​External PostgreSQL Metadata Store

​Self Service Reports Configuration

Helm Chart for Self-Service Analytics

Prerequisites

Required Resources

Working With Helm

Obtain the Chart

Install with the Default Config

Customize the Chart Before Installing

Upgrade a Release and Recover on Failure

Uninstall a Release

Deep Chart Inspection and Customization

Debugging the Chart

Configuring the Chart

The Default Configuration

Deciding on the Configuration

Injecting Self-Service Analytics Configuration Properties

JVM Properties

Injecting Credentials

Horizontal Pod Autoscaling

Overview - About HPA

Self-Service Analytics’s HPA Metrics

HPA Configuration

Enable HPA

Global and Per-Service HPA Configuration

Additional Properties

Caveats of the Default Config

Cluster-Wide Prometheus and Prometheus Adapter

Scaling Configuration

Horizontal Autoscaling

Manual Scaling

Manual Horizontal Scaling

Manual Vertical Scaling

Ingress Configuration

Data Writer Configuration

Screenshot Service Configuration

Apply Licenses

Caveats of Different License Configuration Options

Replace an Existing License

Apply an OEM License

PostgreSQL Metadata Store Configuration

Internal PostgreSQL Metadata Store

Configuring Credentials

Reusing Existing Internal Metadata Database for New Release

External PostgreSQL Metadata Store

Self Service Reports Configuration