Skip to main content
The Self-Service Analytics platform is architected as a set of loosely-coupled Java microservices. Unlike traditional business intelligence software, which is deployed as a monolithic application, a microservices architecture allows for the following benefits:
  • Faster deployment of new functionality
  • Optimal resource management
  • Faster recovery in some failure scenarios
  • Greater deployment flexibility
Each microservice runs in its own Java Virtual Machine (JVM) to optimize memory allocation and runtime attributes to meet its function and load. A failure in one microservice does not take down all microservices, and recovery is faster because only the failed microservice needs to restart. Deploying new functionality, such as an updated or custom data store connector, occurs by auto-discovery, and without requiring a server restart. Separately, Self-Service Analytics centralizes its metadata store in a relational database and includes an internal messaging queue. The communication protocols used by the microservices include WebSockets (for real-time bidirectional communication) and HTTP/S. For more information about individual microservices, see: The following links provide general information about all microservices:

​
Configuration Microservice

The Self-Service Analytics configuration microservice is packaged with the Spring Cloud Configuration server, which allows Self-Service Analytics to easily integrate with its Spring-based microservices. It provides the mechanism by which property settings can be maintained using the Service Monitor. The property settings are persisted in a supported PostgreSQL data store or in a GitHub repository. A config-server-upload.jar utility is available that can be used to migrate the microservice properties from your standalone Self-Service Analytics servers to the Self-Service Analytics configuration data in the PostgreSQL data store, where the configuration microservice can maintain them. See Migrate Properties to the Configuration Server. For complete information, see Maintain Application Properties.

​
Data Connector Microservices

Self-Service Analytics offers the widest set of connectors for modern data stores deployed on-premises or in the cloud, such as Hadoop, NoSQL, search engines, and modern data warehouses. In addition to the set of standard, out-of-the-box, data connectors that we provide, you can create custom data connectors. Self-Service Analytics’s standard, out-of-the-box, data connectors are smart, leveraging the unique capabilities of each data platform to take advantage of query expressiveness and to optimize performance. For example, the data connectors leverage:
  • Data partitions in Impala and other similar sources
  • Faceted search when querying an unstructured data search engine such as Elasticsearch or Solr
  • Native APIs for NoSQL databases
The data connectors use native APIs either in whole or in part to interact with the target data store. For some modern data platforms, all user interactions are converted to native API calls. For data stores that can be queried using SQL over a JDBC driver, the data connector uses SQL when available and native API calls when necessary. For example, a data store may support query cancellation, but its JDBC driver may not. Self-Service Analytics data connectors are smart enough to enrich the data interaction experience beyond standard off-the-shelf functionality. Self-Service Analytics’s microservices-based architecture allows your developers to build connectivity to proprietary data platforms or to data for which a native connector is not currently available. Custom connectors can also be smart by making full use of the Self-Service Analytics query engine, including its Data Sharpening and live mode functionality as well as enabling custom security features such as user delegation and Kerberos authentication. Custom connectors can also be used to extend Self-Service Analytics data connectors with additional business rules. For example, a custom connector can implement the logic necessary to query specific tables based on the end user’s group membership. Each Self-Service Analytics data connector deploys as a standalone Thrift server running in its own Java processing space and registers with Consul and Self-Service Analytics’s Service Monitor as a Self-Service Analytics microservice. For complete information about Self-Service Analytics data connectors, see Connect Self-Service Analytics to Data Stores.

​
Service Discovery Microservice

The Service Discovery microservice integrates Self-Service Analytics with the Spring Cloud Consul. It provides:
  • External configuration capabilities for Self-Service Analytics via a key/value store.
  • A service discovery client backed by the Consul service registry.

​
Data Writer Microservice

Self-Service Analytics offers a multipurpose Data Writer microservice that writes data to a relational database for an enriched analytic experience. Its current uses are to: The Data Writer microservice consists of a microservice that receives requests using a REST API, a message queue, and writable data connectors.

​
User Uploaded Files

Users with privileges to create Self-Service Analytics data sources can use the user interface to upload flat files to your environment. The Data Writer microservice passes the text or CSV file to a message queue to manage backflow, and then writes the file contents to a database. Users then work with it as any other data source. Additional APIs are available to manage uploaded files.

​
Landing Streaming Data

Any streaming engine, such as Kafka, Spark Streaming, Storm, Apex, Nifi, Kinesis, and others, can be used to process and land data in a persistent data storage environment. Alternatively, the Data Writer microservice can receive live streaming data via the REST API, pass it to an internal messaging queue for backflow management, and then write it to a database. After it is landed, data is accessible as any other data source for seamless live mode and historical analysis. insightsoftware took this approach because it provides greater functionality than connecting directly to a live stream, and requires far less administrative overhead and maintenance than a complex lambda architecture.

​
Keysets: User-Directed Set Analysis

Self-Service Analytics offers an elegant approach to “set analysis” that allows users to explore key relationships within and between data, regardless of where the data is stored. The Data Writer microservice provides the backbone for this functionality by receiving an ordered and filtered set of “keys” from the web application, passing the keyset through the message queue, and storing it in a relational database for reuse by authorized persons. Users work independently to apply keysets to data stored on any platform. The user experience is swift and fluid, and the supporting architecture is so elegant and straightforward that no SQL or coding is ever required.

​
Query Engine Microservice

The Self-Service Analytics query engine sits between the web application and the Self-Service Analytics data connectors. The query engine has three primary roles:
  1. It deconstructs and converts your query requests into distributed execution plans.
  2. It optimizes the execution plans based on data platform capabilities, in-memory cached results, and the query engine capabilities.
  3. It executes data functions that include:
    • Communicating with Self-Service Analytics data connectors to execute push-down queries
    • Retrieving data from in-memory cached results, as appropriate
    • Using in-memory processing to combine, append, or manipulate one or more data sets to produce only the values needed to fulfill your request.
The key capabilities of the query engine include push-down processing for select data sources, microqueries and Data Sharpening, adaptive caching, data playback and live mode, and multisource analysis.

​
Push-Down Processing

Self-Service Analytics is built so users can interact directly with data down to the atomic row-level detail. Push-down processing is necessary to support this ad-hoc, interactive user experience on fresh data. As users explore the data and drill down to lower levels of detail, Self-Service Analytics continues to push processing down as new queries. The data store returns only the values that the query engine needs to populate the user’s visuals. The Self-Service Analytics push-down architecture also avoids scaling up the query engine unnecessarily when complex processing can be better executed on high-performance database engines or scalable data platforms. Self-Service Analytics’s default processing strategy is to push down as much work to the underlying data sources as possible, for as many data sources as possible. Internal to the query engine is a query optimizer that evaluates each end-user request, and determines whether to submit all or part of the request to the target data stores. This includes pushing down filtering criteria, derived fields, custom metrics, and offset, limit, sort, and time bucketing operations. The ability to push down filters means that the data platform engine doesn’t need to scan large data sets unnecessarily. It also reduces the amount of data transferred over the network from the data source to Self-Service Analytics. Self-Service Analytics can push down all filters that a user requests in the UI. Push-down of derived fields and custom metrics optimizes performance for the most resource-intensive operations. Self-Service Analytics always pushes down custom metric aggregations: min, max, sum, avg, count, distinct count, last value, and percentiles. Where advantageous, the query engine combines several simpler aggregates to compute more complex metrics. Self-Service Analytics offers automatic time bucketing, which allows you to group and filter data by time categories such as current or prior week, month and year, rolling time periods, and so on. There is no need to pre-aggregate or model time buckets, freeing up technical personnel to work on other work. All that is needed is a date-time field. The query engine does all the work of interpreting and converting user requests to one or more queries and pushing the whole operation to the data store. The benefits of the Self-Service Analytics live-connect approach with push-down processing are:
  • You always have access to fresh data from the data store
  • Computational resources are scaled and managed where they make the most sense
  • Network bandwidth is conserved
  • It works very well for hybrid-cloud deployments, since there’s no need for massive data movement between systems
Self-Service Analytics’s implementation of push-down processing also allows users to explore down to the atomic, row level detail. In this way, Self-Service Analytics is unique in its ability to make the full breadth and depth of big data environments available for exploration.

​
Microqueries and Data Sharpening

Microqueries and Data Sharpening™ are patented technologies that work together to let you interact with big data. The query engine invokes microqueries and Data Sharpening based on criteria such as the type of aggregate values requested and anticipated query run time. Microqueries and Data Sharpening are ideal for big data that is partitioned by date and that runs on a cluster with many processing cores. This functionality can be disabled when you set up the Self-Service Analytics data source configurations for your data store. Microqueries, which are executed first, sample data across database partitions. The query engine submits a full long-running query that runs with the first set of microqueries. A progress indicator estimates the progress of the full long-running query. Both the full query and the microqueries run until the full query runs to completion or the user changes direction, at which point both the long-running query and microqueries are canceled to conserve processing and network resources. Data Sharpening analyzes the sample data and streams predictive results to the your browser (or other client) over a WebSocket connection. Data Sharpening’s predictive results may fluctuate a bit up or down until the final query is reported. However, the relative values of each group usually remain consistent as the data is sharpened. For example, the tallest bar in a bar chart at 10% completion will almost always remain the tallest bar at 100% completion. This means that you can be confident exploring data even as it streams live to the dashboard. When you drill down, filter, change metrics or groupings, or perform any other action that changes data values, Self-Service Analytics cancels the full long-running query and microqueries to free up the data source engine for the next sequence of queries. Canceling active queries, however, is not trivial, and many JDBC drivers do not support it. In these cases, Self-Service Analytics’s data connectors issue native API calls to complete the task.

​
Adaptive Caching

The query engine optionally accelerates performance through adaptive caching. The cache eviction algorithm prefers the most frequently used data, with consideration of cache size and expiration times. Users with the appropriate privileges can configure cache sizes and time-to-live (TTL) schedules, and can forcibly clear caches. Self-Service Analytics uses its cache to enhance performance in scenarios where large numbers of users are concurrently viewing the same shared visuals.
Cached data is shared between users only if they have the same data access permissions and security context.
When caching is enabled, Self-Service Analytics does not requery the data source to obtain the data unless the cache is cleared or unless a refresh schedule is defined in the data source configuration. See Cache Tab and Trigger Refresh Jobs. By default, data caching is enabled for all data sources. The Self-Service Analytics cache stores all the results of aggregated requests from your data source. When a visual is created the request is first sent to the cache. If the required results are found in the cache, they are visualized on your visual. Each visual that is cached has a key that uniquely identifies its content and how it can be reused. The key is calculated based on the request and user. The request provides the information related to data grouping and content, such as the data attributes and metrics. The user provides contextual information, like per-user security filters, user attributes, etc. This way all the security settings are taken into account when storing and using the cached visual. Information from the cached visual is shared between users only when they have the same data access permissions and security context. The data cache optimizes data processing and avoids unnecessarily expensive queries. To avoid an expensive query, all or parts of multiple interactive data caches can be used to fulfill different user requests. The cache uses normalized requests for its cache entry key. This key allows data caches to be safely used by multiple users with different security contexts and visual types. To serve requests that require different data, the query engine applies a number of possible transformations to the cached data. For example, columns can be dropped to enforce field-based security, additional filters can be applied to enforce row-level security, time windows can shift, or the data can be rolled up to higher levels of aggregation. Because there may be several ways to get the same result using different cache entries, the query engine evaluates different transformation approaches for complexity, and selects the simplest approach. Not all data requests are cached. By default, cache entries are stored in the metadata repository, where they are retained after a service restart.

​
Screenshot Microservice

The Screenshot microservice allows you to view snapshots of your saved dashboards. See Screenshot Microservice.

​
Service Monitor Microservice

The Service Monitor microservice provides administrators with real-time views of microservice health, configuration, and logs. The Service Monitor is not installed as part of a default Self-Service Analytics installation, so you must install and configure this before you can use it. Using the Service Monitor, you can:
  • Review all Self-Service Analytics microservices and their log files.
  • Produce a diagnostics bundle to send to insightsoftware Support.
  • Set the logging level and properties for each microservice.
For complete information, see Screenshot Microservice.

​
Web-Based User Interface

Self-Service Analytics provides a single, unified web-based user interface for administrators, dashboard developers, analysts and users who consume the information provided in your visuals, dashboards, and self service reports. The web application is compatible with modern browsers that support the HTML5 standard for web applications and WebSocket communication protocol. Because it supports the HTML5 standard, the user interface responsively adjusts to a tablet form factor without loss of functionality. The application will function on a smartphone, however due to the smaller form factor, such devices do not lend themselves well to the out-of-the-box exploratory experience for which Self-Service Analytics was developed. Custom mobile apps can be developed using mobile frameworks such as Ionic or React Native. The web browser (or any client application using the Self-Service Analytics JavaScript APIs) is responsible for establishing a two-way WebSocket communication channel between itself and the Self-Service Analytics query engine. The web browser is responsible for keeping the WebSocket connection alive and automatically reconnecting to the query engine when network problems occur. Messages transmit over WebSockets using the lightweight JSON (JavaScript object notation) file format. There are several message types used by Self-Service Analytics. The Self-Service Analytics web application delivers unique functionality to the end-user, such as:
  • Data Sharpening™ to stream predictive results when working with very large data sets
  • Smart loading of visuals and dashboards. Smart loading improves the performance for loading visuals on a dashboard.
  • A time bar to simplify time-based analysis
  • Data DVR to rewind, play, and fast forward data streams
  • Live mode to visualize near real-time data streams
  • Highly configurable visuals and dashboards that includes filtering and sorting the data in near real-time
  • Search box and facets to maximize performance when connecting to search-enabled data platforms such as Apache Solr and Elasticsearch.
The web application pulls all these technologies and features together in a “non-blocking” data exploration experience. There are two ways to understand the non-blocking UI. One is that data loads independently in each visual, so viewing the entire dashboard is not slowed down by the longest running query. The second is that at the same time that data rolls in, users can filter, sort, drill down, reformat, change visual type (re-visualize), and otherwise manipulate visuals. The ability to interact with visuals while data loads is significant because it is what enables users to engage in speed-of-thought analysis against massive and live data sources. When a user switches direction to explore different data, Self-Service Analytics efficiently cancels previously running queries to conserve computing and network resources. The value of the Self-Service Analytics non-blocking user experience should not be underestimated. Because many studies suggest that the human attention span for most tasks is less than 10 seconds, it’s important that your users are not stuck staring at a blank screen or a waiting hourglass. True data exploration only happens when users can rapidly and freely decide to explore where the data leads.

​
Self Service Report Microservice

The self service report microservice included in your environment, when enabled, supports a number of self-service analytics capabilities. To support these features, keep some performance details in mind.
  • Self Service Reports: Users with appropriate permissions can create, edit, and send self service reports. If enabled, users can schedule and send reports to other users, or add them to a designated SFTP location.
    • Generate, export, and schedule delivery of PDF reports in a variety of page size and orientation formats.
    • Generate, export, and schedule delivery of reports in Excel (XLSX) format. These reports can include and support formatting and conditional formatting.
  • Scheduled Reports: Limit each scheduled report to fewer than 10 users, and plan time gaps between consecutive runs.
  • Table visuals: Users with appropriate permissions can export table visuals in Excel (XLSX) format. These visuals can include and support formatting and conditional formatting.

​
Environment Configuration

Before you enable the self service report microservice in production, work with your infrastructure team to provision appropriate resources. The microservice requires dedicated memory allocation and can scale with additional pods to accommodate concurrent user load and report complexity. The name of this service, when installing from bootstrap, is report-service.

​
Performance Considerations

Report generation and export performance varies significantly based on report complexity and export format. The following guidelines are provided to help you plan your deployment and use. Performance expectations noted here are based on internal testing we performed across three report service pods. Your results may vary based on resource allocation and concurrent user load. In general, test your typical use report configurations thoroughly before deploying at scale.

​
Report Type Performance

These guidelines are based on the report service running across three pods, with a memory request of 6 Gi and limit of 9 Gi, on one CPU core. Structure your environment to accommodate your reporting needs.
  • Simple grouped reports are the best performers across both PDF and Excel (XLSX) formats.
  • Simple reports perform well in PDF format; response times increase substantially with more complex datasets. XLSX is more resilient than PDF for simple reports.
  • Keep your header and footer image sized between 200kb and 500kb for optimal rendering performance.
  • The maximum generated report file size is 50 MB. Validate with test runs before you put large conditionally-formatted reports into production.
  • Structured reports are the most resource-intensive type. Excel (XLSX) exports for structured reports average a few minutes and are not recommended for high-throughput or user-facing scenarios.

​
Conditional Formatting

Conditional formatting increases report generation time significantly. A report with conditional formatting can take from twice as long to significantly longer than an equivalent report without it. Conditional formatting in complex reports exported to PDF format place a higher load on your environment, resulting in a percentage of error rates.

​
File Size and Media

Keep in mind when designing your reports the limitations we built in to optimize performance, and guidelines users should follow for designing and formatting their reports.
  • Generated report file size is capped at 50 MB.
  • Keep header and footer images in the 200 KB–500 KB range.

​
Landscape Orientation Considerations

  • Reports generated in landscape orientation accommodate more columns per page. This requires additional write operations and memory allocation when compared to portrait, and can result in measurable performance impact.
  • Due to the increased demands of landscape oriented reports, test landscape configurations in your environment before you deploy them to production.
  • Landscape orientation combined with conditional formatting brings the highest load. Explore alternative configurations or expand the resources available to the self service report microservice.

​
Scheduling and Distribution

When you enable scheduled reports for your users, keep these operational limits in mind.
  • Scheduled reports should be limited to a maximum of 10 recipients per report. Leave adequate time gaps between consecutive scheduled runs.
  • Stagger report schedules to avoid concentration of export operations during the same time window.

​
Developer Resources

The following topics provide information about the developer resources available in Self-Service Analytics.

​
Self-Service Analytics Microservice Name Reference

The following table lists the Self-Service Analytics microservices. The order in which microservices should be started is described in Microservice Startup Order.

​
Self-Service Analytics Metadata Stores

Self-Service Analytics uses the following metadata stores:

​
Self-Service Analytics Administration

Self-Service Analytics makes visual analytics easy for end users. Self-Service Analytics administrators are a big part of that. This docs portal will help you install, configure, maintain, and upgrade Self-Service Analytics.

​
How Do I Administer Self-Service Analytics?

Self-Service Analytics is much like any installed software in that the administrator manages the software’s life-cycle. Some tasks need to be done only when you start out or want to upgrade Self-Service Analytics.
Self-Service Analytics uses a packaged PostgreSQL database instance to store its metadata. Use the provided instance due to the specific configuration and version combination:
  • Self-Service Analytics 26.3 and later: PostgreSQL 16
If you would like to use another PostgreSQL instance, contact Technical Support for further guidance.
New installations of Self-Service Analytics use PostgreSQL 16. If you are upgrading your environment to Self-Service Analytics, you can retain your existing PostgreSQL version.
You do other tasks only as needed. You will need to do some maintenance tasks at regular intervals.

​
Reference Information

The following topics provide reference material useful for managing and using Self-Service Analytics.