zoomdata-activity.log. By default, logging to this file is enabled. Some activity types are not automatically logged. To determine which activities are logged by default, see the Activities Log Reference Sheet.
This page covers the following topics:
- Configure the Activity Log
- Enable or Disable the Logging of Specific Activities
- Determine Whether an Activity Is Being Logged
- Example of Self-Service Analytics Activity Logging Monitored by Fluentd
Configure the Activity Log
The activity log defaults can be configured through properties set in thezoomdata.properties file. See Configure Self-Service Analytics for more information about modifying property files.
| Property | Description | Default |
|---|---|---|
| log.file.count | Number of log files to keep | 1 |
| activity.log.file.size | Maximum log file size in Mb | 10 |
| activity.logs.dir | File path to store log files. Verify that this log directory has all the necessary permissions and that the owner of the directory is set to zoomdata.The /home directory cannot be used for logging. | Linux: /opt/zoomdata/logs/ Windows: <install-path>/logs/ |
Enable or Disable the Logging of Specific Activities
Enabling or disabling the logging of specific activities is performed via a series of REST API calls or by directly updating the appropriate activity property in thezoomdata.properties file. For a list of the activities that can be logged and the file in which they can be logged, see the Activities Log Reference Sheet.
To enable or disable logging for a specific activity using the REST API, run the following cURL command:
<host>:<port>- Specify the host IP address and port of the Self-Service Analytics instance.<activityType>- Specify the name of selected activity (for example,AUTHENTICATIONorUSER).<option>- Specify eithertrueorfalse. Set the value totrueto enable the selected activity output orfalseto disable it.
-
Locate the
zoomdata.propertiesfile. For information about accessing Self-Service Analytics configuration (properties) files, see Configure Self-Service Analytics. -
Add or edit the appropriate activity property in the file using the following format:
Activity names (
<activity-name>) are listed in the Activities Log Reference Sheet. For<option>, specify eithertrueorfalse. Set the value totrueto enable the selected activity output orfalseto disable it. - Save the properties file.
- Restart Self-Service Analytics microservices. See Restart Microservices.
Determine Whether an Activity Is Being Logged
To determine whether a specific activity type is being logged in the log file, run the following cURL command:<host>:<port>- Specify the address of the instance on which Self-Service Analytics is installed.<activityType>- Specify the type of selected activity (for example,AUTHENTICATIONorUSER). For a list of the activities that can be logged, see the Activities Log Reference Sheet.
Example of Self-Service Analytics Activity Logging Monitored by Fluentd
This example shows how to set up Fluentd to monitor Self-Service Analytics activity log files. It does not show how to connect your Fluentd service to an external data store or how to pipe the log information to the external data store. Before you use this example, make sure Fluentd is installed. Next, create a log file and grant thetd-agent service permissions to write to it. Finally, modify the td-agent.conf file.
Linux
Create the file/opt/zoomdata/logs/zoomdata-activity.log.pos and run the following command to grant the td-agent service permissions to write to it:
td-agent.conf file in the /etc/td-agent folder using the template below.
Windows
Create the file<install-path>/logs/zoomdata-activity.log.pos. Grant the td-agent service permissions to read, write and execute permission to the owner, group and public.
Finally, modify the td-agent.conf file in the /etc/td-agent folder using the template below. Replace pattern /zoomdata/service/health with <install-path>/service/health.
Template
- Substitute Self-Service Analytics fully qualified log file name for
<logfile>and<logfilen>(for example,opt/zoomdata/logs/zoomdata-activity.log) for Linux, or<install-path>/logs/zoomdata-activity.logfor Windows. Self-Service Analytics log files names are provided in Self-Service Analytics Log Files Reference. - For each log file you select, specify rules in
<rule>sections for the information you want to extract from the Self-Service Analytics log files.
Manage Activity Logs
The Self-Service Analytics server records user- and server-based activities in log files. These files can be used by administrators to troubleshoot issues that may occur with the Self-Service Analytics server, for example, activities related to setting up data sources or creating visuals and dashboards. When Self-Service Analytics is installed in your network environment, a log file is created. By default, activity logging is enabled, but some activities are not logged by default. For details, see the Activities Log Reference Sheet. You can enable or disable activity logging by calling REST API endpoints. See Activity Logging. A list of the types of activities that are logged can be found in the Activities Log Reference Sheet. For more information about other Self-Service Analytics log files, see Self-Service Analytics Log Files Reference.User Auditing
User auditing allows you as a Self-Service Analytics administrator or client application administrator to track your users’ access and actions to data considered sensitive. This includes tracking:- What period of time any user interacted with data from dashboards, visuals, sources, and custom SQL queries.
- How the data was accessed: directly in the Self-Service Analytics user interface, via an API, or as embedded objects in your applications.
- Actions performed with the data.
- For all sensitive data your applications need to track data access for, such as regulated industry information, PII, HIPAA, GDPR, and more.
- Enable and Configure User Auditing
- Consuming Audit Data
- User Auditing for Multi Tenancy Environments
- Enable User Audit Data for Self-Service Analytics User Accounts
- Enable User Audit Data for Multi Tenant Accounts
Enable and Configure User Auditing
Write your audit logs to the PostgreSQL database installed with Self-Service Analytics, or an alternative PostgreSQL database of your choice. Create the database, update thezoomdata.properties file, and restart Self-Service Analytics to begin capturing audit events.
If you expect a large number of user audit events, accumulating a significant amount of information over time, consider using a separate PostgreSQL database.
Create a Database
Create the database for user auditing. There are two ways to do so:PostgreSQL databases are the only supported database type.
- Bootstrap creation. When you install or upgrade Self-Service Analytics using a
bootstrap-zoomdatascript, a user audit data table is created on the local PostrgreSQL instance. The user audit database is namedzoomdata-user-auditing, and installed at the same location as the Logi metadata for your Self-Service Analytics instance. - Manual creation. To use a separate database or PostgreSQL instance from your Self-Service Analytics metadata database, edit the
zoomdata.propertiesfile to reflect the target parameters to use. Set the default value ofdestination.params.passwordto the same database password used in your default PostgreSQL database. Self-Service Analytics creates the collections automatically in that database as needed.
Enable User Auditing
After you have created or linked your database, copy these properties into thezoomdata.properties file.
| Property | Default Value | Description |
|---|---|---|
| enabled | false | Use to enable or disable user auditing. Default is false, to disable user auditing. To enable, set to true. Required. |
| destination.name | PostgreSQL | The name of the type of database. Only PostgreSQL is supported. |
| destination.type | postgresql | The database type. Only postgresql is supported. |
| destination.schema | public | |
| destination.collection | The name of the collection of the user audit data. Use to separate data by accounts to prevent access to audit data by unauthorized users. For example, audit_records<Account ID>. | |
| destination.collection-per-account | false | Default is false, to disable To enable, set to true and include a field for Account ID in destination.collection. |
| destination.params.user_name | zoomdata | The user name sent to the database to write audit data. Required. |
| destination.params.password | The password sent to the database to write audit data. Required. Can be the same as your default PostgreSQL database. | |
| destination.params.jdbc_url | jdbc:postgresql://localhost:5432/zoomdata-user-auditing | The path to the audit data database. This can be the same database as the user audit schema as shown, or a separate database or partition, as needed. Required. |
| tenant.attribute | Include any new or existing user attribute to support data capture by tenant customer users. Use in addition to account information that is always captured and retained in the database. |
Restart Self-Service Analytics
Restart Self-Service Analytics to begin capturing audit events. If you used the bootstrap script to install Self-Service Analytics and audit setup, the audit database table is created when the first event is logged. Test your setup by performing any of the event-triggering actions. If you have set up your database manually, set up the Data Writer microservice to support the user audit processes.Enable User Audit Data for Self-Service Analytics User Accounts
Audit data for Self-Service Analytics users, including administrators, are not visible to other Self-Service Analytics accounts. You define this level of separation in the database in one of three ways:- Tables: Configure Self-Service Analytics to collect and write user audit data to different tables, then give users database access only to their own audit data table.
- Views: Disable user auditing by account, then give users access to their own audit data, using specific views and database access rights.
- Row level security: Disable user auditing by account, then configure row level security, limiting access for each user to their own data.
Tables for a Self-Service Analytics Account
- Use separate collections for accounts by defining
user-auditing.destination.collection-per-account=truein thezoomdata.propertiesfile. - Generate an audit event for all accounts to trigger audit data table creation.
- Create user database accounts for each user. Define user access rights for each user to query only their own account’s audit data table.
- Create connections to the audit database using each account.
Views by Self-Service Analytics Account
- Disable use of separate collections for accounts by defining
user-auditing.destination.collection-per-account=falsein thezoomdata.propertiesfile. - Generate an audit event for any account to trigger audit data table creation.
- Create a view for each account that filters the audit data table, using a condition such as
accountID='<Account ID>'. - Create user database accounts for each user. Define user access rights for each user to query only their own account’s audit data view.
- Create connections to the audit database using each account.
Row Level Security in the Database
- Enable user auditing by defining
user-auditing.destination.collection-per-account=falsein thezoomdata.propertiesfile. - Generate an audit event for any account to trigger audit data table creation.
- Create user database accounts for each user, and configure row level security for the audit data table to limit users to their own data. For example, use a filter such as
accountID='<Account ID>. - Create connections to the audit database using each account.
Enable User Audit Data for Multi Tenant Accounts
Configure user auditing for tenants to restrict access to user audit data to their own user tenant data. Self-Service Analytics system administrators can see aggregated data for all tenants in the Self-Service Analytics account. You can define this level of separation in the database in one of several ways, depending on tenant connection creation privileges.Tenants Without Connection Creation Permissions
In environments where tenants do not have permission to create connections, you can set up and use user attribute value, such as${User.tenant} define row level filters to limit access to user audit data.
- Set up an attribute for each user, such as
${User.tenant}, to identify the tenant for each user. - Add the attribute name to the
zoomdata.propertiesfile. In this example, addtenant:user-auditing.tenant.attribute=tenant. - Generate an audit event for any account to trigger audit data table creation.
- As a system administrator, create connection and data sources for the audit data in Self-Service Analytics.
- Add a row level filter to the data source. Compare the column tenant with the user attribute you created. In this example,
tenant=${User.tenant}
Tenants With Connection Creation Permissions
If your tenants do have permission to create connections, there are two ways to restrict user audit data access: by setting up views at the tenant level, or applying row level security. Use the features of your database to control access to the data, once attributes are defined and applied.Views Per Tenant
- Set up an attribute for each user, such as
${User.tenant}, to identify the tenant for each user. - Add the attribute name to the
zoomdata.propertiesfile. In this example, addtenant:user-auditing.tenant.attribute=tenant. - Generate an audit event for any account to trigger audit data table creation.
- Create a view for each tenant that filters the audit data table, using a condition such as
tenant='<Tenant>'. - Create user database accounts for each tenant. Define access rights for account user to query only their own tenant audit data table.
- Create connections to the audit database using each user database account.
Row Level Security
- Set up an attribute for each user, such as
${User.tenant}, to identify the tenant for each user. - Add the attribute name to the
zoomdata.propertiesfile. In this example, addtenant:user-auditing.tenant.attribute=tenant. - Generate an audit event for any account to trigger audit data table creation.
- Create a separate database user for each tenant, and configure row level security for each account user to query only their own tenant audit data table, using a condition such as
tenant='<Tenant>' - Create connections to the audit database using each user database account.
User Auditing for Multi Tenancy Environments
Self-Service Analytics supports two different levels of separation for user audit data. By default, user audit data is separated between tenant accounts. If needed, you can further separate user audit data within a Self-Service Analytics tenant account by adding and using a custom attribute to separate the data. Use Self-Service Analytics to write audit data to a single table in your database or multiple tables to suit your organization’s needs.Separation Between Self-Service Analytics Tenant Accounts
Audit data for Self-Service Analytics users, including administrators, are not visible to other tenant accounts. You define this level of separation in the database in one of three ways:- Tables: Configure Self-Service Analytics to collect and write user audit data to different tables, then give users database access only to their own audit data table.
- Views: Disable user auditing by tenant account, then give users access to their own audit data, using specific views and database access rights.
- Row level security: Disable user auditing by tenant account, then configure row level security, limiting access for each user to their own data.
Separation Between Tenants
Configure user auditing for tenants to restrict access to user audit data to their own user tenant data. Self-Service Analytics system administrators can see aggregated data for all tenants in the Self-Service Analytics tenant account. You can define this level of separation in the database in one of several ways, depending on tenant connection creation privileges.- If tenant users do not have permission to create new connections, add a user attribute value, then make the data available by applying the appropriate row level filter to that value.
-
If tenant users do have permission to create new connections, you must set up database-level access control mechanisms.
- Views per tenant: Define a tenant attribute, then define what user audit data users can access, using specific views and database access rights.
- Database level row level security: Define a tenant attribute, then configure row level security for the audit table data, allowing users access to appropriate user audit data through individual user database accounts.
Consuming Audit Data
Users with appropriate access can review your audit data in one of three ways:- Create visuals and a dedicated dashboard using Self-Service Analytics.
- Direct database queries using any third-party tool.
- Add the user auditing database as a source and connection, then export the information using the Self-Service Analytics API.
Consume Audit Data Using Self-Service Analytics
Use Self-Service Analytics to create a User Auditing dashboard.- Create a connection to your user auditing database.
- Create a source from this connection.
- Create a dashboard for User Auditing with visuals as needed.
If you use a table visualization for user auditing data, add cross-visual filters such as List Filters and Tree Maps for a contextual look at your data.
Consume Audit Data Using Third-Party Tools
You can use any third-party tools you prefer to access the information for user auditing. Access to the tool or to the database using that tool should be restricted to appropriate users.Activities Log Reference Sheet
This reference sheet lists the types of activities and corresponding fields that are logged in the Self-Service Analytics activity log. By default, activity logging is enabled with a few exceptions (described later in this topic).Activity Logging Defaults
The following table lists the activities that are logged by default and those that are not. Activities that are not logged by default must have logging manually enabled. See Activity Logging.| Activity Log File | Activities Logged by Default | Activity Not Logged by Default |
|---|---|---|
zoomdata-activity.log | account authentication group SOURCE USER VIS | raw_data_export raw_data_export_csv upload |
Common Log Fields
For each activity, the following common fields are logged as part of each activity record:
Additional fields are logged as part of each activity.
Activity Log Fields
The following activity types can be logged by Self-Service Analytics. For each activity, specific fields are logged, in addition to the common fields.- account: Account Maintenance
- authentication: User Authentication
- group: Group Maintenance
- raw_data_export: Text Search
- raw_data_export_csv: Data Export to CSV
- source: Data Source Maintenance
- upload: Flat File Upload
- user: User Account Maintenance
- vis: Visual Maintenance
account: Account Maintenance
This activity is logged when a Self-Service Analytics tenant account is created, updated, or deleted. Logging for this activity is enabled by default and captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
authentication: User Authentication
This activity is logged while user authentication at login. Logging for this activity is enabled by default and captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
| Field | Value |
|---|---|
| activityType | authentication |
| status | SUCCEEDED / FAILED |
| authenticationType | BASE, SAML, LDAP, x.509 |
group: Group Maintenance
This activity is logged when a user group is created, updated, or updated. Logging for this activity is enabled by default and captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
| Field | Value |
|---|---|
| groupId | Group ID |
| label | Name of the group |
| description | Group description added by the user |
| group | Privileges that have been assigned for the group |
raw_data_export: Text Search
This activity is logged when a text search (applicable only for Elasticsearch, Apache Solr, and Cloudera Search sources) is performed using the /api/stream/search endpoint. Logging for this activity is disabled by default. After you have manually enabled logging for this activity, its log records are captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
| Field | Value |
|---|---|
| activityType | raw_data_export |
| status | SUCCEEDED |
| exportType | REST (view details) FILE (export file) |
| count | Number of exported rows |
| storageType | Data source |
| query | Queries sent to the data source |
| cid | Created ID |
| actionStartedOn | Export start time |
| duration | Request processing time |
raw_data_export_csv: Data Export to CSV
This activity is logged when a user exports the raw data to a CSV file. Logging for this activity is disabled by default. After you have manually enabled logging for this activity, its log records are captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
| Field | Value |
|---|---|
| activityType | raw_data_export_csv |
| status | SUCCEEDED |
| count | Number of exported rows |
| location | DB or FILE |
| file | location=FILE - the full path to the file in local file system |
| cid | Created ID |
| actionStartedOn | Export start time |
| duration | Request processing time |
source: Data Source Maintenance
This activity is logged when a source is created, updated, or deleted. Logging for this activity is enabled by default and captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
| Field | Value |
|---|---|
| activityType | source |
| status | SAVED, UPDATED, DELETED |
| sourceId | Generated source ID |
| sourceName | Name of the source |
| streamType | Demo_record, CSV, API |
| storageType | Data source type |
| sourceAsString | Contains microservice info about the data source |
| sourceDescription | Source description added by the user |
upload: Flat File Upload
This activity is logged when a user uploads the flat file while creating a new data source. Logging for this activity is disabled by default. After you have manually enabled logging for this activity, its log records are captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
| Field | Value |
|---|---|
| activityType | upload |
| status | SUCCEEDED |
| source | Data source ID to which the file will be uploaded |
| fileName | Name of the uploaded file |
| contentType | Type of the uploaded file |
| filesize | Size of the uploaded file |
user: User Account Maintenance
This activity is logged when user account is created, updated, assigned to or removed from a group, or deleted. Logging for this activity is enabled by default and captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
vis: Visual Maintenance
This activity is logged when a visual is created or updated. Logging for this activity is enabled by default and captured in the zoomdata-activity.log file.
The following fields are recorded for this activity:
| Field | Value |
|---|---|
| activityType | vis |
| status | CREATED / UPDATED |
| visualizationID | |
| visualizationName |