Manage the Python Connector

Describes the Python connector.

Connect your data in Self-Service Analytics using a Python script. Access information exposed by APIs, or other data generated by calculation or prediction models.

The Python connector is available as a Docker image; you must install Docker on your server running the Python connector. See Install the Python Connector.

Python code is executed using JEP. To circumvent some Global Interpreter Lock issues in Python, some queries use processed based parallelism. Based on the request type, the connector functions in one of two ways:

• Interpret the script in the same process for validation and describe requests.
• Interpret the script in the same process and invoke the function in a sub process for fetch data requests.

For more information, see Use the Python Connector.

Connector Feature Support

Connector support for specific features is shown in the following table.

Feature	Supported?
Admin-Defined Functions	N
Box Plots	Y
Custom SQL Queries	Y
Derived Fields (Row-Level Expressions)	Y
Distinct Counts	Y
Fast Distinct Values	N
Group By Multiple Fields	Y
Group By Time	Y
Group By UNIX Time	Y
Histogram Floating Point Values	Y
Histograms	Y
Kerberos Authentication	N
Last Value	Y
Live Mode and Playback	N
Multivalued Fields	N
Nested Fields	N
Partitions	N
Pushdown Joins for Fusion Data Sources	N
Schemas	N
Text Search	N
TLS	N
User Delegation	N
Wildcard Filters	Y
Wildcard Filters, Case-Insensitive Mode	Y
Wildcard Filters, Case-Sensitive Mode	Y

Install the Python Connector

The python connector is available as a Docker image only. Docker must be installed on the Self-Service Analytics server running the Python connector.

Download and Install the Docker Image in a Linux Environment

To download the Docker image:

docker pull insightsoftware/zoomdata-edc-python:<RELEASE_TAG>

You can find your required <RELEASE_TAG> in this repo: https://hub.docker.com/r/insightsoftware/zoomdata-edc-python/tags.

Use the docker run command to run the Python connector.

You must provide a Consul host and ensure that the connector is registered in Consul with a host that is accessible to other Self-Service Analytics services.

In the example below, the docker run command runs the setup on the same machine that has both the connector installed and other Self-Service Analytics services installed using the bootstrap script:

docker run --env DISCOVERY_REGISTRY_HOST=localhost --network=host --name=zoomdata-edc-python --detach insightsoftware/zoomdata-edc-python:<RELEASE_TAG>

Adjust the command to work with your specific network configuration.

• Consul host is passed to Python Connector using the DISCOVERY_REGISTRY_HOST environment variable.
• Because the container network in this case is connected to the host machine’s network (due to --network=host), Consul is accessed on localhost.
• Use --name param to assign a meaningful name to the container. --detach runs the container in the background and prints the container ID.
• See docker documentation for more information on docker run arguments.

The host networking driver only works in Linux environments. See Run the Python Connector on Non-Linux Servers for generic networking requirements.

Run the Python Connector on Non-Linux Servers

Define a networking configuration that:

• Runs the connector inside a container that can access the Consul host and register with Consul.
• Allows other Self-Service Analytics services to access the connector running inside the container.

For example, in a case when the Consul is running on a different host, --env DISCOVERY_REGISTRY_HOST=localhost --network=host is not suitable. You’ll need to make sure that Consul listens on external port 8500, and takes its hostname. Additionally, you may need to use the --expose 8153 argument to expose the port that the Python Connector listens on. Also, you can use the --hostname argument to control the value of the service address that will be registered in Consul for Python Connector. Putting it together:

docker run --expose 8153 --env DISCOVERY_REGISTRY_HOST=<consul-host> --hostname <connector_host> --name=zoomdata-edc-python --detach insightsoftware/zoomdata-edc-python:<RELEASE_TAG>

Verify the Installation

To verify correct installation of the Python connector, run the following command shortly after Self-Service Analytics starts:

curl localhost:8500/v1/health/service/edc-python

All checks must return the status passing.

After that, log in to Self-Service Analytics create a new connection. Python should be available in the Connection Type list.

View Python Logs

To view the Python connector’s logs use the docker logs command:

docker logs --follow zoomdata-edc-python

Python Packages

The Docker image is shipped with the python3-pip package installed. This includes preinstalled pip packages of numpy, pandas, requests, and jep. To install additional pip packages, use the ADDITIONAL_PYTHON_LIBS environment variable when running the container.

docker run --expose 8153 --env DISCOVERY_REGISTRY_HOST=<consul-host> --env ADDITIONAL_PYTHON_LIBS="boto3 python-dateutil" --detach insightsoftware/zoomdata-edc-python:<RELEASE_TAG>

This command runs the Python connector and installs both the boto3 and python-dateutil packages inside the container.

For more information on using the Python connector and how it works, see Use the Python Connector.

Use the Python Connector

You can use the Python connector using arbitrary Python scripts as connection parameter.

The Python connector functions in raw data mode only and does not support push down of aggregations. Minimize the amount of data in your request using filter operations. This will increase performance speed and improve loading time for dashboards and visuals.

Python Script Conventions

Data sources are resolved from Python script using the following conventions:

• Each function definition is a separate data source
• Private functions (that start with an underscore _) are not resolved as a data source

Conventions for return values of functions include:

• Pandas dataframes.
• Dictionaries: The key is a string (column name) and values a list: return {"column1": [1, 2, 3], "column2": ["one", "two", "three"]}.
• List of dictionaries: return [{"column1": 1, "column2": "one"}, {"column1": 2, "column2": "two"}].
• List of lists: return [[1, 2], [3, 4]]. Each enclosed list resolves as a row.
• List: return [1, 2, 3, 4]. Resolves as a single column with index 1.
• Single value of any of supported types: return 1 OR return decimal.Decimal("3.14")

Regardless of return type, you must use uniform columns of the same size, containing the same value type, to prevent unexpected behavior.

Conversion Values

The connector applies the following rules when reading values:

Python Type	Connector Field Type
str	STRING
int	INTEGER
float	DOUBLE
decimal.Decimal	DOUBLE
datetime.date	DATE
datetime.datetime	DATE
arbitrary object	STRING

Python Script Writing Tips

Avoid using top level statements

Top level statement are executed in a single thread for all users. You can add function calls in a top level statement to validate a connection, but do not call functions when you save your script. See How the Connector Works.

Avoid overriding internal names

Python is used within your environment to invoke data source functions and convert data. Since this code is executed in the same namespace as your scripts, if you try to override the names listed below, you may receive unexpected results. Avoid using the following names in your code:

• __convert
• __convert_list_of_dicts_to_dict_of_lists
• f
• __fork
• __emulate
• all_functions

To use this connector, we import these modules. Attempting to use these names for variables and functions may return unexpected results.

• pandas
• numbers
• datetime
• multiprocessing
• queue
• inspect
• types

Your python script has limited access to the file system; the container is run as a non-root user. Edit access is still available on the folders below:

Folders with write access include:

• /opt/zoomdata/logs
• /opt/zoomdata/temp
• /opt/zoomdata/lib
• /opt/zoomdata/wrappers

How the Connector Works

Python code is executed using JEP to interpret Python code in the same process where the Java app is running. To circumvent some Global Interpreter Lock issues in Python, some queries use processed based parallelism. Based on the request type, the connector functions in one of two ways:

• Interpret the script in the same process for validation and describe requests.
• Interpret the script in the same process and invoke the function in a sub process for fetch data requests.

New sub processes are created by forking the Java process. Each request starts a new Python interpreter. Scripts are always interpreted first in one parent process. Limit top level statements to imports and function definitions for optimal performance.

Function invocation happens in a separate process, so global variables are not available. For example:

x = 40

def side_effect():
	x = x + 1
	return {"result": [x]}

While the script is valid, using it as a connection parameter and attempting to set side_effect as an entity for the source will return an error such as:

UnboundLocalError: local variable 'x' referenced before assignment

Logging

Outputs of your Python scripts are not preserved. Statements such as print("Message") to write data to stdout or stderr will not be retained.

Python Script Conversion

Python script types conversion converts all values returned from public functions to Pandas Dataframe. To resolve the type, the connector relies on DataFrame.dtypes.kind.

The following table includes the conversion rules used:

Numpy kind (character code)	Numpy kind (type name)	Field Type
b	boolean	STRING
i	signed integer	INTEGER
u	unsigned integer	INTEGER
f	floating-point	DOUBLE
c	complex floating-point	STRING
m	timedelta	STRING
M	datetime	DATE
O	object	STRING
S	(byte-)string	STRING
U	Unicode	STRING