> ## Documentation Index
> Fetch the complete documentation index at: https://insightsoftware.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage the Python Connector

> Describes the Python connector.

Connect your data in Self-Service Analytics using a Python script. Access information exposed by APIs, or other data generated by calculation or prediction models.

<Warning>
  The Python connector is available as a Docker image; you must install Docker on your server running the Python connector. See [Install the Python Connector](#install-the-python-connector).
</Warning>

Python code is executed using [JEP](https://github.com/ninia/jep). To circumvent some Global Interpreter Lock issues in Python, some queries use [processed based parallelism](https://docs.python.org/3/library/multiprocessing.html). Based on the request type, the connector functions in one of two ways:

* Interpret the script in the same process for validation and describe requests.
* Interpret the script in the same process and invoke the function in a sub process for fetch data requests.

For more information, see [Use the Python Connector](#use-the-python-connector).

## Connector Feature Support

Connector support for specific [features](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support) is shown in the following table.

| Feature                                                                                                                                                                                      | Supported? |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| [Admin-Defined Functions](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/data/admin-fx-ov)                                                                       | **N**      |
| [Box Plots](/simba-embedded-analytics/docs/self-service-analytics/26.3/analyze-data/visuals/scatter-plot#box-plots)                                                                          | **Y**      |
| [Custom SQL Queries](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#custom-sql-queries-2)                                   | **Y**      |
| [Derived Fields (Row-Level Expressions)](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/data/derived-fields)                                                     | **Y**      |
| [Distinct Counts](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/data/fx-aggregate#distinct-counts)                                                              | **Y**      |
| [Fast Distinct Values](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#fast-distinct-values)                                 | **N**      |
| [Group By Multiple Fields](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#group-by-multiple-fields)                         | **Y**      |
| [Group By Time](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#group-by-time)                                               | **Y**      |
| [Group By UNIX Time](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#group-by-unix-time)                                     | **Y**      |
| [Histogram Floating Point Values](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#histogram-floating-point-values)           | **Y**      |
| [Histograms](/simba-embedded-analytics/docs/self-service-analytics/26.3/analyze-data/visuals/bar-standard#bars-histograms)                                                                   | **Y**      |
| [Kerberos Authentication](/simba-embedded-analytics/docs/self-service-analytics/26.3/administer/security/kerberos-sso)                                                                       | **N**      |
| [Last Value](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#last-value)                                                     | **Y**      |
| [Live Mode and Playback](/simba-embedded-analytics/docs/self-service-analytics/26.3/analyze-data/time-bar#live-mode-and-historical-playback)                                                 | **N**      |
| [Multivalued Fields](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#multivalued-fields-2)                                   | **N**      |
| [Nested Fields](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/data/nested-data-structures)                                                                      | **N**      |
| [Partitions](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#partitions)                                                     | **N**      |
| [Pushdown Joins for Fusion Data Sources](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/sources/data-fusion-overview#optimize-joins)                             | **N**      |
| [Schemas](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#schemas-2)                                                         | **N**      |
| [Text Search](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#text-search)                                                   | **N**      |
| [TLS](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#tls)                                                                   | **N**      |
| [User Delegation](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connectors-ov#enable-user-delegation)                                                | **N**      |
| [Wildcard Filters](/simba-embedded-analytics/docs/self-service-analytics/26.3/analyze-data/filters/filters-attr#apply-wildcard-filters-to-a-visual-filter-snippet-or-dashboard)              | **Y**      |
| [Wildcard Filters, Case-Insensitive Mode](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#wildcard-case-insensitive-filters) | **Y**      |
| [Wildcard Filters, Case-Sensitive Mode](/simba-embedded-analytics/docs/self-service-analytics/26.3/connect-to-data/connectors/connector-feature-support#wildcard-case-sensitive-filters)     | **Y**      |

<h2 id="install-the-python-connector">
  Install the Python Connector
</h2>

The python connector is available as a Docker image only. Docker must be installed on the Self-Service Analytics server running the Python connector.

### Download and Install the Docker Image in a Linux Environment

To download the Docker image:

```bash theme={null}
docker pull insightsoftware/zoomdata-edc-python:<RELEASE_TAG>
```

You can find your required `<RELEASE_TAG>` in this repo: [https://hub.docker.com/r/insightsoftware/zoomdata-edc-python/tags](https://hub.docker.com/r/insightsoftware/zoomdata-edc-python/tags).

Use the **docker run** command to run the Python connector.

<Warning>
  You must provide a Consul host and ensure that the connector is registered in Consul with a host that is accessible to other Self-Service Analytics services.
</Warning>

In the example below, the **docker run** command runs the setup on the same machine that has both the connector installed and other Self-Service Analytics services installed using the bootstrap script:

```bash theme={null}
docker run --env DISCOVERY_REGISTRY_HOST=localhost --network=host --name=zoomdata-edc-python --detach insightsoftware/zoomdata-edc-python:<RELEASE_TAG>
```

<Note>
  Adjust the command to work with your specific network configuration.
</Note>

* Consul host is passed to Python Connector using the `DISCOVERY_REGISTRY_HOST` environment variable.
* Because the container network in this case is connected to the host machine’s network (due to `--network=host`), Consul is accessed on `localhost`.
* Use `--name param` to assign a meaningful name to the container. `--detach` runs the container in the background and prints the container ID.
* See [docker documentation](https://docs.docker.com/engine/reference/commandline/run/) for more information on `docker run` arguments.

<Warning>
  The host networking driver only works in Linux environments. See [Run the Python Connector on Non-Linux Servers](#run-the-python-connector-on-non-linux-servers) for generic networking requirements.
</Warning>

<h4 id="run-the-python-connector-on-non-linux-servers">
  Run the Python Connector on Non-Linux Servers
</h4>

Define a networking configuration that:

* Runs the connector inside a container that can access the Consul host and register with Consul.
* Allows other Self-Service Analytics services to access the connector running inside the container.

For example, in a case when the Consul is running on a different host, `--env DISCOVERY_REGISTRY_HOST=localhost --network=host` is not suitable. You’ll need to make sure that Consul listens on external port 8500, and takes its hostname. Additionally, you may need to use the `--expose 8153` argument to expose the port that the Python Connector listens on. Also, you can use the `--hostname` argument to control the value of the service address that will be registered in Consul for Python Connector. Putting it together:

```bash theme={null}
docker run --expose 8153 --env DISCOVERY_REGISTRY_HOST=<consul-host> --hostname <connector_host> --name=zoomdata-edc-python --detach insightsoftware/zoomdata-edc-python:<RELEASE_TAG>
```

### Verify the Installation

To verify correct installation of the Python connector, run the following command shortly after Self-Service Analytics starts:

```bash theme={null}
curl localhost:8500/v1/health/service/edc-python
```

All checks must return the status `passing`.

After that, log in to Self-Service Analytics create a new connection. Python should be available in the Connection Type list.

### View Python Logs

To view the Python connector’s logs use the `docker logs` command:

```bash theme={null}
docker logs --follow zoomdata-edc-python
```

### Python Packages

The Docker image is shipped with the `python3-pip` package installed. This includes preinstalled pip packages of `numpy`, `pandas`, `requests`, and `jep`. To install additional pip packages, use the `ADDITIONAL_PYTHON_LIBS` environment variable when running the container.

```bash theme={null}
docker run --expose 8153 --env DISCOVERY_REGISTRY_HOST=<consul-host> --env ADDITIONAL_PYTHON_LIBS="boto3 python-dateutil" --detach insightsoftware/zoomdata-edc-python:<RELEASE_TAG>
```

This command runs the Python connector and installs both the `boto3` and `python-dateutil` packages inside the container.

For more information on using the Python connector and how it works, see [Use the Python Connector](#use-the-python-connector).

<h2 id="use-the-python-connector">
  Use the Python Connector
</h2>

You can use the Python connector using arbitrary Python scripts as connection parameter.

<Warning>
  The Python connector functions in raw data mode only and does not support push down of aggregations. Minimize the amount of data in your request using filter operations. This will increase performance speed and improve loading time for dashboards and visuals.
</Warning>

### Python Script Conventions

Data sources are resolved from Python script using the following conventions:

* Each function definition is a separate data source
* Private functions (that start with an underscore `_`) are not resolved as a data source

Conventions for return values of functions include:

* [Pandas](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) dataframes.
* Dictionaries: The key is a string (column name) and values a list: `return {"column1": [1, 2, 3], "column2": ["one", "two", "three"]}`.
* List of dictionaries: return `[{"column1": 1, "column2": "one"}, {"column1": 2, "column2": "two"}]`.
* List of lists: `return [[1, 2], [3, 4]]`. Each enclosed list resolves as a row.
* List: `return [1, 2, 3, 4]`. Resolves as a single column with index 1.
* Single value of any of supported types: `return 1` OR `return decimal.Decimal("3.14")`

Regardless of return type, you must use uniform columns of the same size, containing the same value type, to prevent unexpected behavior.

#### Conversion Values

The connector applies the following rules when reading values:

| Python Type       | Connector Field Type |
| ----------------- | -------------------- |
| str               | STRING               |
| int               | INTEGER              |
| float             | DOUBLE               |
| decimal.Decimal   | DOUBLE               |
| datetime.date     | DATE                 |
| datetime.datetime | DATE                 |
| arbitrary object  | STRING               |

### Python Script Writing Tips

Avoid using top level statements

Top level statement are executed in a single thread for all users. You can add function calls in a top level statement to validate a connection, but do not call functions when you save your script. See [How the Connector Works](#how-the-connector-works).

Avoid overriding internal names

Python is used within your environment to invoke data source functions and convert data. Since this code is executed in the same namespace as your scripts, if you try to override the names listed below, you may receive unexpected results. Avoid using the following names in your code:

* `__convert`
* `__convert_list_of_dicts_to_dict_of_lists`
* `f`
* `__fork`
* `__emulate`
* `all_functions`

To use this connector, we import these modules. Attempting to use these names for variables and functions may return unexpected results.

* `pandas`
* `numbers`
* `datetime`
* `multiprocessing`
* `queue`
* `inspect`
* `types`

Your python script has limited access to the file system; the container is run as a non-root user. Edit access is still available on the folders below:

Folders with write access include:

* `/opt/zoomdata/logs`
* `/opt/zoomdata/temp`
* `/opt/zoomdata/lib`
* `/opt/zoomdata/wrappers`

<h3 id="how-the-connector-works">
  How the Connector Works
</h3>

Python code is executed using [JEP](https://github.com/ninia/jep) to interpret Python code in the same process where the Java app is running. To circumvent some Global Interpreter Lock issues in Python, some queries use [processed based parallelism](https://docs.python.org/3/library/multiprocessing.html). Based on the request type, the connector functions in one of two ways:

* Interpret the script in the same process for validation and describe requests.
* Interpret the script in the same process and invoke the function in a sub process for fetch data requests.

New sub processes are created by [forking](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) the Java process. Each request starts a new Python interpreter. Scripts are always interpreted first in one parent process. Limit top level statements to imports and function definitions for optimal performance.

Function invocation happens in a separate process, so global variables are not available. For example:

```
x = 40

def side_effect():
	x = x + 1
	return {"result": [x]}
```

While the script is valid, using it as a connection parameter and attempting to set `side_effect` as an entity for the source will return an error such as:

```yaml theme={null}
UnboundLocalError: local variable 'x' referenced before assignment
```

### Logging

Outputs of your Python scripts are not preserved. Statements such as `print("Message")` to write data to `stdout` or `stderr` will not be retained.

### Python Script Conversion

Python script types conversion converts all values returned from public functions to [Pandas Dataframe](https://pandas.pydata.org/docs/reference/frame.html). To resolve the type, the connector relies on [DataFrame.dtypes.kind](https://numpy.org/doc/stable/reference/generated/numpy.dtype.kind.html#numpy.dtype.kind).

The following table includes the conversion rules used:

| Numpy kind (character code) | Numpy kind (type name) | Field Type |
| --------------------------- | ---------------------- | ---------- |
| b                           | boolean                | STRING     |
| i                           | signed integer         | INTEGER    |
| u                           | unsigned integer       | INTEGER    |
| f                           | floating-point         | DOUBLE     |
| c                           | complex floating-point | STRING     |
| m                           | timedelta              | STRING     |
| M                           | datetime               | DATE       |
| O                           | object                 | STRING     |
| S                           | (byte-)string          | STRING     |
| U                           | Unicode                | STRING     |