Managing Third-Party Python Packages

The Hyperscience Platform supports a special type of block called PythonBlock that allows users to run custom Python code. By default, users can only use packages that are part of the Python Standard Library. To be able to use third-party Python packages, you will need to have them installed on the Hyperscience Platform. If you no longer need packages you’ve installed, you can uninstall them.

The tutorials below will guide you through the installation and removal of third-party Python packages.

Managing packages in v36 and later

In v36 of the Hyperscience application, CodeBlock and PythonBlock were updated to use python3.9 instead of the python3.7 used in previous platform versions.

If v36 is the first version you’re creating flows for, you do not need to complete any additional steps before installing and using third-party Python packages.

Warning

If you are migrating from a pre-v36 instance of Hyperscience, some of your packages might be incompatible with the new Python version. You should test to ensure the packages still work on the updated application version (e.g., by creating a submission for a flow that uses code from those packages).

If your testing shows that a package is incompatible with python3.9, you can update it in the application. To do so:

Uninstall the package by following the steps in Uninstalling third-party Python packages.
Download the package again, explicitly targeting python-version 3.9.
Install the downloaded package by following the steps in Installing third-party Python packages.

Installing third-party Python packages

The steps below will walk you through downloading an example package called regex and installing it on a running Hyperscience instance.

Download the regex package using pip download:
```
pip download --python-version 3.9 --platform=manylinux2014_x86_64 --only-binary=:all: regex
```
Note that this command requires access to PyPi repositories and can be run on your local machine.

Do not omit --platform=manylinux2014_x86_64, especially if you are using Mac or Windows. Otherwise, you will download the incorrect package.

NOTE: When working with your own packages, make sure that any packages you download for installation do not require GPU acceleration. Doing so ensures compatibility with any machine running the packages.
In the application, go to Flows > Python Packages, and click Add Packages.
Click Choose Files, and find and open the package’s wheel file (e.g., regex-2022.10.31-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl) on your machine.

TIP: You can upload multiple packages at once by selecting their wheel files (.whl) in this step.
Click Upload.

The package appears in the Python Packages table.

Uninstalling third-party Python packages

Now we will uninstall the regex package we just installed.

In the application, go to Flows > Python Packages, and find the regex package in the Python Packages table.
Click the Uninstall Package button ( ) in the package’s row in the table.

The package is removed from from the instance.

More information

To learn more about managing Python packages in v36—including updating and downloading installed packages—see Developing Flows in our application documentation.

Managing packages in v35

In v35, we added a number of package-management features to the Hyperscience application. These features do not require you to run commands in your virtual environment.

Expand for details on managing packages in v35

Installing third-party Python packages in v35

The steps below will walk you through downloading an example package called regex and installing it on a running Hyperscience instance.

Download the regex package using pip download:
```
pip download --python-version 3.7 --platform=manylinux2014_x86_64 --only-binary=:all: regex
```
Note that this command requires access to PyPi repositories and can be run on your local machine.

Do not omit --platform=manylinux2014_x86_64, especially if you are using Mac or Windows. Otherwise, you will download the incorrect package.

NOTE: When working with your own packages, make sure that any packages you download for installation do not require GPU acceleration. Doing so ensures compatibility with any machine running the packages.
In the application, go to Flows > Python Packages, and click Add Packages.
Click Choose Files, and find and open the package’s wheel file (.whl, e.g., regex-2022.10.31-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl) on your machine.

TIP: You can upload multiple packages at once by selecting their wheel files in this step.
Click Upload.

The package appears in the Python Packages table.

More information on v35

To learn more about managing Python packages in v35—including updating and downloading installed packages—see Developing Flows in our application documentation.

Managing packages in v34 Docker-based deployments

In v34, you need to SSH into your application machine to install and manage Python packages.

Expand for details on managing packages in v34

The steps below will walk you through downloading an example package called regex and installing it on a running Hyperscience instance. This process requires SSH access to a machine running the Hyperscience Platform.

Download the wheel file using pip download:
```
pip download --python-version 3.7 --platform=manylinux2014_x86_64 --only-binary=:all: regex
```
Note that this command requires access to PyPi repositories and can be run on your local machine.

*Do not omit --platform=manylinux2014_x86_64, especially if you are using Mac or Windows. Otherwise, you will download the incorrect package.

NOTE: When working with your own packages, make sure that any packages you download for installation do not require GPU acceleration. Doing so ensures compatibility with any machine running the packages.
Upload the wheel file (e.g., regex-2022.3.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl) to an application machine. Note that an application machine is a machine that is running the Hyperscience Platform, not a machine that is running the trainer.
```
scp regex-2022.3.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl root@app_machine.hyperscience.com:/mnt/hs/media/
```
Always upload the wheel files to /mnt/hs/media so that the Docker containers on the application machine will have access to them. /mnt/hs/media is a shared volume that is mapped to a directory inside the containers. We’ll learn more about that directory in step 4.
ssh to the application machine.
```
ssh root@app_machine.hyperscience.com
```
cd to the directory where Hyperscience has been installed.

This directory is usually /opt/hs/bundle_install/, but may vary depending on your installation.
```
cd <hyperscience install directory>
```
Execute the install command:
```
run.sh manage python_packages install /var/www/forms/forms/media/regex-2022.4.24-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
```
/var/www/forms/forms/media/ is the directory inside the container that /mnt/hs/media maps to. So, while you put files in /mnt/hs/media, you need to replace /mnt/hs/media with /var/www/forms/forms/media/ in the command. Do not make any other replacement for /mnt/hs/media. The install command takes in either a wheel file or a directory that contains wheel files. Look for the following output for a successful installation:
```
[2022-04-26 03:53:33.220 | INFO | MainThread | sdm.management.commands.python_packages:179] regex 2022.4.24 has been uploaded for installation.
[2022-04-26 03:53:33.223 | INFO | MainThread | sdm.management.commands.python_packages:295] Installation has been triggered.
[2022-04-26 03:53:33.223 | INFO | MainThread | sdm.management.commands.python_packages:299] Please note: It may take up to several minutes for changes to packages to be picked up by the flow execution processes on all machines
```
Users can go to Flows > Python Packages in the Hyperscience Platform to see the list of third-party Python packages that have been installed.

If there is an existing version of the regex package available in the system, your install command will be rejected. To overwrite the existing version, use the -f/--force option. You may find this option useful when upgrading packages.
```
run.sh manage python_packages install -f /var/www/forms/forms/media/regex-2022.4.24-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
```
When you have more than one wheel file that you wish to install, you should supply the directory that contains the wheel files instead of the wheel files themselves as the argument. The program will take in all files with the .whl extension.
```
run.sh manage python_packages install -f /var/www/forms/forms/media/
```

Managing packages in v34 on-premise Kubernetes deployments

In v34, you need to use kubectl to install and manage Python packages for on-premise Kubernetes deployments.

Expand for details on managing packages in v34 Kubernetes deployments

The steps below will walk you through downloading an example package called regex and installing it on a running Hyperscience instance.

Download the package’s wheel file using pip download:
```
pip download --python-version 3.7 --platform=manylinux2014_x86_64 --only-binary=:all: regex
```
Note that this command requires access to PyPi repositories and can be run on your local machine.

*Do not omit --platform=manylinux2014_x86_64, especially if you are using Mac or Windows. Otherwise, you will download the incorrect package.

NOTE: When working with your own packages, make sure that any packages you download for installation do not require GPU acceleration. Doing so ensures compatibility with any machine running the packages.

Upload the wheel file (e.g., regex-2022.3.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl) to a Kubernetes pod via kubectl:

KUBERNETES_NAMESPACE=<namespace>
WHEEL_FILEPATH=<wheel path>
CONTAINER=gunicorn 
POD=$(kubectl -n $KUBERNETES_NAMESPACE get pods -l app.kubernetes.io/component=frontend \
                                                --field-selector=status.phase=Running \
                                                -o=jsonpath="{.items[0].metadata.name}")

kubectl cp -n $KUBERNETES_NAMESPACE -c $CONTAINER $WHEEL_FILEPATH $POD:/tmp/$WHEEL_FILEPATH

Install the wheel file:

This command installs the package in the Hyperscience Platform and removes the wheel file previously uploaded to the pod.
```
kubectl exec $POD -c $CONTAINER -- bash -c "source /var/www/entrypoint.sh; /var/www/forms/forms/manage.py python_packages install /tmp/$WHEEL_FILEPATH; rm /tmp/$WHEEL_FILEPATH"
```
NOTE: Make sure to run step 3 in the same session as step 2, as it requires the previously set environment variables.

If there is an existing version of the regex package, the install command will be rejected. To overwrite the existing version, use the -f/--force option. You may find this option useful when upgrading packages.
```
kubectl exec $POD -c $CONTAINER -- bash -c "source /var/www/entrypoint.sh; /var/www/forms/forms/manage.py python_packages install -f /tmp/$WHEEL_FILEPATH; rm /tmp/$WHEEL_FILEPATH"
```

After step 3 is completed, the regex package should be listed in the Hyperscience Platform under Flows > Python Packages.