Managing Third-Party Python Packages

The Hyperscience Platform supports a special type of block called PythonBlock that allows users to run custom Python code. By default, users can only use packages that are part of the Python Standard Library. To be able to use third-party Python packages, you will need to have them installed on the Hyperscience Platform.

Installing Third-Party Python Packages

The steps below will walk you through downloading an example package called regex and installing it on a running Hyperscience instance. This process requires SSH access to a machine running the Hyperscience Platform.

  1. Download the wheel files using pip download. Note that this command requires access to PyPi repositories and can be run on your local machine.

    pip download --python-version 3.7 --platform=manylinux2014_x86_64 --only-binary=:all: regex
    
  2. Upload the wheel files to an application machine. Note that an application machine is a machine that is running the Hyperscience Platform, not a machine that is running the trainer.

    scp regex-2022.3.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl root@app_machine.hyperscience.com:/mnt/hs/media/
    

    Always upload the wheel files to /mnt/hs/media so that the Docker containers on the application machine will have access to them. /mnt/hs/media is a shared volume that is mapped to a directory inside the containers. We’ll learn more about that directory in step 4.

  3. SSH to the application machine.

    ssh root@app_machine.hyperscience.com
    
  4. cd to the directory where Hyperscience has been installed

    This is usually /opt/hs/bundle_install/, but may vary depending on your installation.

    cd <hyperscience install directory>
    
  5. Execute the install command.

    run.sh manage python_packages install /var/www/forms/forms/media/regex-2022.4.24-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    

    /var/www/forms/forms/media/ is the directory inside the container that /mnt/hs/media maps to. So, while you put files in /mnt/hs/media, you need to replace /mnt/hs/media with /var/www/forms/forms/media/ in the command. Do not make any other replacement for /mnt/hs/media. The install command takes in either a wheel file or a directory that contains wheel files. Look for the following output for a successful installation:

    [2022-04-26 03:53:33.220 | INFO | MainThread | sdm.management.commands.python_packages:179] regex 2022.4.24 has been uploaded for installation.
    [2022-04-26 03:53:33.223 | INFO | MainThread | sdm.management.commands.python_packages:295] Installation has been triggered.
    [2022-04-26 03:53:33.223 | INFO | MainThread | sdm.management.commands.python_packages:299] Please note: It may take up to several minutes for changes to packages to be picked up by the flow execution processes on all machines
    

    Users can go to Flows > Python Packages in the Hyperscience Platform to see the list of third-party Python packages that have been installed.

    If there is an existing version of the regex package available in the system, your install command will be rejected. To overwrite the existing version, use the -f/--force option. You may find this option useful when upgrading packages.

    run.sh manage python_packages install -f /var/www/forms/forms/media/regex-2022.4.24-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    

    When you have more than one wheel file that you wish to install, you should supply the directory that contains the wheel files instead of the wheel files themselves as the argument. The program will take in all files with the .whl extension.

    run.sh manage python_packages install -f /var/www/forms/forms/media/
    

Uninstalling Third-Party Python Packages

The steps below show you how to uninstall the regex package from the Hyperscience Platform.

  1. Find the name of the package to uninstall.

    Go to Flows > Python Packages in the Hyperscience Platform, and find the name of the package to uninstall in the Python Package Name column. In this example, the regex package we just installed has the name “regex”. This name uniquely identifies the package.

  2. Execute the uninstall command.

    cd <hyperscience install directory>
    run.sh manage python_packages uninstall regex