Search code examples
pythonpython-3.xdependenciesboto3splunk

Install pyOpenSSL and boto3 manually? Drag and drop doesn't work because of missing dependencies

I have an app that I wrote for Splunk that has dependencies on boto3 and pyOpenSSL libraries. I haven't found a good way to get app dependencies into the apps bin folder other than drag/drop, which isn't working for boto3 and pyOpenSSL.

To this point in time, every time we needed to make a python module available to a single app in Splunk, we would drag and drop the python modules into $SPLUNK_HOME/etc/apps/APP_NAME/bin/MODULE . This has worked until we needed the pyOpenSSL and boto3 libraries which have lots of cryptography and single script dependencies that don't come over correctly.

What I've tried:

1| python3 -m venv $SPLUNK_HOME/etc/apps/APP_NAME/
2| python3 -m pip install (pyOpenSSL, boxsdk, pyJWT, boto3) < base dependencies
3| move $SPLUNK_HOME/etc/apps/APP_NAME/lib/python3.7/site-packages/ > $/SPLUNK_HOME/etc/apps/APP_NAME/bin
4| Put all my app scripts in $/SPLUNK_HOME/etc/apps/APP_NAME/bin alongside all the modules I just installed to that folder using venv
5| Start Splunk
6| search | search_command arg=0

At this point, Splunk tells me that the enum34, ipaddress, chainmap, cryptography (_constant_time module buried in here somewhere doesn't exist where it should) modules don't exist.

I then shut down Splunk, redid steps 1-6 but also installing all those missing modules on step 2. The error I'm getting now is this:

External search command 'boxfiles' returned error code 1. First 1000 (of 1456) bytes of script output: "No module named constant_time ERROR "Error 'No module named constant_time'. Traceback (most recent call last): File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/box_connector/init.py"", line 3, in from box_connector import BoxConnector File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/box_connector/box_connector.py"", line 10, in from OpenSSL import crypto File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/OpenSSL/init.py"", line 8, in from OpenSSL import crypto, SSL File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/OpenSSL/crypto.py"", line 12, in from cryptography import x509 File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/cryptography/x509/init.py"", line 8, in from cryptography.x509.base import ( File ""/Applications/Splunk/etc/apps/TA-box-connector/bin/cryptography/x509/base.py"", line 18, in from cryptography.x509.extensions import Exte".

I'd like to solve this error, but I've been working through this dependency issue for some time now, so if there's a better solution to getting these packages on here, I would love to hear about it.

Solution

  • You can use the following commands to install the dependencies into the appropriate directory. Make sure $SPLUNK_HOME is set properly, and replace APP_NAME with your new app's name.

    pip install pyOpenSSL -t $SPLUNK_HOME/etc/apps/APP_NAME/bin
pip install boxsdk -t $SPLUNK_HOME/etc/apps/APP_NAME/bin
pip install pyJWT -t $SPLUNK_HOME/etc/apps/APP_NAME/bin
pip install boto3 -t $SPLUNK_HOME/etc/apps/APP_NAME/bin

    You may also need to look at following what the Splunk Add-On for AWS does, and deliberately add the required module's to Python's path. (Splunk_TA_aws/bin/aws_bootstrap_env.py).

    '''
Add common libs to sys.path
'''


import os
import os.path
import re
import sys
import logging


def setup_python_path():
    # Exclude folder beneath other apps, Fix bug for rest_handler.py
    ta_name = os.path.basename(os.path.dirname(os.path.dirname(__file__)))
    pattern = re.compile(r"[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$")
    new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
    new_paths.insert(0, os.path.dirname(__file__))
    sys.path = new_paths

    bindir = os.path.dirname(os.path.abspath(__file__))
    # We sort the precedence in a decending order since sys.path.insert(0, ...)
    # do the reversing.
    # Insert library folder
    sharedpath = os.path.join(bindir, '3rdparty', 'python3')
    sys.path.insert(0, sharedpath)


# preventing splunklib initialize an unexpected root handler
def setup_null_handler():
    logging.root.addHandler(logging.NullHandler())


def run_module(name):
    instance = __import__(name, fromlist=['main'])
    instance.main()

setup_python_path()
setup_null_handler()

    References