Importing a custom APM processor in a Python FastAPI app

Question

I am using Elasticsearch APM in Python to log requests for a FastAPI app.

There is a mechanism to load custom processors by simply specifying the module path to the custom processor function.

My app layout is as follows:

app
├── __init__.py
├── apm_processors.py
├── main.py
└── version.py

My apm_processors.py file has the following function defined:

def ip_processor(client, event):
    # process the event
    pass

So, in my app, I do:

# other imports
from elasticapm.contrib.starlette import make_apm_client, ElasticAPM
from . import apm_processors

And to set up APM, I call:

apm_config = {
    "SERVICE_NAME": config('ELASTIC_APM_SERVICE_NAME', cast=str, default="Server"),
    "SECRET_TOKEN": str(config('ELASTIC_APM_SECRET_TOKEN', cast=Secret, default="")),
    "SERVER_URL": config('ELASTIC_APM_SERVER_URL', cast=str, default="http://localhost:8200"),
    "PROCESSORS": (
        "elasticapm.processors.sanitize_http_request_cookies",
        "elasticapm.processors.sanitize_http_headers",
        "apm_processors.ip_processor"
    )
}

apm = make_apm_client(apm_config)

app = FastAPI()
# other app setup

app.add_middleware(ElasticAPM, client=apm)

This throws an error:

ModuleNotFoundError: No module named 'apm_processors'

With the following stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/config.py", line 458, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 24, in import_from_string
    raise exc from None
  File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/statisticsserver/./app/main.py", line 75, in <module>
    apm = make_apm_client(apm_config)
  File "/usr/local/lib/python3.9/site-packages/elasticapm/contrib/starlette/__init__.py", line 70, in make_apm_client
    return client_cls(config, **defaults)
  File "/usr/local/lib/python3.9/site-packages/elasticapm/base.py", line 153, in __init__
    "processors": self.load_processors(),
  File "/usr/local/lib/python3.9/site-packages/elasticapm/base.py", line 640, in load_processors
    return [seen.setdefault(path, import_string(path)) for path in processors if path not in seen]
  File "/usr/local/lib/python3.9/site-packages/elasticapm/base.py", line 640, in <listcomp>
    return [seen.setdefault(path, import_string(path)) for path in processors if path not in seen]
  File "/usr/local/lib/python3.9/site-packages/elasticapm/utils/module_import.py", line 47, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'apm_processors'

As far as I can tell it uses import_module to import the function as specified, but for some reason it claims there is no such module — although I am actually able to import it.

How can I specify the processor?

Answer 1

Since the context/main module of the application is app, the import should be done as follows:

"PROCESSORS": (
    "elasticapm.processors.sanitize_http_request_cookies",
    "elasticapm.processors.sanitize_http_headers",
    "app.apm_processors.ip_processor"
)

This way, the import succeeds and the function will be called for each event.