Installing python module in Robomaker ROS workspace with colcon

I'm working on a cloud-based robotic application with AWS RoboMaker. I'm using ROS Kinetic, with the build tool colcon.

My robot application depends on a custom python module, which has to be in my workspace. This python module is built by colcon as a python package, not a ROS package. This page explains how to do that with catkin, but this example shows how to adapt it to colcon. So finally my workspace looks like that :

my_workspace/
        |--src/
            |--my_module/
            |     |--setup.py
            |     |--package.xml
            |     |--subfolders and python scripts...
            |--some_ros_pkg1/  
            |--some_ros_pkg2/
            |...

However the command : colcon build <my_workspace> builds all ROS packages but fails to build my python module as a package.

Here's the error I get :

Starting >>> my-module
[54.297s] WARNING:colcon.colcon_ros.task.ament_python.build:Package 'my-module' doesn't explicitly install a marker in the package index (colcon-ros currently does it implicitly but that fallback will be removed in the future)
[54.298s] WARNING:colcon.colcon_ros.task.ament_python.build:Package 'my-module' doesn't explicitly install the 'package.xml' file (colcon-ros currently does it implicitly but that fallback will be removed in the future)



--- stderr: my-module                                                                     
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'egg_info'
---
Failed   <<< my-module [0.56s, exited with code 1]

I found this issue that seems correlated, and thus tried : pip install --upgrade setuptools ...Which fails with the error message :

Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/7c/1b/9b68465658cda69f33c31c4dbd511ac5648835680ea8de87ce05c81f95bf/setuptools-50.3.0.zip
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "setuptools/__init__.py", line 16, in <module>
        import setuptools.version
      File "setuptools/version.py", line 1, in <module>
        import pkg_resources
      File "pkg_resources/__init__.py", line 1365
        raise SyntaxError(e) from e
                                ^
    SyntaxError: invalid syntax
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-uwFamt/setuptools/

And with pip3 install --upgrade setuptools, I get :

Defaulting to user installation because normal site-packages is not writeable
Requirement already up-to-date: setuptools in /home/ubuntu/.local/lib/python3.5/site-packages (50.3.0)

I have both Python 3.5.2 an Python 2.7, but I don't know which one is used by colcon.

So I don't know what to try next, and what the real problem is. Any help welcome !

Solution

I managed to correctly install my package and its dependencies. I develop the method below, in case it may help someone someday !

I have been mainly inspired by this old DeepRacer repository.

The workspace tree in the question is wrong. It should look like this:

my_workspace/
        |--src/
            |--my_wrapper_package/
            |     |--setup.py
            |     |--my_package/
            |          |--__init__.py
            |          |--subfolders and python scripts...
            |--some_ros_pkg1/  
            |--some_ros_pkg2/

my_wrapper_package may contain more than one python custom package.
A good setup.py example is this one.
You shouldn't put a package.xml next to setup.py : colcon will only look at the dependencies declared in package.xml, and won't collect pip packages.
It may help sometimes to delete the folders my_wrapper_package generated by colcon in install/ and build/. Doing so you force colcon to rebuild and bundle from scratch.