Search code examples
postgresqlapache-kafkapipelinedb

how to build pipeline_kafka extension


i'm currently trying to compile the pipeline_kafka extension from Github to add it to my pipelinedb instance but i'm running in a problem neither i nor google seems to be able to solve.

when i execute make command i get the following error

stc@vm:~/pipeline_kafka$ make gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I. -I./ -I/usr/include/postgresql/9.4/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/tcl8.6 -c -o pipeline_kafka.o pipeline_kafka.c pipeline_kafka.c:23:40: fatal error: catalog/pipeline_stream_fn.h: No such file or directory #include "catalog/pipeline_stream_fn.h" ^ compilation terminated. : recipe for target 'pipeline_kafka.o' failed make: *** [pipeline_kafka.o] Error 1

does anyone has an idea what to do? I'm running on debian 8.

Does anyone know why this extension is not part of pipelinedb anymore?

cheers and thanks in advance


Solution

  • the reason for the error is my entire approach towards using/installing pipelinedb and pipeline_kafka.

    the following script worked for me on a standard Debian 8 installation. Maybe it helps.

    Download and install PipelineDB

    wget -O pipelinedb_install_package "https://www.pipelinedb.com/download/0.9.1/debian8"
    
    sudo dpkg -i pipelinedb_install_package
    
    mkdir ~/logs
    
    pipeline-init -D ~/data/pipelinedb
    pipeline-ctl -D ~/data/pipelinedb -l ~/logs/pipelinedb.log start
    sleep 5.0
    pipeline pipeline 
    

    Download and install pipeline_kafka extension

    sudo apt-get install git gcc g++ zlib1g-dev
    
    mkdir ~/git
    
    git clone https://github.com/edenhill/librdkafka.git ~/git/librdkafka
    git clone https://github.com/pipelinedb/pipeline_kafka.git  >~/git/pipeline_kafka
    
    cd ~/git/librdkafka
    ./configure --prefix=/usr 
    make
    sudo make install
    
    cd ~/git/pipeline_kafka
    ./configure 
    make
    sudo make install
    
    pipeline pipeline -c "CREATE EXTENSION pipeline_kafka;"
    
    sed -i 's/#shared_preload_libraries/shared_preload_libraries/' ~/data/pipelinedb/pipelinedb.conf
    sed -i "s/shared_preload_libraries = ''/shared_preload_libraries=pipeline_kafka/" ~/data/pipelinedb/pipelinedb.conf