Search code examples
typo3ddev

custom DDEV pull provider to update local database and user generated files

I'm trying to create a custom DDEV Provider, to import the current database and also user generated files from the web server.

I want to use it with TYPO3 Projects, where I develop the EXT locally with DDEV (because its awesome :) ) and I want to update my local database and also the "fileadmin" files with the help of the ddev pull function.

I've read the docs: Introduction to Hosting Provider Integration and I tested the bash commands locally within the DDEV Container (ddev ssh) and I'm able to connect to the remote Webserver and make a database dump and transfer it to the local DDEV container.

So I added the bash commands to the my custom provider .yaml file in the /provider/ folder.

Here is the current file:

environment_variables:
  DB_NAME: db_name
  DB_USER: password
  DB_PASSWORD: password
  HOST_IP: 11.11.11.11
  SSH_USERNAME: username
  SSH_PASSWORD: password
  SSH_PORT: 22

db_pull_command:
  command: |
    # Creates the .download folder if it doesn't exist
    mkdir -p /var/www/html/.ddev/.downloads
    # execute the mysqldump on the remote webserver via SSH
    ssh -p ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP} 'mysqldump -h 127.0.0.1 -u ${DB_USER} -p ${DB_PASSWORD} ${DB_NAME} > /tmp/${DB_NAME}.sql.gz'
    # download to sql file to the ddev folder
    scp -P ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP}:/tmp/${DB_NAME}.sql.gz /var/www/html/.ddev/.downloads/db.sql.gz.

If I execute the pull with ddev pull my-provider I get the following Error:

Downloading database...
bash: 03: command not found
Pull failed: Failed to exec mkdir -p /var/www/html/.ddev/.downloads

I assumed that the commands are executed like I would within the DDEV Container (with ddev ssh). What am I missing?

My Environment:

  • TYPO3 v10.4.20
  • Windows 10 (WSL)
  • Docker Desktop 3.5.2
  • DDEV-Local version v1.17.7
  • architecture amd64
  • db drud/ddev-dbserver-mariadb-10.3:v1.17.7
  • dba phpmyadmin:5
  • ddev-ssh-agent drud/ddev-ssh-agent:v1.17.0
  • docker 20.10.7
  • docker-compose 1.29.2

The web server is running on Plesk.

Note: I only tried to implement the db pull command so far.

UPDATE 09.11.21:

So I've gotten this far that I'm able update and also download the files. However I'm only able to do it, if I hardcode the variables. Everytime I'm trying to setup the environment_variables: I get the following error, if I run the ddev pull myProvider:

Downloading database...
bash: 03: command not found

Here is my current .yaml file with the environment_variables:, which currently don't work. I've tested all the commands within ddev ssh and it works if I call them manually.

environment_variables:
  DB_NAME: db_name
  DB_USER: db_user
  DB_PASSWORD: 'Password$'
  HOST_IP: 10.10.10.10
  SSH_USERNAME: username
  SSH_PORT: 21

auth_command:
  command: |
    ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )

db_pull_command:
  command: |
    mkdir -p /var/www/html/.ddev/.downloads
    ssh -p ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP} "mysqldump -h 127.0.0.1 -u ${DB_USER} -p'${DB_PASSWORD}' ${DB_NAME} > /tmp/${DB_NAME}.sql"
    scp -P ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP}:/tmp/${DB_NAME}.sql /var/www/html/.ddev/.downloads/db.sql
    gzip -f /var/www/html/.ddev/.downloads/db.sql

files_pull_command:
  command: |
    scp -P ${SSH_PORT} -r ${SSH_USERNAME}@${HOST_IP}:/path/to/public/fileadmin/user_upload /var/www/html/.ddev/.downloads/files

Do I declare the variables the wrong way? Or what is it that I'm missing?

For anyone who has trouble connecting via ssh without the password promt, you can run the following commands:

  1. ssh-keygen -t rsa
  2. ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 username@host
  3. Afterward you should be able to connect without a password promt. Try the following: ssh -p 22 username@host
  4. before you try to ddev puul you have to execute ddev auth ssh
Solution

  • Thanks to @rfay for pointing me into the right direction.

    The Problem was, that my password containted a special charater (not a $ though) which needed to be escaped.

    After escpaing it correctly like so

    environment_variables:
  DB_PASSWORD: 'Password\&\'

    the ddev pull works.

    I hope my .yaml file helps someone else that needs to pull from a webserver.