How to get files recursively from and FTPS server with command line?

Question

I need to download a file structure (over 4GB) from an FTPS server (implicit FTP over TLS protocol). Unfortunately wget doesn't support FTPS, but curl does. However curl doesn't support recursion.

I need a command line tool solution on Ubuntu.

Any idea?

Answer 1

No solution found, so I wrote a simple ruby script, which can be a starting point for everybody:

#!/usr/bin/env ruby
# encoding: utf-8

require 'optparse'

FILENAME_MATCH_REGEXP=/^[d\-rwx]+ [0-9]+ \w+ \w+[ ]+\d+[ ]+[a-zA-Z]+[ ]+\d+[ ]+\d+[ ]+(.*)$/

options = {}
opts = OptionParser.new do |opts|
  opts.banner = "Usage: curl_ftp_get_recursive.rb [options] url directory"

  opts.on("-v", "--[no-]verbose", "Run verbosely") do |o|
    options[:verbose] = o
  end

  options[:curl_options] = '-sS'
  opts.on("-c", "--curl-options OPTIONS", "Curl options") do |o|
    options[:curl_options] += " #{o}"
  end

  options[:output_dir] = '.'
  opts.on("-o", "--output DIR", "Output directory, default=current directory") do |o|
    options[:output_dir] = o
  end
end

begin
  opts.parse!
  raise "Invalid number of arguments" if ARGV.count != 2
rescue Exception => e
  p "#{$0}: #{e.message}"
  p opts
  exit 1
end

# Remove trailing '/' if any
url = ARGV[0].sub /\/*$/, ''
root_dir = ARGV[1].sub /\/*$/, ''
options[:output_dir] = options[:output_dir].sub /\/*$/, ''

def get_dir options, url, dir
  p "Reading directory '#{dir}'..." if options[:verbose]

  output = `/usr/bin/curl #{options[:curl_options]} "#{url}/#{dir}"/`
  exit 1 if $? != 0
  dir_list = output.split "\n"
  dir_list.each do |line|
    p "Processing line '#{line}'..." if options[:verbose]
    file_name = "#{dir}/#{line.match(FILENAME_MATCH_REGEXP)[1]}"
    local_dir = "#{options[:output_dir]}/#{dir}"
    if line.match /^d/
      get_dir options, url, file_name
    else
      p "Getting file '#{file_name}'..."
      `mkdir -p "#{local_dir}"`
      exit 1 if $? != 0
      `/usr/bin/curl -o "#{options[:output_dir]}/#{file_name}" #{options[:curl_options]} "#{url}/#{file_name}"`
      exit 1 if $? != 0
    end
  end
end

get_dir options, url, root_dir

A usage example:

./curl_ftp_get_recursive.rb -v -c --insecure ftps://MyUSerName:MayPassword@MyHost:MyPort '/Directory to copy'