Search code examples
oracle-databaseplsqlsqlplusspool

SQLPlus - spooling to multiple files from PL/SQL blocks


I have a query that returns a lot of data into a CSV file. So much, in fact, that Excel can't open it - there are too many rows. Is there a way to control spool to spool to a new file everytime 65000 rows have been processed? Ideally, I'd like to have my output in files named in sequence, such as large_data_1.csv, large_data_2.csv, large_data_3.csv, etc...

I could use dbms_output in a PL/SQL block to control how many rows are output, but then how would I switch files, as spool does not seem to be accessible from PL/SQL blocks?

(Oracle 10g)

UPDATE:

I don't have access to the server, so writing files to the server would probably not work.

UPDATE 2:

Some of the fields contain free-form text, including linebreaks, so counting line breaks AFTER the file is written is not as easy as counting records WHILE the data is being returned...


Solution

  • Got a solution, don't know why I didn't think of this sooner...

    The basic idea is that the master sqplplus script generates an intermediate script that will split the output to multiple files. Executing the intermediate script will execute multiple queries with different ranges imposed on rownum, and spool to a different file for each query.

    set termout off
    set serveroutput on
    set echo off
    set feedback off
    variable v_rowCount number;
    spool intermediate_file.sql
    declare
         i number := 0;
         v_fileNum number := 1;
         v_range_start number := 1;
         v_range_end number := 1;
         k_max_rows constant number := 65536;
    begin
        dbms_output.enable(10000);
        select count(*) 
        into :v_err_count
        from ...
        /* You don't need to see the details of the query... */
    
        while i <= :v_err_count loop
    
              v_range_start := i+1;
              if v_range_start <= :v_err_count then
                i := i+k_max_rows;
                v_range_end := i;
    
                dbms_output.put_line('set colsep ,  
    set pagesize 0
    set trimspool on 
    set headsep off
    set feedback off
    set echo off
    set termout off
    set linesize 4000
    spool large_data_file_'||v_fileNum||'.csv
    select data_string
    from (select rownum rn, data_object
          from 
          /* Details of query omitted */
         )
    where rn >= '||v_range_start||' and rn <= '||v_range_end||';
    spool off');
              v_fileNum := v_fileNum +1;
             end if;
        end loop;
    end;
    /
    spool off
    prompt     executing intermediate file
    @intermediate_file.sql;
    set serveroutput off