Search code examples
ssisoledbfoxprodataflowtruncation

SSIS OLE DB Data Flow Source: outputting a column that may have different lengths

I created an SSIS package so I can import data from a legacy FoxPro database at scheduled intervals. A copy of the FoxPro databaseis installed for several customers. Overall, the package is working very well and accomplishing all that I need.

However, I have one annoying situation where at least one customer (maybe more) has a modified FP database, where they increased the length of one column in one table. When I run the package on such a customer, it fails because of truncation.

I thought I could just give myself some wiggle room and change the length from 3 to 10. That way the mutants with a length of 10 would be accommodated, as well as everyone else using 3. However, SSIS complains when the column lengths don't match, period.

I suppose I have a few options:

  1. On the task, set 'ValidateExternalMetadata' to false. However, I'm not sure that is the most responsible option... or is it?
  2. Get our implementation team to change the length to 10 for all customers. This could be a problem, but at least it would be their problem.
  3. Create a copy of the task that works for solutions with the different column length. Implementation will likely use the wrong package at some point, and everyone will ask me why I didn't just give them a single package that couldn't handle all scenarios and blame this on me.
  4. Use some other approach you might be able to fill me in on.
Solution

  • If you are using the Visual FoxPro OleDB, and you are concerned about the columns widths, you can explicitly force them by using PADR() during your call. I don't know how many tables / queries this impacts but would guarantee you get your expected character column lengths. If dealing with numeric, decimal, date/time, logical (boolean), should not be an issue... Anyhow, you could do this as your select to get the data

    select 
      t1.Fld1,
      t1.Fld2,
      padr( t1.CharFld3, 20 ) CharFld3,
      padr( t1.CharFld4, 5 ) CharFld4,
      t1.OtherFld5,
      padr( t1.CharFld6, 35 ) CharFld5
   from
      YourTable t1
   where
      SomeCondition

    This will force character based (implied sample) fields "CharFld3", "CharFld4", "CharFld6" to a force width of 20, 5 and 35 respectively regardless of the underlying structure length. Now, if someone updates the structure LONGER than what you have it will be truncated down to proper length, but won't crash. Additionally, if they have a shorter column length, it will be padded out to the full size you specify via the PADR() function (pad right).