Search code examples
node.jsstreamfast-csv

How to identify different input schemas with FastCSV?


The following code works fine with the "Winner" type. The tech is typescript with node streams.

Sometimes someone uploads a winner-2 type though. I'd like to look at the header and change format type based on the header.

I could possibly

  • write a function that reads the header and returns the stream based on 'parse'. This is part of a class, so I could set the type. It would take a row and return one.
  • make the specification of Winner|Winner2 and see what happens. Look at the result in transform
  • make an uber winner interface and pull out the values that are set.

I'm planning on rewriting the headers as there are inconsistencies.

How to solve the problem of normalising these different CSV inputs into one idealised structure? rxjs?

import {parse, RowTransformCallback} from "@fast-csv/parse";

 stream
     .pipe(parse({headers: true}))
     .pipe(format<Winner, Winner>({headers: true}))
     .transform((row, next): void => {
           this.processRow(row, next)
  })
     .pipe(process.stdout)
     .on('error', reject)
            .on('end',
                (rowCount: number) => console.log(`Parsed ${rowCount} rows`));
    });

Solution

  • I rewrote the headers in a map function to take schema a and schema b and turn them into the target schema

     stream
        .pipe(parse({
          headers: headers => headers.map(h => {
            if (h === 'Email') {
               return 'email'
            }
            if (h === 'Firstname') {
               return 'firstName'                  
            }
            return h
         }),
        }))