Search code examples
csvf#data-cleaningf#-data

F#: CSV cleaning

I'm trying using F# to clean a CSV dataset. For example I want to change string values in one column to lower case. I don't know if it's better to work with loaded data as CsvProvider Rows or I should create some struct and convert these Rows to struct list. Either way, how can I load CSV, modify values in one column and save the modified dataset to CSV again?

I've been searching the FSharp.Data documentation and other pages and I didn't find any relevant examples.

Solution

  • The CSV type provider is great for reading CSV data, but it is less easy to use for data cleaning and transformations. For this, I'd recommend using the Deedle data exploration library instead. With Deedle, you can do this easily:

    #r "nuget:Deedle"
open Deedle

let df = Frame.ReadCsv(@"C:\Temp\titanic.csv")
let newName = 
  df.GetColumn<string>("Name") 
  |> Series.mapValues (fun s -> s.ToLower())
df.ReplaceColumn("Name", newName)
df.SaveCsv(@"C:\Temp\titanic.csv")