Search code examples
rtidy

R: using gather to clean up a dataset

I have a csv dataset from the USDA that has education levels obtained by adults by county in the US for 1970, 1980, 1990 and 2000. I have imported this csv using the read_csv function, I then clean up the dataset like so:

colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "State"] <- "state"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Area name"] <- "area_name"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Less than a high school diploma, 1970"] <- "Less Than Diploma, 1970"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "High school diploma only, 1970"] <- "Diploma, 1970"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Some college (1-3 years), 1970"] <- "AA or more, 1970"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Four years of college or higher, 1970"] <- "BA or more, 1970"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with less than a high school diploma, 1970"] <- "%Less Than Diploma, 1970"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with a high school diploma only, 1970"] <- "% Diploma, 1970"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults completing some college (1-3 years), 1970"] <- "% AA or more, 1970"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults completing four years of college or higher, 1970"] <- "% BA or more, 1970"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Less than a high school diploma, 1980"] <- "Less Than Diploma, 1980"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "High school diploma only, 1980"] <- "Diploma, 1980" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Some college (1-3 years), 1980"] <- "AA or more, 1980" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Four years of college or higher, 1980"] <- "BA or more, 1980" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with less than a high school diploma, 1980"] <- "% Less Than Diploma, 1980" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with a high school diploma only, 1980"] <- "% Diploma, 1980" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults completing some college (1-3 years), 1980"] <- "% AA or more, 1980" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults completing four years of college or higher, 1980"] <- "% BA or more, 1980"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Less than a high school diploma, 1990"] <- "Less Than Diploma, 1990"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "High school diploma only, 1990"] <- "Diploma, 1990" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Some college or associate's degree, 1990"] <- "AA or more, 1990" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Bachelor's degree or higher, 1990"] <- "BA or more, 1990" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with less than a high school diploma, 1990"] <- "% Less Than Diploma, 1990" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with a high school diploma only, 1990"] <- "% Diploma, 1990" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults completing some college or associate's degree, 1990"] <- "% AA or more, 1990" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with a bachelor's degree or higher, 1990"] <- "% BA or more, 1990"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Less than a high school diploma, 2000"] <- "Less Than Diploma, 2000"
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "High school diploma only, 2000"] <- "Diploma, 2000" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Some college or associate's degree, 2000"] <- "AA or more, 2000" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Bachelor's degree or higher, 2000"] <- "BA or more, 2000" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with less than a high school diploma, 2000"] <- "% Less Than Diploma, 2000" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with a high school diploma only, 2000"] <- "% Diploma, 2000" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults completing some college or associate's degree, 2000"] <- "% AA or more, 2000" 
colnames(eduLevelsbyCounty)[colnames(eduLevelsbyCounty) == "Percent of adults with a bachelor's degree or higher, 2000"] <- "% BA or more, 2000"

So now I have a very large tibble, but the problem is I would now like to clean it up further by separating year into it's own column and the name of the education level reached in the other respective columns. I know gather() can kind of accomplish what I am trying to do, but the problem is my dataset contains multiple years: 1970, 1980, 1990 and 2000.

I hope I have made this clear, if not I can add information as necessary. Any help would be greatly appreciated.

Solution

  • I feel that the way you naming variables makes it unnecessarily complicated. Otherwise, privot_longer, a newer function to replace gather may solve this problem. I have renamed your original names a little:

    using pivot_longer to pivot data from wide to long

    library(tidyverse)
long<-pivot_longer(df, -c("state", "area_name"),
            names_to = c(".value", "year"), 
            names_sep = "_", values_drop_na = TRUE) 

    > long              
# A tibble: 4 x 11
  state area_name year  Less.Than.Diploma Diploma AA.or.more BA.or.more percent.Less.Than.D~ percent.Diploma percent.AA.or.m~ percent.BA.or.m~
  <dbl>     <dbl> <chr>             <dbl>   <dbl>      <dbl>      <dbl>                <dbl>           <dbl>            <dbl>            <dbl>
1     1         2 1970                 71      72         73         74                   75              76               77               78
2     1         2 1980                 81      82         83         84                   85              86               87               88
3     1         2 1990                 91      92         93         94                   95              96               97               98
4     1         2 2000                 21      22         23         24                   25              26               27               28
>

    Data

    df <-data.frame(
  "state" = 1, 
  "area_name" =2,
  "Less Than Diploma_1970" = 71,
  "Diploma_1970" = 72,
  "AA or more_1970"  = 73,
  "BA or more_1970"  = 74,
  "percent Less Than Diploma_1970"  = 75,
  "percent Diploma_1970"  = 76,
  "percent AA or more_1970"  = 77,
  "percent BA or more_1970"  = 78,
  "Less Than Diploma_1980"  = 81,
  "Diploma_1980" = 82,
  "AA or more_1980" = 83, 
  "BA or more_1980" = 84, 
  "percent Less Than Diploma_1980" = 85, 
  "percent Diploma_1980" = 86, 
  "percent AA or more_1980" = 87, 
  "percent BA or more_1980" = 88,
  "Less Than Diploma_1990" = 91,
  "Diploma_1990" = 92, 
  "AA or more_1990" = 93, 
  "BA or more_1990" = 94,
  "percent Less Than Diploma_1990" = 95 ,
  "percent Diploma_1990" = 96, 
  "percent AA or more_1990"= 97, 
  "percent BA or more_1990" = 98,
  "Less Than Diploma_2000" = 21,
  "Diploma_2000"  = 22, 
  "AA or more_2000"  = 23, 
  "BA or more_2000"  = 24, 
  "percent Less Than Diploma_2000"  = 25, 
  "percent Diploma_2000"  = 26, 
  "percent AA or more_2000"  = 27, 
  "percent BA or more_2000"  = 28)