Search code examples
windowssplitsyntaxyamlyq

yq wont recognize single quotation marks around the field when trying to split large YAML file into smaller ones

I have a large YAML file containing info about video games.

---
-
    Name: Metal Gear Solid HD Collection
    Console: PS3
    Genre: Stealth 
    Progression: Unplayed
    Maintenance: Perfect!
-
    Name: Metal Gear Solid 4 Guns of the Patriots
    Console: PS3
    Genre: Stealth 
    Progression: Unplayed
    Maintenance: Perfect!
-
    Name: Metal Gear Rising Revengeance
    Console: PS3
    Genre: Power Fantasy
    Progression: Compleat (Blade Wold DLC to play)
    Maintenance: Perfect!
-

I am trying to split the large file into a file per game. I found a great solution to this with using:

yq e GameCollection.yaml -s .Name

But this gives the following error

Error: cannot index array with 'Name' (strconv.ParseInt: parsing "Name": invalid syntax)

I'm assuming that this is because it is trying to treat Name as a string so I wrap it in single quotes to make it more like a String. I found this solution here, (in the comments under the most upvoted answer), which allowed me to see a different error message and move past this issue.

yq e GameCollection.yaml -s '.Name'

This gives me a different Error and I think it's because Name is not a unique identifier as there are sometimes the same game on multiple consoles (such as Skyrim)

Error: bad split document expression: 1:1: invalid input text "'.Name'"

I found, what I thought was a great solution to this, by adding an index so that duplicate names can just increment! I found that solution here.

yq e GameCollection.yaml -s '.Name + "_" + $index'

But it gives me a similar error.

Error: bad split document expression: 1:1: invalid input text "'.Name"

I tried again a few times as I think I just don't understand the syntax.

yq e GameCollection.yaml -s '.Name' + "_" + '$index'
Error: bad split document expression: 1:1: invalid input text "'.Name'"

yq e GameCollection.yaml -s '.Name' + "_" + $index
Error: bad split document expression: 1:1: invalid input text "'.Name'"

yq e GameCollection.yaml -s .Name + "_" + '$index'
Error: cannot index array with 'Name' (strconv.ParseInt: parsing "Name": invalid syntax)

It's almost like it doesn't like the single quotes.

What can I try to fix it?

Solution

  • The problem has nothing to do with the use of quotation marks. The cause is much simpler: your file contains documents that are single-element arrays. So there is no .Name at the top level, but it's a key to the first element of each array. Therefore, you need to specify it like this:

    yq e .[] GameCollection.yaml -s .Name

    Some background info

    Compare your file with the one from the other question you looked at.

    The one from the other question:

    ---
name: first
metadata: a

    Your file:

    ---
-
    Name: Metal Gear Solid HD Collection

    Notice the extra - line: it's significant.

    Another way to debug the issue is to convert your file into JSON format, which some people find clearer and helps to show structure more explicitly:

    yq -o json GameCollection.yaml

    Note all the [ ... ] showing that these are arrays.