3.4.1

This is to patch a high security vulnerability.
Closes #550

Signed-off-by: Peter Benjamin <petermbenjamin@gmail.com>

.gitbook/assets/convert.md

@@ -1,60 +0,0 @@
-## Yaml to Json
-To convert output to json, use the --tojson (or -j) flag. This is supported by all commands.
-
-Each matching yaml node will be converted to json and printed out on a separate line.
-
-Given a sample.yaml file of:
-```yaml
-b:
-  c: 2
-```
-then
-```bash
-yq r -j sample.yaml
-```
-
-will output
-```json
-{"b":{"c":2}}
-```
-
-Given a sample.yaml file of:
-```yaml
-bob:
-  c: 2
-bab:
-  c: 5
-```
-then
-```bash
-yq r -j sample.yaml b*
-```
-
-will output
-```json
-{"c":2}
-{"c":5}
-```
-
-## Json to Yaml
-To read in json, just pass in a json file instead of yaml, it will just work :)
-
-e.g given a json file
-
-```json
-{"a":"Easy! as one two three","b":{"c":2,"d":[3,4]}}
-```
-then
-```bash
-yq r sample.json
-```
-will output
-```yaml
-a: Easy! as one two three
-b:
-  c: 2
-  d:
-  - 3
-  - 4
-```
-

.github/workflows/go.yml

@@ -7,10 +7,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
 
-    - name: Set up Go 1.14
+    - name: Set up Go 1.15
       uses: actions/setup-go@v1
       with:
-        go-version: 1.14
+        go-version: 1.15
       id: go
 
     - name: Check out code into the Go module directory

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.14 as builder
+FROM golang:1.15 as builder
 
 WORKDIR /go/src/mikefarah/yq
 
@@ -12,7 +12,7 @@ RUN CGO_ENABLED=0 make local build
 
 # Choose alpine as a base image to make this useful for CI, as many
 # CI tools expect an interactive shell inside the container
-FROM alpine:3.8 as production
+FROM alpine:3.12 as production
 
 COPY --from=builder /go/src/mikefarah/yq/yq /usr/bin/yq
 RUN chmod +x /usr/bin/yq

Dockerfile.dev

@@ -1,4 +1,4 @@
-FROM golang:1.14
+FROM golang:1.15
 
 COPY scripts/devtools.sh /opt/devtools.sh
 

README.md

@@ -1,92 +1,166 @@
----
-description: yq is a lightweight and portable command-line YAML processor
----
-
 # yq
 
- ![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg) ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases \(by Release\)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq)
+![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq)
+
+
+a lightweight and portable command-line YAML processor
+
+The aim of the project is to be the [jq](https://github.com/stedolan/jq) or sed of yaml files.
 
 ## Install
 
-`yq` has pre-built binaries for most platforms - checkout the [releases page](https://github.com/mikefarah/yq/releases) for the latest build. Alternatively - you can use one of the methods below:
+### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
 
-### On MacOS:
-
-```bash
+### MacOS:
+```
 brew install yq
 ```
 
-### On Windows:
-
-```bash
-choco install yq
+### Ubuntu and other Linux distros supporting `snap` packages:
 ```
-
-Kindly maintained by @chillum \([https://github.com/chillum/choco-packages/tree/master/yq](https://github.com/chillum/choco-packages/tree/master/yq)\)
-
-### On Ubuntu and other Linux distributions supporting `snap` packages:
-
-```bash
 snap install yq
 ```
 
 #### Snap notes
+`yq` installs with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can:
 
-`yq` installs with with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can:
-
-```bash
-sudo cat /etc/myfile | yq -r - somecommand
+```
+sudo cat /etc/myfile | yq r - a.path
 ```
 
 And to write to a root file you can either use [sponge](https://linux.die.net/man/1/sponge):
-
-```bash
-sudo cat /etc/myfile | yq -r - somecommand | sudo sponge /etc/myfile
 ```
-
+sudo cat /etc/myfile | yq w - a.path value | sudo sponge /etc/myfile
+```
 or write to a temporary file:
-
-```bash
-sudo cat /etc/myfile | yq -r - somecommand | sudo tee /etc/myfile.tmp
+```
+sudo cat /etc/myfile | yq w - a.path value | sudo tee /etc/myfile.tmp
 sudo mv /etc/myfile.tmp /etc/myfile
 rm /etc/myfile.tmp
 ```
 
-### On Ubuntu 16.04 or higher from Debian package:
+### wget
+
+Use wget to download the pre-compiled binaries:
 
 ```bash
-sudo add-apt-repository ppa:rmescandon/yq
-sudo apt update
-sudo apt install yq -y
+wget https://github.com/mikefarah/yq/releases/download/{VERSION}/{BINARY} -O /usr/bin/yq &&\
+    chmod +x /usr/bin/yq
 ```
 
-Kindly maintained by @rmescandon
+For instance, VERSION=3.4.0 and BINARY=yq_linux_amd64
 
-### go get:
 
-```text
-GO111MODULE=on go get github.com/mikefarah/yq/v3
-```
+### Run with Docker
 
-## Docker
-
-Oneshot use:
+#### Oneshot use:
 
 ```bash
-docker run --rm -v ${PWD}:/workdir mikefarah/yq yq [flags] <command> FILE...
+docker run --rm -v "${PWD}":/workdir mikefarah/yq yq [flags] <command> FILE...
 ```
 
-Run commands interactively:
+#### Run commands interactively:
 
 ```bash
-docker run --rm -it -v ${PWD}:/workdir mikefarah/yq sh
+docker run --rm -it -v "${PWD}":/workdir mikefarah/yq sh
 ```
 
 It can be useful to have a bash function to avoid typing the whole docker command:
 
 ```bash
 yq() {
-  docker run --rm -i -v ${PWD}:/workdir mikefarah/yq yq $@
+  docker run --rm -i -v "${PWD}":/workdir mikefarah/yq yq "$@"
 }
 ```
 
+### Go Get:
+```
+GO111MODULE=on go get github.com/mikefarah/yq/v3
+```
+
+## Community Supported Installation methods
+As these are supported by the community :heart: - however, they may be out of date with the officially supported releases.
+
+
+### Windows:
+```
+choco install yq
+```
+Supported by @chillum (https://chocolatey.org/packages/yq)
+
+### Alpine Linux
+- Enable edge/community repo by adding ```$MIRROR/alpine/edge/community``` to ```/etc/apk/repositories```
+- Update database index with ```apk update```
+- Install yq with ```apk add yq```
+
+Supported by Tuan Hoang
+https://pkgs.alpinelinux.org/package/edge/community/x86/yq
+
+
+### On Ubuntu 16.04 or higher from Debian package:
+```sh
+sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CC86BB64
+sudo add-apt-repository ppa:rmescandon/yq
+sudo apt update
+sudo apt install yq -y
+```
+Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
+
+## Features
+- Written in portable go, so you can download a lovely dependency free binary
+- [Colorize the output](https://mikefarah.gitbook.io/yq/usage/output-format#colorize-output)
+- [Deep read a yaml file with a given path expression](https://mikefarah.gitbook.io/yq/commands/read#basic)
+- [List matching paths of a given path expression](https://mikefarah.gitbook.io/yq/commands/read#path-only)
+- [Return the lengths of arrays/object/scalars](https://mikefarah.gitbook.io/yq/commands/read#printing-length-of-the-results)
+- Update a yaml file given a [path expression](https://mikefarah.gitbook.io/yq/commands/write-update#basic) or [script file](https://mikefarah.gitbook.io/yq/commands/write-update#basic)
+- Update creates any missing entries in the path on the fly
+- Deeply [compare](https://mikefarah.gitbook.io/yq/commands/compare) yaml files
+- Keeps yaml formatting and comments when updating
+- [Validate a yaml file](https://mikefarah.gitbook.io/yq/commands/validate)
+- Create a yaml file given a [deep path and value](https://mikefarah.gitbook.io/yq/commands/create#creating-a-simple-yaml-file) or a [script file](https://mikefarah.gitbook.io/yq/commands/create#creating-using-a-create-script)
+- [Prefix a path to a yaml file](https://mikefarah.gitbook.io/yq/commands/prefix)
+- [Convert to/from json to yaml](https://mikefarah.gitbook.io/yq/usage/convert)
+- [Pipe data in by using '-'](https://mikefarah.gitbook.io/yq/commands/read#from-stdin)
+- [Merge](https://mikefarah.gitbook.io/yq/commands/merge) multiple yaml files with various options for [overriding](https://mikefarah.gitbook.io/yq/commands/merge#overwrite-values) and [appending](https://mikefarah.gitbook.io/yq/commands/merge#append-values-with-arrays)
+- Supports multiple documents in a single yaml file for [reading](https://mikefarah.gitbook.io/yq/commands/read#multiple-documents), [writing](https://mikefarah.gitbook.io/yq/commands/write-update#multiple-documents) and [merging](https://mikefarah.gitbook.io/yq/commands/merge#multiple-documents)
+- General shell completion scripts (bash/zsh/fish/powershell) (https://mikefarah.gitbook.io/yq/commands/shell-completion)
+
+## [Usage](https://mikefarah.gitbook.io/yq/)
+
+Check out the [documentation](https://mikefarah.gitbook.io/yq/) for more detailed and advanced usage.
+
+```
+Usage:
+  yq [flags]
+  yq [command]
+
+Available Commands:
+  compare          yq x [--prettyPrint/-P] dataA.yaml dataB.yaml 'b.e(name==fr*).value'
+  delete           yq d [--inplace/-i] [--doc/-d index] sample.yaml 'b.e(name==fred)'
+  help             Help about any command
+  merge            yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--append/-a] sample.yaml sample2.yaml
+  new              yq n [--script/-s script_file] a.b.c newValue
+  prefix           yq p [--inplace/-i] [--doc/-d index] sample.yaml a.b.c
+  read             yq r [--printMode/-p pv] sample.yaml 'b.e(name==fr*).value'
+  shell-completion Generates shell completion scripts
+  validate         yq v sample.yaml
+  write            yq w [--inplace/-i] [--script/-s script_file] [--doc/-d index] sample.yaml 'b.e(name==fr*).value' newValue
+
+Flags:
+  -C, --colors        print with colors
+  -h, --help          help for yq
+  -I, --indent int    sets indent level for output (default 2)
+  -P, --prettyPrint   pretty print
+  -j, --tojson        output as json. By default it prints a json document in one line, use the prettyPrint flag to print a formatted doc.
+  -v, --verbose       verbose mode
+  -V, --version       Print version information and quit
+
+Use "yq [command] --help" for more information about a command.
+```
+
+## Upgrade from V2
+If you've been using v2 and want/need to upgrade, checkout the [upgrade guide](https://mikefarah.gitbook.io/yq/upgrading-from-v2).
+
+## Known Issues / Missing Features
+- `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
+- You cannot (yet) select multiple paths/keys from the yaml to be printed out (https://github.com/mikefarah/yq/issues/287)

SUMMARY.md

@@ -1,25 +0,0 @@
-# Table of contents
-
-* [yq](README.md)
-* [Upgrading from V2](upgrading-from-v2.md)
-
-## Commands
-
-* [Read](commands/read.md)
-* [Validate](commands/validate.md)
-* [Compare](commands/compare.md)
-* [Write](commands/write-update.md)
-* [Create](commands/create.md)
-* [Delete](commands/delete.md)
-* [Merge](commands/merge.md)
-* [Prefix](commands/prefix.md)
-* [Shell Completion](commands/shell-completion.md)
-
-## Usage
-
-* [Output format](usage/output-format.md)
-* [Path Expressions](usage/path-expressions.md)
-* [Value Parsing](usage/value-parsing.md)
-* [Working with JSON](usage/convert.md)
-* [Github Page](https://github.com/mikefarah/yq)
-

cmd/constant.go

@@ -26,7 +26,8 @@ var defaultValue = ""
 var indent = 2
 var overwriteFlag = false
 var autoCreateFlag = true
-var appendFlag = false
+var arrayMergeStrategyFlag = "update"
+var commentsMergeStrategyFlag = "setWhenBlank"
 var verbose = false
 var version = false
 var docIndex = "0"

cmd/merge.go

@@ -11,14 +11,14 @@ func createMergeCmd() *cobra.Command {
 	var cmdMerge = &cobra.Command{
 		Use:     "merge [initial_yaml_file] [additional_yaml_file]...",
 		Aliases: []string{"m"},
-		Short:   "yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--append/-a] sample.yaml sample2.yaml",
+		Short:   "yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--arrayMerge/-a strategy] sample.yaml sample2.yaml",
 		Example: `
 yq merge things.yaml other.yaml
 yq merge --inplace things.yaml other.yaml
 yq m -i things.yaml other.yaml
 yq m --overwrite things.yaml other.yaml
 yq m -i -x things.yaml other.yaml
-yq m -i -a things.yaml other.yaml
+yq m -i -a=append things.yaml other.yaml
 yq m -i --autocreate=false things.yaml other.yaml
       `,
 		Long: `Updates the yaml file by adding/updating the path(s) and value(s) from additional yaml file(s).
@@ -32,7 +32,17 @@ If append flag is set then existing arrays will be merged with the arrays from e
 	cmdMerge.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the yaml file inplace")
 	cmdMerge.PersistentFlags().BoolVarP(&overwriteFlag, "overwrite", "x", false, "update the yaml file by overwriting existing values")
 	cmdMerge.PersistentFlags().BoolVarP(&autoCreateFlag, "autocreate", "c", true, "automatically create any missing entries")
-	cmdMerge.PersistentFlags().BoolVarP(&appendFlag, "append", "a", false, "update the yaml file by appending array values")
+	cmdMerge.PersistentFlags().StringVarP(&arrayMergeStrategyFlag, "arrays", "a", "update", `array merge strategy (update/append/overwrite)
+update: recursively update arrays by their index
+append: concatenate arrays together
+overwrite: replace arrays
+`)
+	cmdMerge.PersistentFlags().StringVarP(&commentsMergeStrategyFlag, "comments", "", "setWhenBlank", `comments merge strategy (setWhenBlank/ignore/append/overwrite)
+setWhenBlank: set comment if the original document has no comment at that node
+ignore: leave comments as-is in the original
+append: append comments together
+overwrite: overwrite comments completely
+`)
 	cmdMerge.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
 	return cmdMerge
 }
@@ -41,9 +51,9 @@ If append flag is set then existing arrays will be merged with the arrays from e
 * We don't deeply traverse arrays when appending a merge, instead we want to
 * append the entire array element.
  */
-func createReadFunctionForMerge() func(*yaml.Node) ([]*yqlib.NodeContext, error) {
+func createReadFunctionForMerge(arrayMergeStrategy yqlib.ArrayMergeStrategy) func(*yaml.Node) ([]*yqlib.NodeContext, error) {
 	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
-		return lib.Get(dataBucket, "**", !appendFlag)
+		return lib.GetForMerge(dataBucket, "**", arrayMergeStrategy)
 	}
 }
 
@@ -53,19 +63,59 @@ func mergeProperties(cmd *cobra.Command, args []string) error {
 	if len(args) < 1 {
 		return errors.New("Must provide at least 1 yaml file")
 	}
+	var arrayMergeStrategy yqlib.ArrayMergeStrategy
+
+	switch arrayMergeStrategyFlag {
+	case "update":
+		arrayMergeStrategy = yqlib.UpdateArrayMergeStrategy
+	case "append":
+		arrayMergeStrategy = yqlib.AppendArrayMergeStrategy
+	case "overwrite":
+		arrayMergeStrategy = yqlib.OverwriteArrayMergeStrategy
+	default:
+		return errors.New("Array merge strategy must be one of: update/append/overwrite")
+	}
+
+	var commentsMergeStrategy yqlib.CommentsMergeStrategy
+
+	switch commentsMergeStrategyFlag {
+	case "setWhenBlank":
+		commentsMergeStrategy = yqlib.SetWhenBlankCommentsMergeStrategy
+	case "ignore":
+		commentsMergeStrategy = yqlib.IgnoreCommentsMergeStrategy
+	case "append":
+		commentsMergeStrategy = yqlib.AppendCommentsMergeStrategy
+	case "overwrite":
+		commentsMergeStrategy = yqlib.OverwriteCommentsMergeStrategy
+	default:
+		return errors.New("Comments merge strategy must be one of: setWhenBlank/ignore/append/overwrite")
+	}
 
 	if len(args) > 1 {
 		// first generate update commands from the file
 		var filesToMerge = args[1:]
 
 		for _, fileToMerge := range filesToMerge {
-			matchingNodes, errorProcessingFile := doReadYamlFile(fileToMerge, createReadFunctionForMerge(), false, 0)
+			matchingNodes, errorProcessingFile := doReadYamlFile(fileToMerge, createReadFunctionForMerge(arrayMergeStrategy), false, 0)
 			if errorProcessingFile != nil {
 				return errorProcessingFile
 			}
+			log.Debugf("finished reading for merge!")
 			for _, matchingNode := range matchingNodes {
-				mergePath := lib.MergePathStackToString(matchingNode.PathStack, appendFlag)
-				updateCommands = append(updateCommands, yqlib.UpdateCommand{Command: "update", Path: mergePath, Value: matchingNode.Node, Overwrite: overwriteFlag})
+				log.Debugf("matched node %v", lib.PathStackToString(matchingNode.PathStack))
+				yqlib.DebugNode(matchingNode.Node)
+			}
+			for _, matchingNode := range matchingNodes {
+				mergePath := lib.MergePathStackToString(matchingNode.PathStack, arrayMergeStrategy)
+				updateCommands = append(updateCommands, yqlib.UpdateCommand{
+					Command:               "merge",
+					Path:                  mergePath,
+					Value:                 matchingNode.Node,
+					Overwrite:             overwriteFlag,
+					CommentsMergeStrategy: commentsMergeStrategy,
+					// dont update the content for nodes midway, only leaf nodes
+					DontUpdateNodeContent: matchingNode.IsMiddleNode && (arrayMergeStrategy != yqlib.OverwriteArrayMergeStrategy || matchingNode.Node.Kind != yaml.SequenceNode),
+				})
 			}
 		}
 	}

cmd/merge_test.go

@@ -60,7 +60,7 @@ func TestMergeOverwriteCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `a: other # better than the original
+	expectedOutput := `a: other # just the best
 b: [3, 4]
 c:
   test: 1
@@ -68,9 +68,36 @@ c:
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestMergeOverwriteDeepExampleCmd(t *testing.T) {
+	content := `c:
+  test: 1
+  thing: whatever
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `c:
+  test: 5
+`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --autocreate=false --overwrite %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `c:
+  test: 5
+  thing: whatever
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestMergeAppendCmd(t *testing.T) {
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --autocreate=false --append ../examples/data1.yaml ../examples/data2.yaml")
+	result := test.RunCmd(cmd, "merge --autocreate=false --arrays=append ../examples/data1.yaml ../examples/data2.yaml")
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
@@ -96,7 +123,7 @@ func TestMergeAppendArraysCmd(t *testing.T) {
 	defer test.RemoveTempYamlFile(mergeFilename)
 
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("merge --append -d* %s %s", filename, mergeFilename))
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --arrays=append -d* %s %s", filename, mergeFilename))
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
@@ -109,13 +136,56 @@ func TestMergeAppendArraysCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
-func TestMergeOverwriteAndAppendCmd(t *testing.T) {
+func TestMergeAliasArraysCmd(t *testing.T) {
+	content := `
+vars:
+  variable1: &var1 cat
+
+usage:
+  value1: *var1
+  valueAnother: *var1
+  valuePlain: thing
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `
+vars:
+  variable2: &var2 puppy
+
+usage:
+  value2: *var2
+  valueAnother: *var2
+  valuePlain: *var2
+`
+
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --autocreate=false --append --overwrite ../examples/data1.yaml ../examples/data2.yaml")
+	result := test.RunCmd(cmd, fmt.Sprintf("merge -x %s %s", filename, mergeFilename))
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `a: other # better than the original
+	expectedOutput := `vars:
+  variable1: &var1 cat
+  variable2: &var2 puppy
+usage:
+  value1: *var1
+  valueAnother: *var2
+  valuePlain: *var2
+  value2: *var2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOverwriteAndAppendCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --autocreate=false --arrays=append --overwrite ../examples/data1.yaml ../examples/data2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: other # just the best
 b: [1, 2, 3, 4]
 c:
   test: 1
@@ -123,13 +193,161 @@ c:
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
-func TestMergeArraysCmd(t *testing.T) {
+var commentContentA = `
+a: valueA1 # commentA1
+b: valueB1
+`
+
+var commentContentB = `
+a: valueA2 # commentA2
+b: valueB2 # commentB2
+c: valueC2 # commentC2
+`
+
+func TestMergeCommentsSetWhenBlankCmd(t *testing.T) {
+	filename := test.WriteTempYamlFile(commentContentA)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeFilename := test.WriteTempYamlFile(commentContentB)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --append ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --comments=setWhenBlank %s %s", filename, mergeFilename))
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `[1, 2, 3, 4, 5]
+
+	expectedOutput := `a: valueA1 # commentA1
+b: valueB1 # commentB2
+c: valueC2 # commentC2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeCommentsIgnoreCmd(t *testing.T) {
+	filename := test.WriteTempYamlFile(commentContentA)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeFilename := test.WriteTempYamlFile(commentContentB)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --comments=ignore %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: valueA1 # commentA1
+b: valueB1
+c: valueC2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeCommentsAppendCmd(t *testing.T) {
+	filename := test.WriteTempYamlFile(commentContentA)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeFilename := test.WriteTempYamlFile(commentContentB)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --comments=append %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: valueA1 # commentA1 # commentA2
+b: valueB1 # commentB2
+c: valueC2 # commentC2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeCommentsOverwriteCmd(t *testing.T) {
+	filename := test.WriteTempYamlFile(commentContentA)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeFilename := test.WriteTempYamlFile(commentContentB)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --comments=overwrite %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: valueA1 # commentA2
+b: valueB1 # commentB2
+c: valueC2 # commentC2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOverwriteArraysTooCmd(t *testing.T) {
+	content := `a: simple # just the best
+b: [1, 2]
+c:
+  test: 1
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `a: things
+b: [6]`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --autocreate=false --arrays=overwrite --overwrite %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: things # just the best
+b: [6]
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeRootArraysCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --arrays=append ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- 1
+- 2
+- 3
+- 4
+- 5
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOverwriteArraysCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --arrays=overwrite ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- 4
+- 5
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeUpdateArraysCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge -x --arrays=update ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- 4
+- 5
+- 3
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
@@ -145,9 +363,7 @@ func TestMergeCmd_Multi(t *testing.T) {
 another:
   document: here
 a: simple # just the best
-b:
-  - 1
-  - 2
+b: [1, 2]
 c:
   test: 1
 ---
@@ -316,9 +532,7 @@ func TestMergeAllowEmptyTargetCmd(t *testing.T) {
 		t.Error(result.Error)
 	}
 	expectedOutput := `a: simple # just the best
-b:
-  - 1
-  - 2
+b: [1, 2]
 c:
   test: 1
 `

cmd/new.go

@@ -2,7 +2,6 @@ package cmd
 
 import (
 	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	errors "github.com/pkg/errors"
 	"github.com/spf13/cobra"
 )
 
@@ -36,11 +35,7 @@ Note that you can give a create script to perform more sophisticated yaml. This
 
 func newProperty(cmd *cobra.Command, args []string) error {
 	var badArgsMessage = "Must provide <path_to_update> <value>"
-	if len(args) != 2 {
-		return errors.New(badArgsMessage)
-	}
-
-	var updateCommands, updateCommandsError = readUpdateCommands(args, 2, badArgsMessage)
+	var updateCommands, updateCommandsError = readUpdateCommands(args, 2, badArgsMessage, false)
 	if updateCommandsError != nil {
 		return updateCommandsError
 	}

cmd/new_test.go

@@ -1,6 +1,7 @@
 package cmd
 
 import (
+	"fmt"
 	"testing"
 
 	"github.com/mikefarah/yq/v3/test"
@@ -18,6 +19,24 @@ func TestNewCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestNewCmdScript(t *testing.T) {
+	updateScript := `- command: update
+  path: b.c
+  value: 7`
+	scriptFilename := test.WriteTempYamlFile(updateScript)
+	defer test.RemoveTempYamlFile(scriptFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("new --script %s", scriptFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 7
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestNewAnchorCmd(t *testing.T) {
 	cmd := getRootCommand()
 	result := test.RunCmd(cmd, "new b.c 3 --anchorName=fred")

cmd/read_test.go

@@ -94,6 +94,28 @@ func TestReadUnwrapJsonByDefaultCmd(t *testing.T) {
 	test.AssertResult(t, "\"frog\"\n", result.Output)
 }
 
+func TestReadOutputJsonNonStringKeysCmd(t *testing.T) {
+
+	content := `
+true: true
+5:
+  null:
+    0.1: deeply
+    false: things`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read %s -j", filename))
+
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `{"true":true,"5":{"null":{"0.1":"deeply","false":"things"}}}
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestReadWithAdvancedFilterCmd(t *testing.T) {
 	cmd := getRootCommand()
 	result := test.RunCmd(cmd, "read ../examples/sample.yaml b.e(name==sam).value")
@@ -192,7 +214,7 @@ func TestReadArrayLengthCmd(t *testing.T) {
 }
 
 func TestReadArrayLengthDeepCmd(t *testing.T) {
-	content := `holder: 
+	content := `holder:
 - things
 - whatever
 `
@@ -208,12 +230,12 @@ func TestReadArrayLengthDeepCmd(t *testing.T) {
 }
 
 func TestReadArrayLengthDeepMultipleCmd(t *testing.T) {
-	content := `holderA: 
+	content := `holderA:
 - things
 - whatever
 skipMe:
 - yep
-holderB: 
+holderB:
 - other things
 - cool
 `
@@ -272,10 +294,10 @@ func TestReadCollectArrayCmd(t *testing.T) {
 }
 
 func TestReadArrayLengthDeepMultipleWithPathCmd(t *testing.T) {
-	content := `holderA: 
+	content := `holderA:
 - things
 - whatever
-holderB: 
+holderB:
 - other things
 - cool
 `
@@ -306,7 +328,7 @@ dog: bark
 }
 
 func TestReadObjectLengthDeepCmd(t *testing.T) {
-	content := `holder: 
+	content := `holder:
   cat: meow
   dog: bark
 `
@@ -322,10 +344,10 @@ func TestReadObjectLengthDeepCmd(t *testing.T) {
 }
 
 func TestReadObjectLengthDeepMultipleCmd(t *testing.T) {
-	content := `holderA: 
+	content := `holderA:
   cat: meow
   dog: bark
-holderB: 
+holderB:
   elephant: meow
   zebra: bark
 `
@@ -341,10 +363,10 @@ holderB:
 }
 
 func TestReadObjectLengthDeepMultipleWithPathsCmd(t *testing.T) {
-	content := `holderA: 
+	content := `holderA:
   cat: meow
   dog: bark
-holderB: 
+holderB:
   elephant: meow
   zebra: bark
 `
@@ -400,7 +422,7 @@ func TestReadSingleQuotedStringCmd(t *testing.T) {
 
 func TestReadQuotedMultinlineStringCmd(t *testing.T) {
 	content := `test: |
-  abcdefg    
+  abcdefg
   hijklmno
 `
 	filename := test.WriteTempYamlFile(content)
@@ -411,7 +433,7 @@ func TestReadQuotedMultinlineStringCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `abcdefg    
+	expectedOutput := `abcdefg
 hijklmno
 
 `
@@ -420,7 +442,7 @@ hijklmno
 
 func TestReadQuotedMultinlineNoNewLineStringCmd(t *testing.T) {
 	content := `test: |-
-  abcdefg    
+  abcdefg
   hijklmno
 `
 	filename := test.WriteTempYamlFile(content)
@@ -431,7 +453,7 @@ func TestReadQuotedMultinlineNoNewLineStringCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `abcdefg    
+	expectedOutput := `abcdefg
 hijklmno
 `
 	test.AssertResult(t, expectedOutput, result.Output)
@@ -561,7 +583,7 @@ func TestReadMergeAnchorsExplodeJsonCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `{"bar":{"b":2,"c":"oldbar","thing":"coconut"},"foo":{"a":"original","thing":"coolasdf","thirsty":"yep"},"foobar":{"a":"original","c":3,"thing":"ice","thirsty":"yep","thirty":"well beyond"},"foobarList":{"a":"original","b":2,"c":"newbar","thing":"coconut","thirsty":"yep"}}
+	expectedOutput := `{"foo":{"a":"original","thing":"coolasdf","thirsty":"yep"},"bar":{"b":2,"thing":"coconut","c":"oldbar"},"foobarList":{"c":"newbar","b":2,"thing":"coconut","a":"original","thirsty":"yep"},"foobar":{"thirty":"well beyond","thing":"ice","c":3,"a":"original","thirsty":"yep"}}
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
@@ -595,6 +617,44 @@ pointer: *value-pointer`
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestReadMergeAnchorsExplodeMissingCmd(t *testing.T) {
+	content := `a:
+  <<: &anchor
+  c: d
+  e: f
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -X %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a:
+  c: d
+  e: f
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestReadMergeAnchorsExplodeKeyCmd(t *testing.T) {
+	content := `name: &nameField Mike
+*nameField: Great Guy`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -X %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `name: Mike
+Mike: Great Guy
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestReadMergeAnchorsExplodeSimpleArrayCmd(t *testing.T) {
 	content := `- things`
 	filename := test.WriteTempYamlFile(content)
@@ -850,9 +910,9 @@ func TestReadEmptyContentCmd(t *testing.T) {
 }
 
 func TestReadEmptyNodesPrintPathCmd(t *testing.T) {
-	content := `map: 
+	content := `map:
   that: {}
-array: 
+array:
   great: []
 null:
   indeed: ~`
@@ -902,6 +962,7 @@ b:
       value: 3
     - name: sam
       value: 4
+ab: must appear last
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
@@ -941,6 +1002,7 @@ b:
           value: 3
         - name: sam
           value: 4
+ab: must appear last
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
@@ -1209,7 +1271,7 @@ func TestReadBadDataCmd(t *testing.T) {
 
 func TestReadDeepFromRootCmd(t *testing.T) {
 	content := `state:
-  country: 
+  country:
     city: foo
 `
 	filename := test.WriteTempYamlFile(content)
@@ -1379,3 +1441,23 @@ func TestReadFindValueDeepObjectCmd(t *testing.T) {
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
+
+func TestReadKeepsKeyOrderInJson(t *testing.T) {
+	const content = `{
+	"z": "One",
+	"a": 1,
+	"w": ["a", "r"],
+	"u": {"d": "o", "0": 11.5},
+}`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("r -j %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `{"z":"One","a":1,"w":["a","r"],"u":{"d":"o","0":11.5}}
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}

cmd/root.go

@@ -22,6 +22,7 @@ func New() *cobra.Command {
 			return nil
 		},
 		PersistentPreRun: func(cmd *cobra.Command, args []string) {
+			cmd.SetOut(cmd.OutOrStdout())
 			var format = logging.MustStringFormatter(
 				`%{color}%{time:15:04:05} %{shortfunc} [%{level:.4s}]%{color:reset} %{message}`,
 			)

cmd/utils.go

@@ -17,7 +17,7 @@ type readDataFn func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error)
 
 func createReadFunction(path string) func(*yaml.Node) ([]*yqlib.NodeContext, error) {
 	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
-		return lib.Get(dataBucket, path, true)
+		return lib.Get(dataBucket, path)
 	}
 }
 
@@ -163,6 +163,9 @@ func setIfNotThere(node *yaml.Node, key string, value *yaml.Node) {
 }
 
 func applyAlias(node *yaml.Node, alias *yaml.Node) {
+	if alias == nil {
+		return
+	}
 	for index := 0; index < len(alias.Content); index = index + 2 {
 		keyNode := alias.Content[index]
 		log.Debugf("applying alias key %v", keyNode.Value)
@@ -185,12 +188,14 @@ func explodeNode(node *yaml.Node) error {
 		return nil
 	case yaml.AliasNode:
 		log.Debugf("its an alias!")
-		node.Kind = node.Alias.Kind
-		node.Style = node.Alias.Style
-		node.Tag = node.Alias.Tag
-		node.Content = node.Alias.Content
-		node.Value = node.Alias.Value
-		node.Alias = nil
+		if node.Alias != nil {
+			node.Kind = node.Alias.Kind
+			node.Style = node.Alias.Style
+			node.Tag = node.Alias.Tag
+			node.Content = node.Alias.Content
+			node.Value = node.Alias.Value
+			node.Alias = nil
+		}
 		return nil
 	case yaml.MappingNode:
 		for index := 0; index < len(node.Content); index = index + 2 {
@@ -202,6 +207,10 @@ func explodeNode(node *yaml.Node) error {
 				if errorInContent != nil {
 					return errorInContent
 				}
+				errorInContent = explodeNode(keyNode)
+				if errorInContent != nil {
+					return errorInContent
+				}
 			} else {
 				if valueNode.Kind == yaml.SequenceNode {
 					log.Debugf("an alias merge list!")
@@ -477,7 +486,7 @@ type updateCommandParsed struct {
 	Value   yaml.Node
 }
 
-func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string) ([]yqlib.UpdateCommand, error) {
+func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string, allowNoValue bool) ([]yqlib.UpdateCommand, error) {
 	var updateCommands []yqlib.UpdateCommand = make([]yqlib.UpdateCommand, 0)
 	if writeScript != "" {
 		var parsedCommands = make([]updateCommandParsed, 0)
@@ -511,8 +520,9 @@ func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string)
 		log.Debug("args %v", args)
 		log.Debug("path %v", args[expectedArgs-2])
 		log.Debug("Value %v", args[expectedArgs-1])
-		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: valueParser.Parse(args[expectedArgs-1], customTag, customStyle, anchorName, makeAlias), Overwrite: true}
-	} else if len(args) == expectedArgs-1 {
+		value := valueParser.Parse(args[expectedArgs-1], customTag, customStyle, anchorName, makeAlias)
+		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: value, Overwrite: true, CommentsMergeStrategy: yqlib.IgnoreCommentsMergeStrategy}
+	} else if len(args) == expectedArgs-1 && allowNoValue {
 		// don't update the value
 		updateCommands = make([]yqlib.UpdateCommand, 1)
 		log.Debug("args %v", args)

cmd/validate.go

@@ -15,7 +15,7 @@ yq v - # reads from stdin
 `,
 		RunE:          validateProperty,
 		SilenceUsage:  true,
-		SilenceErrors: true,
+		SilenceErrors: false,
 	}
 	cmdRead.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
 	return cmdRead

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "3.3.2"
+	Version = "3.4.1"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release

cmd/write.go

@@ -53,7 +53,7 @@ format is list of update commands (update or delete) like so:
 }
 
 func writeProperty(cmd *cobra.Command, args []string) error {
-	var updateCommands, updateCommandsError = readUpdateCommands(args, 3, "Must provide <filename> <path_to_update> <value>")
+	var updateCommands, updateCommandsError = readUpdateCommands(args, 3, "Must provide <filename> <path_to_update> <value>", true)
 	if updateCommandsError != nil {
 		return updateCommandsError
 	}

cmd/write_test.go

@@ -27,6 +27,24 @@ func TestWriteCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestWriteKeepCommentsCmd(t *testing.T) {
+	content := `b:
+  c: 3 # comment
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 7 # comment
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestWriteWithTaggedStyleCmd(t *testing.T) {
 	content := `b:
   c: dog

commands/compare.md

@@ -1,103 +0,0 @@
----
-description: Deeply compare two yaml documents
----
-
-# Compare
-
-```bash
-yq compare <yaml_file_1> <yaml_file_2> <path_expression>
-```
-
-Compares the matching yaml nodes at path expression in the two yaml documents. See [path expression](../usage/path-expressions.md) for more details. Difference calculated line by line, and is printed out line by line where the first character of each line is either:
-
-*  `` a space, indicating no change at this line
-* `-` a minus ,indicating the line is not present in the second document \(it's removed\)
-* `+` a plus, indicating that the line is not present in the first document \(it's added\)
-
-If there are differences then `yq` will print out the differences and exit with code 1. If there are no differences, then nothing will be printed and the exit code will be 0.
-
-## Example data
-
-Given data1.yaml
-
-```yaml
-"apples": are nice
-somethingElse: cool # this is nice
-favouriteNumbers: [1,2,3]
-noDifference: it's the same
-```
-
-and data2.yaml
-
-```yaml
-apples: are nice
-somethingElse: cool # yeah i like it
-favouriteNumbers:
-- 1
-- 3
-- 4
-noDifference: it's the same
-```
-
-## Basic
-
-Basic will compare the yaml documents 'as-is'
-
-```yaml
-yq compare data1.yaml data2.yaml
-```
-
-yields
-
-```text
--"apples": are nice
--somethingElse: cool # this is nice
--favouriteNumbers: [1, 2, 3]
-+apples: are nice
-+somethingElse: cool # yeah i like it
-+favouriteNumbers:
-+- 1
-+- 3
-+- 4
- noDifference: it's the same
-```
-
-## Formatted
-
-Most of the time, it will make sense to [format](../usage/output-format.md#pretty-print) the documents before comparing:
-
-```text
-yq compare --prettyPrint data1.yaml data2.yml
-```
-
-yields
-
-```text
- apples: are nice
--somethingElse: cool # this is nice
-+somethingElse: cool # yeah i like it
- favouriteNumbers:
- - 1
--- 2
- - 3
-+- 4
- noDifference: it's the same
-```
-
-## Using path expressions
-
-Use [path expressions](../usage/path-expressions.md) to compare subsets of yaml documents
-
-```text
-yq compare -P data1.yaml data2.yml favouriteNumbers
-```
-
-yields
-
-```text
- - 1
--- 2
- - 3
-+- 4
-```
-

commands/create.md

@@ -1,58 +0,0 @@
-# Create
-
-```text
-yq n <path_expression> <new value>
-```
-
-This works in the same way as the write command, but you don't pass in an existing Yaml file. Currently this does not support creating multiple documents in a single yaml file.
-
-See docs for [path expression](../usage/path-expressions.md) and [value parsing](../usage/value-parsing.md) for more details, including controlling quotes and tags.
-
-## Creating a simple yaml file
-
-```bash
-yq n b.c cat
-```
-
-will output:
-
-```yaml
-b:
-  c: cat
-```
-
-## Creating using a create script
-
-Create scripts follow the same format as the update scripts.
-
-Given a script create\_instructions.yaml of:
-
-```yaml
-- command: update 
-  path: b.c
-  value:
-    #great 
-    things: frog # wow!
-```
-
-then
-
-```bash
-yq n -s create_instructions.yaml
-```
-
-will output:
-
-```yaml
-b:
-  c:
-    #great
-    things: frog # wow!
-```
-
-You can also pipe the instructions in:
-
-```bash
-cat create_instructions.yaml | yq n -s -
-```
-

commands/delete.md

@@ -1,110 +0,0 @@
----
-description: Deletes all the matching nodes for the path expression in the given yaml input
----
-
-# Delete
-
-```text
-yq delete <yaml_file|-> <path_expression>
-```
-
-See docs for [path expression](../usage/path-expressions.md) for more details.
-
-## Deleting from a simple document
-
-Given a sample.yaml file of:
-
-```yaml
-b:
-  c: 2
-  apples: green
-```
-
-then
-
-```bash
-yq d sample.yaml b.c
-```
-
-will output
-
-```yaml
-b:
-  apples: green
-```
-
-## From STDIN
-
-Use "-" \(without quotes\) in-place of a file name if you wish to pipe in input from STDIN.
-
-```bash
-cat sample.yaml | yq d - b.c
-```
-
-## Deleting in-place
-
-```bash
-yq d -i sample.yaml b.c
-```
-
-will update the sample.yaml file so that the 'c' node is deleted
-
-## Multiple Documents
-
-### Delete from single document
-
-Given a sample.yaml file of:
-
-```yaml
-something: else
-field: leaveMe
----
-b:
-  c: 2
-field: deleteMe
-```
-
-then
-
-```bash
-yq w -d1 sample.yaml field
-```
-
-will output:
-
-```yaml
-something: else
-field: leaveMe
----
-b:
-  c: 2
-```
-
-### Delete from all documents
-
-Given a sample.yaml file of:
-
-```yaml
-something: else
-field: deleteMe
----
-b:
-  c: 2
-field: deleteMeToo
-```
-
-then
-
-```bash
-yq w -d'*' sample.yaml field
-```
-
-will output:
-
-```yaml
-something: else
----
-b:
-  c: 2
-```
-

commands/merge.md

@@ -1,232 +0,0 @@
----
-description: Merge multiple yaml files into a one
----
-
-# Merge
-
-Yaml files can be merged using the 'merge' command. Each additional file merged with the first file will set values for any key not existing already or where the key has no value.
-
-```text
-yq m <yaml_file> <yaml_file2> <yaml_file3>...
-```
-
-## Merge example
-
-Given a data1.yaml file of:
-
-```yaml
-a: simple
-b: [1, 2]
-```
-
-and data2.yaml file of:
-
-```yaml
-a: other
-c:
-  test: 1
-```
-
-then
-
-```bash
-yq merge data1.yaml data2.yaml
-```
-
-will output:
-
-```yaml
-a: simple
-b: [1, 2]
-c:
-  test: 1
-```
-
-## Updating files in-place
-
-```bash
-yq m -i data1.yaml data2.yaml
-```
-
-will update the data1.yaml file with the merged result.
-
-## Overwrite values
-
-Given a data1.yaml file of:
-
-```yaml
-a: simple
-b: [1, 2]
-d: left alone
-```
-
-and data2.yaml file of:
-
-```yaml
-a: other
-b: [3, 4]
-c:
-  test: 1
-```
-
-then
-
-```bash
-yq m -x data1.yaml data2.yaml
-```
-
-will output:
-
-```yaml
-a: other
-b: [3, 4]
-c:
-  test: 1
-d: left alone
-```
-
-Notice that 'b' does not result in the merging of the values within an array.
-
-## Append values with arrays
-
-Given a data1.yaml file of:
-
-```yaml
-a: simple
-b: [1, 2]
-d: hi
-```
-
-and data2.yaml file of:
-
-```yaml
-a: something
-b: [3, 4]
-c:
-  test: 2
-  other: true
-```
-
-then
-
-```bash
-yq m -a data1.yaml data2.yaml
-```
-
-will output:
-
-```yaml
-a: simple
-b: [1, 2, 3, 4]
-c:
-  test: 2
-  other: true
-d: hi
-```
-
-Note that the 'b' array has concatenated the values from the second data file. Also note that other map keys are not overridden \(field a\).
-
-## Auto-create
-
-By default, `yq` will automatically create any missing entries in the target yaml file. This can be turned off so that only matching paths are merged in. When turned off - you will most likely want to use the [override flag](merge.md#overwrite-values).
-
-Given a data1.yml file of:
-
-```yaml
-a: thing
-b: something else
-```
-
-and data2.yml file of:
-
-```yaml
-b: new value
-d: not in original
-```
-
-Then
-
-```yaml
-yq m --overwrite --autocreate=false data1.yml data2.yml
-```
-
-will yield
-
-```yaml
-a: thing
-b: new value
-```
-
-
-
-## Multiple Documents
-
-### Merge into single document
-
-Currently yq only has multi-document support for the _first_ document being merged into. The remaining yaml files will have their first document selected.
-
-Given a data1.yaml file of:
-
-```yaml
-something: else
----
-a: simple
-b: cat
-```
-
-and data3.yaml file of:
-
-```yaml
-b: dog
-```
-
-then
-
-```bash
-yq m -x -d1 data1.yaml data3.yaml
-```
-
-will output:
-
-```yaml
-something: else
----
-a: simple
-b: dog
-```
-
-### Merge into all documents
-
-Currently yq only has multi-document support for the _first_ document being merged into. The remaining yaml files will have their first document selected.
-
-Given a data1.yaml file of:
-
-```yaml
-something: else
----
-a: simple
-b: cat
-```
-
-and data3.yaml file of:
-
-```yaml
-b: dog
-```
-
-then
-
-```bash
-yq m -x -d'*' data1.yaml data3.yaml
-```
-
-will output:
-
-```yaml
-b: dog
-something: else
----
-a: simple
-b: dog
-```
-

commands/prefix.md

@@ -1,103 +0,0 @@
----
-description: >-
-  Prefixes a yaml document with the given path expression. The complete yaml
-  content will be nested inside the new prefix path.
----
-
-# Prefix
-
-```text
-yq p <yaml_file> <path>
-```
-
-See docs for [path expression](../usage/path-expressions.md) for more details.
-
-## Prefix a document
-
-Given a data1.yaml file of:
-
-```yaml
-a:
-  b: [1, 2]
-```
-
-then
-
-```bash
-yq p data1.yaml c.d
-```
-
-will output:
-
-```yaml
-c:
-  d:
-    a:
-      b: [1, 2]
-```
-
-## Updating files in-place
-
-```bash
-yq p -i data1.yaml c
-```
-
-will update the data1.yaml file so that the path 'c' prefixes the document.
-
-## Multiple Documents
-
-### Prefix a single document
-
-Given a data1.yaml file of:
-
-```yaml
-something: else
----
-a: simple
-b: cat
-```
-
-then
-
-```bash
-yq p -d1 data1.yaml c
-```
-
-will output:
-
-```yaml
-something: else
----
-c:
-  a: simple
-  b: cat
-```
-
-### Prefix all documents
-
-Given a data1.yaml file of:
-
-```yaml
-something: else
----
-a: simple
-b: cat
-```
-
-then
-
-```bash
-yq p -d'*' data1.yaml c
-```
-
-will output:
-
-```yaml
-c:
-  something: else
----
-c:
-  a: simple
-  b: cat
-```
-

commands/read.md

@@ -1,326 +0,0 @@
----
-description: Returns matching nodes/values of a path expression for a given yaml document
----
-
-# Read
-
-```text
-yq r <yaml_file|json_file> <path_expression>
-```
-
-Returns the matching nodes of the path expression for the given yaml file \(or STDIN\).
-
-See docs for [path expression](../usage/path-expressions.md) for more details.
-
-## Basic
-
-Given a sample.yaml file of:
-
-```yaml
-b:
-  c: 2
-```
-
-then
-
-```bash
-yq r sample.yaml b.c
-```
-
-will output the value of '2'.
-
-## From Stdin
-
-Given a sample.yaml file of:
-
-```bash
-cat sample.yaml | yq r - b.c
-```
-
-will output the value of '2'.
-
-## Default values
-
-Using the `--defaultValue/-D` flag you can specify a default value to be printed when no matching nodes are found for an expression
-
-```text
-yq r sample.yaml --defaultValue frog path.not.there
-```
-
-will yield \(assuming `path.not.there` does not match any nodes\):
-
-```text
-frog
-```
-
-## Printing matching paths
-
-By default, yq will only print the value of the path expression for the yaml document. By specifying `--printMode` or `-p` you can print the matching paths.
-
-```yaml
-a:
-  thing_a: 
-    animal: cat
-  other: 
-    animal: frog
-  thing_b: 
-    vehicle: car
-```
-
-### Path Only
-
-```bash
-yq r --printMode p "a.thing*.*"
-```
-
-will print
-
-```text
-a.thing_a.animal
-a.thing_b.vehicle
-```
-
-### Path and Value
-
-```bash
-yq r --printMode pv "a.thing*.*"
-```
-
-will print
-
-```text
-a.thing_a.animal: cat
-a.thing_b.vehicle: car
-```
-
-## Collect results into an array
-
-By default, results are printed out line by line as independent matches. This is handy for both readability as well as piping into tools like `xargs`. However, if you would like to collect the matching results into an array then use the `--collect/-C` flag. This is particularly useful with the `length` flag described below.
-
-Given:
-
-```yaml
-a:
-  thing_a: 
-    animal: cat
-  other: 
-    animal: frog
-  thing_b: 
-    vehicle: car
-```
-
-```text
-yq r sample.yaml --collect a.*.animal
-```
-
-will print
-
-```text
-- cat
-- frog
-```
-
-## Printing length of the results
-
-Use the `--length/-L` flag to print the length of results. For arrays this will be the number of items, objects the number of entries and scalars the length of the value.
-
-Given
-
-```text
-animals:
-  - cats
-  - dog
-  - cheetah
-```
-
-```text
-yq r sample.yml --length animals
-```
-
-will print 
-
-```text
-3
-```
-
-### Length of filtered results
-
-By default, filtered results are printed _independently_ so you will get the length of each result printed on a separate line:
-
-```text
-yq r sample.yaml --length --printMode pv 'animals.(.==c*)'
-```
-
-```text
-animals.[0]: 4
-animals.[2]: 7
-```
-
-However, you'll often want to know the count of the filtered results - use the `--collect` flag to collect the results in the array. The length will then return the size of the array. 
-
-```text
-yq r sample.yaml --length --collect 'animals.(.==c*)' 
-```
-
-```text
-2
-```
-
-## Anchors and Aliases
-
-The read command will print out the anchors of a document and can also traverse them.
-
-Lets take a look at a simple example file:
-
-```yaml
-foo: &foo
-  a: 1
-
-foobar: *foo
-```
-
-### Printing anchors
-
-```bash
-yq r sample.yml foo
-```
-
-will print out
-
-```yaml
-&foo
-a: 1
-```
-
-Similarly,
-
-```text
-yq r sample.yml foobar
-```
-
-prints out
-
-```yaml
-*foo
-```
-
-### Traversing anchors
-
-To traverse an anchor, we need to either explicitly reference merged in values:
-
-```text
-yq r sample.yml foobar.a
-```
-
-to get
-
-```text
-1
-```
-
-or we can use deep splat to get all the values
-
-```bash
-yq r sample.yml -p pv "foobar.**"
-```
-
-prints
-
-```yaml
-foobar.a: 1
-```
-
-The same methods work for the `<<: [*blah, *thing]`anchors.
-
-### Exploding Anchors
-
-By default anchors are not exploded \(or expanded/de-referenced\) for viewing, and the yaml is shown as-is. Use the `--explodeAnchors/-X` flag to show the anchor values.
-
-Given sample.yml:
-
-```yaml
-foo: &foo
-  a: original
-  thing: coolasdf
-  thirsty: yep
-
-bar: &bar
-  b: 2
-  thing: coconut
-  c: oldbar
-
-foobarList:
-  <<: [*foo,*bar]
-  c: newbar
-```
-
-Then
-
-```text
-yq r -X sample.yml foobarList
-```
-
-yields
-
-```text
-c: newbar
-b: 2
-thing: coconut
-a: original
-thirsty: yep
-```
-
-Note that yq processes the merge anchor list in reverse order, to ensure that the last items in the list override the preceding.
-
-## Multiple Documents
-
-### Reading from a single document
-
-Given a sample.yaml file of:
-
-```yaml
-something: else
----
-b:
-  c: 2
-```
-
-then
-
-```bash
-yq r -d1 sample.yaml b.c
-```
-
-will output the value of '2'.
-
-### Read from all documents
-
-Reading all documents will return the result as an array. This can be converted to json using the '-j' flag if desired.
-
-Given a sample.yaml file of:
-
-```yaml
-name: Fred
-age: 22
----
-name: Stella
-age: 23
----
-name: Android
-age: 232
-```
-
-then
-
-```bash
-yq r -d'*' sample.yaml name
-```
-
-will output:
-
-```text
-Fred
-Stella
-Android
-```
-

commands/shell-completion.md

@@ -1,51 +0,0 @@
----
-description: >-
-  Generate a shell completion file for supported shells
-  (bash/fish/zsh/powershell)
----
-
-# Shell Completion
-
-```bash
-yq shell-completion --variation=zsh
-```
-
-Prints to StdOut a shell completion script for zsh shell.
-
-### Bash \(default\)
-
-```bash
-yq shell-completion
-```
-
-To configure your bash shell to load completions for each session add to your bashrc
-
-```text
-# ~/.bashrc or ~/.profile
-. <(yq shell-completion)
-```
-
-### zsh
-
-```bash
-yq shell-completion --variation=zsh
-```
-
-The generated completion script should be put somewhere in your $fpath named \_yq
-
-### fish
-
-```bash
-yq shell-completion --variation=fish
-```
-
-Save the output to a '.fish' file and add it to your completions directory.
-
-### PowerShell
-
-```bash
-yq shell-completion --variation=powershell
-```
-
-Users need PowerShell version 5.0 or above, which comes with Windows 10 and can be downloaded separately for Windows 7 or 8.1. They can then write the completions to a file and source this file from their PowerShell profile, which is referenced by the $Profile environment variable.
-

commands/validate.md

@@ -1,54 +0,0 @@
----
-description: Validate a given yaml file
----
-
-# Validate
-
-```text
-yq v <yaml_file|->
-```
-
-Validates the given yaml file, does nothing if its valid, otherwise it will print errors to Stderr and exit with a non 0 exit code. This works like the [read command](read.md) - but does not take a path expression and does not print the yaml if it is valid.
-
-## Basic - valid
-
-```text
-yq v valid.yaml
-```
-
-This will not print anything, and finish with a successful \(0\) exit code.
-
-## Basic - invalid, from stdin
-
-```text
-echo '[1234' | yq v -
-```
-
-This will print the parsing error to stderr:
-
-```bash
-10:43:09 main [ERRO] yaml: line 1: did not find expected ',' or ']'
-```
-
-And return a error exit code \(1\)
-
-## Multiple documents
-
-Like other commands, by default the validate command will only run against the first document in the yaml file. Note that when running against other specific document indexes, _all previous documents will also be validated._
-
-### Validating a single document
-
-```bash
-yq v -d1 multidoc.yml
-```
-
-This will validate both document 0 and document 1 \(but not document 2\)
-
-### Validating all documents
-
-```bash
-yq v -d'*' multidoc.yml
-```
-
-This will validate all documents in the yaml file. Note that \* is quoted to avoid the CLI from processing the wildcard.
-

commands/write-update.md

@@ -1,285 +0,0 @@
----
-description: >-
-  Updates all the matching nodes of path expression in a yaml file to the
-  supplied value.
----
-
-# Write
-
-```bash
-yq w <yaml_file> <path_expression> <new value>
-```
-
-See docs for [path expression](../usage/path-expressions.md) and [value parsing](../usage/value-parsing.md) for more details, including controlling quotes and tags.
-
-## Basic
-
-Given a sample.yaml file of:
-
-```yaml
-b:
-  c: 2
-```
-
-then
-
-```bash
-yq w sample.yaml b.c cat
-```
-
-will output:
-
-```yaml
-b:
-  c: cat
-```
-
-### Updating files in-place
-
-```bash
-yq w -i sample.yaml b.c cat
-```
-
-will update the sample.yaml file so that the value of 'c' is cat.
-
-## From STDIN
-
-```bash
-cat sample.yaml | yq w - b.c blah
-```
-
-## Adding new fields
-
-Any missing fields in the path will be created on the fly.
-
-Given a sample.yaml file of:
-
-```yaml
-b:
-  c: 2
-```
-
-then
-
-```bash
-yq w sample.yaml b.d[+] "new thing"
-```
-
-will output:
-
-```yaml
-b:
-  c: cat
-  d:
-    - new thing
-```
-
-## Appending value to an array field
-
-Given a sample.yaml file of:
-
-```yaml
-b:
-  c: 2
-  d:
-    - new thing
-    - foo thing
-```
-
-then
-
-```bash
-yq w sample.yaml "b.d[+]" "bar thing"
-```
-
-will output:
-
-```yaml
-b:
-  c: cat
-  d:
-    - new thing
-    - foo thing
-    - bar thing
-```
-
-Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
-
-## Multiple Documents
-
-### Update a single document
-
-Given a sample.yaml file of:
-
-```yaml
-something: else
----
-b:
-  c: 2
-```
-
-then
-
-```bash
-yq w -d1 sample.yaml b.c 5
-```
-
-will output:
-
-```yaml
-something: else
----
-b:
-  c: 5
-```
-
-### Update all documents
-
-Given a sample.yaml file of:
-
-```yaml
-something: else
----
-b:
-  c: 2
-```
-
-then
-
-```bash
-yq w -d'*' sample.yaml b.c 5
-```
-
-will output:
-
-```yaml
-something: else
-b:
-  c: 5
----
-b:
-  c: 5
-```
-
-## Writing Anchors
-
-The `---anchorName` flag can be used to set the anchor name of a node
-
-Given a sample document of:
-
-```yaml
-commonStuff:
-    flavour: vanilla
-```
-
-Then:
-
-```bash
-yq write sample.yaml commonStuff --anchorName=commonBits
-```
-
-Will yield
-
-```yaml
-commonStuff: &commonBits
-    flavour: vanilla
-```
-
-## Writing Aliases
-
-The `--makeAlias` flag can create \(or update\) a node to be an alias to an anchor.
-
-Given a sample file of:
-
-```yaml
-commonStuff: &commonBits
-    flavour: vanilla
-```
-
-Then
-
-```bash
-yq write sample.yaml --makeAnchor foo commonBits
-```
-
-Will yield:
-
-```yaml
-commonStuff: &commonBits
-    flavour: vanilla
-foo: *commonBits
-```
-
-## Updating only styles/tags without affecting values
-
-You can use the write command to update the quoting style of nodes, or their tags, without re-specifying the values. This is done by omitting the value argument:
-
-Given a sample document:
-
-```yaml
-a:
-  c: things
-  d: other things
-```
-
-Then
-
-```bash
-yq write sample.yaml --style=single a.*
-```
-
-Will yield:
-
-```yaml
-a:
-  c: 'things'
-  d: 'other things'
-```
-
-## Using a script file to update
-
-Given a sample.yaml file of:
-
-```yaml
-b:
-  d: be gone
-  c: 2
-  e:
-    - name: Billy Bob # comment over here
-```
-
-and a script update\_instructions.yaml of:
-
-```yaml
-- command: update 
-  path: b.c
-  value:
-    #great 
-    things: frog # wow!
-- command: delete
-  path: b.d
-```
-
-then
-
-```bash
-yq w -s update_instructions.yaml sample.yaml
-```
-
-will output:
-
-```yaml
-b:
-  c:
-    #great
-    things: frog # wow!
-  e:
-  - name: Billy Bob # comment over here
-```
-
-And, of course, you can pipe the instructions in using '-':
-
-```bash
-cat update_instructions.yaml | yq w -s - sample.yaml
-```
-

debian/changelog

@@ -1,3 +1,10 @@
+yq (3.3.2) focal; urgency=medium
+
+  * Bug fix: existStatus bug (#459)
+  * Automatically makes a os temp directory if it does not exist (#461)
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Fri, 07 Aug 2020 18:53:01 +0200
+
 yq (3.3-0) focal; urgency=medium
 
   * You can control string styles (quotes) using the new --style flag

examples/sample.json

@@ -1 +1 @@
-{"a":"Easy! as one two three","b":{"c":2,"d":[3,4],"e":[{"name":"fred","value":3},{"name":"sam","value":4}]}}
+{"a":"Easy! as one two three","b":{"c":2,"d":[3,4],"e":[{"name":"fred","value":3},{"name":"sam","value":4}]},"ab":"must appear last"}

go.mod

@@ -2,16 +2,16 @@ module github.com/mikefarah/yq/v3
 
 require (
 	github.com/fatih/color v1.9.0
-	github.com/goccy/go-yaml v1.7.5
+	github.com/goccy/go-yaml v1.8.1
 	github.com/kylelemons/godebug v1.1.0
-	github.com/mattn/go-colorable v0.1.6 // indirect
+	github.com/mattn/go-colorable v0.1.7 // indirect
 	github.com/pkg/errors v0.9.1
 	github.com/spf13/cobra v1.0.0
 	github.com/spf13/pflag v1.0.5 // indirect
-	golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980 // indirect
-	golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 // indirect
+	golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
-	gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c
+	gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
 )
 
-go 1.14
+go 1.15

go.sum

@@ -29,8 +29,8 @@ github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V
 github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
 github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
-github.com/goccy/go-yaml v1.7.5 h1:dWvj+p3BG11S/GlUzwzt1WZz0lhBTzTIDtmXT/ZOaPY=
-github.com/goccy/go-yaml v1.7.5/go.mod h1:wS4gNoLalDSJxo/SpngzPQ2BN4uuZVLCmbM4S3vd4+Y=
+github.com/goccy/go-yaml v1.8.1 h1:JuZRFlqLM5cWF6A+waL8AKVuCcqvKOuhJtUQI+L3ez0=
+github.com/goccy/go-yaml v1.8.1/go.mod h1:wS4gNoLalDSJxo/SpngzPQ2BN4uuZVLCmbM4S3vd4+Y=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
 github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
@@ -64,8 +64,8 @@ github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgx
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
 github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
-github.com/mattn/go-colorable v0.1.6 h1:6Su7aK7lXmJ/U79bYtBjLNaha4Fs1Rg9plHpcH+vvnE=
-github.com/mattn/go-colorable v0.1.6/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
+github.com/mattn/go-colorable v0.1.7 h1:bQGKb3vps/j0E9GfJQ03JyhRuxsvdAanXlT9BTw3mdw=
+github.com/mattn/go-colorable v0.1.7/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
 github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
 github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
 github.com/mattn/go-isatty v0.0.11 h1:FxPOTFNqGkuDUGi3H/qkUbQO4ZiBa2brKq5r0l8TGeM=
@@ -144,19 +144,20 @@ golang.org/x/sys v0.0.0-20191026070338-33540a1f6037 h1:YyJpGZS1sBuBCzLAR1VEpK193
 golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980 h1:OjiUf46hAmXblsZdnoSXsEUSKU8r1UEzcL5RVZ4gO9Y=
-golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f h1:Fqb3ao1hUmOR3GkUOg/Y+BadLwykBIzs5q8Ez2SbHyc=
+golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
@@ -175,6 +176,6 @@ gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bl
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c h1:grhR+C34yXImVGp7EzNk+DTIk+323eIUWOmEevy6bDo=
-gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ=
+gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

pkg/yqlib/data_navigator.go

@@ -22,13 +22,8 @@ func NewDataNavigator(NavigationStrategy NavigationStrategy) DataNavigator {
 }
 
 func (n *navigator) Traverse(value *yaml.Node, path []interface{}) error {
-	realValue := value
 	emptyArray := make([]interface{}, 0)
 	log.Debugf("Traversing path %v", pathStackToString(path))
-	if realValue.Kind == yaml.DocumentNode {
-		log.Debugf("its a document! returning the first child")
-		return n.doTraverse(value.Content[0], "", path, emptyArray)
-	}
 	return n.doTraverse(value, "", path, emptyArray)
 }
 
@@ -39,7 +34,8 @@ func (n *navigator) doTraverse(value *yaml.Node, head interface{}, tail []interf
 	var nodeContext = NewNodeContext(value, head, tail, pathStack)
 
 	var errorDeepSplatting error
-	if head == "**" && value.Kind != yaml.ScalarNode && n.navigationStrategy.ShouldDeeplyTraverse(nodeContext) {
+	// no need to deeply traverse the DocumentNode, as it's already covered by its first child.
+	if head == "**" && value.Kind != yaml.DocumentNode && value.Kind != yaml.ScalarNode && n.navigationStrategy.ShouldDeeplyTraverse(nodeContext) {
 		if len(pathStack) == 0 || pathStack[len(pathStack)-1] != "<<" {
 			errorDeepSplatting = n.recurse(value, head, tail, pathStack)
 		}
@@ -51,7 +47,11 @@ func (n *navigator) doTraverse(value *yaml.Node, head interface{}, tail []interf
 		return errorDeepSplatting
 	}
 
-	if len(tail) > 0 && value.Kind != yaml.ScalarNode {
+	if value.Kind == yaml.DocumentNode {
+		log.Debugf("its a document, diving into %v", head)
+		DebugNode(value)
+		return n.recurse(value, head, tail, pathStack)
+	} else if len(tail) > 0 && value.Kind != yaml.ScalarNode {
 		log.Debugf("diving into %v", tail[0])
 		DebugNode(value)
 		return n.recurse(value, tail[0], tail[1:], pathStack)
@@ -73,6 +73,7 @@ func (n *navigator) recurse(value *yaml.Node, head interface{}, tail []interface
 	nodeContext := NewNodeContext(value, head, tail, pathStack)
 
 	if head == "**" && !n.navigationStrategy.ShouldOnlyDeeplyVisitLeaves(nodeContext) {
+		nodeContext.IsMiddleNode = true
 		errorVisitingDeeply := n.navigationStrategy.Visit(nodeContext)
 		if errorVisitingDeeply != nil {
 			return errorVisitingDeeply
@@ -108,6 +109,8 @@ func (n *navigator) recurse(value *yaml.Node, head interface{}, tail []interface
 			return n.recurse(value.Alias, head, tail, pathStack)
 		}
 		return nil
+	case yaml.DocumentNode:
+		return n.doTraverse(value.Content[0], head, tail, pathStack)
 	default:
 		return n.navigationStrategy.Visit(nodeContext)
 	}

pkg/yqlib/encoder.go

@@ -3,6 +3,7 @@ package yqlib
 import (
 	"bytes"
 	"encoding/json"
+	"fmt"
 	"io"
 
 	yaml "gopkg.in/yaml.v3"
@@ -58,6 +59,21 @@ type jsonEncoder struct {
 	encoder *json.Encoder
 }
 
+func mapKeysToStrings(node *yaml.Node) {
+
+	if node.Kind == yaml.MappingNode {
+		for index, child := range node.Content {
+			if index%2 == 0 { // its a map key
+				child.Tag = "!!str"
+			}
+		}
+	}
+
+	for _, child := range node.Content {
+		mapKeysToStrings(child)
+	}
+}
+
 func NewJsonEncoder(destination io.Writer, prettyPrint bool, indent int) Encoder {
 	var encoder = json.NewEncoder(destination)
 	var indentString = ""
@@ -72,10 +88,162 @@ func NewJsonEncoder(destination io.Writer, prettyPrint bool, indent int) Encoder
 }
 
 func (je *jsonEncoder) Encode(node *yaml.Node) error {
-	var dataBucket interface{}
+	var dataBucket orderedMap
+	// firstly, convert all map keys to strings
+	mapKeysToStrings(node)
 	errorDecoding := node.Decode(&dataBucket)
 	if errorDecoding != nil {
 		return errorDecoding
 	}
 	return je.encoder.Encode(dataBucket)
 }
+
+// orderedMap allows to marshal and unmarshal JSON and YAML values keeping the
+// order of keys and values in a map or an object.
+type orderedMap struct {
+	// if this is an object, kv != nil. If this is not an object, kv == nil.
+	kv     []orderedMapKV
+	altVal interface{}
+}
+
+type orderedMapKV struct {
+	K string
+	V orderedMap
+}
+
+func (o *orderedMap) UnmarshalJSON(data []byte) error {
+	switch data[0] {
+	case '{':
+		// initialise so that even if the object is empty it is not nil
+		o.kv = []orderedMapKV{}
+
+		// create decoder
+		dec := json.NewDecoder(bytes.NewReader(data))
+		_, err := dec.Token() // open object
+		if err != nil {
+			return err
+		}
+
+		// cycle through k/v
+		var tok json.Token
+		for tok, err = dec.Token(); err != io.EOF; tok, err = dec.Token() {
+			// we can expect two types: string or Delim. Delim automatically means
+			// that it is the closing bracket of the object, whereas string means
+			// that there is another key.
+			if _, ok := tok.(json.Delim); ok {
+				break
+			}
+			kv := orderedMapKV{
+				K: tok.(string),
+			}
+			if err := dec.Decode(&kv.V); err != nil {
+				return err
+			}
+			o.kv = append(o.kv, kv)
+		}
+		// unexpected error
+		if err != nil && err != io.EOF {
+			return err
+		}
+		return nil
+	case '[':
+		var arr []orderedMap
+		return json.Unmarshal(data, &arr)
+	}
+
+	return json.Unmarshal(data, &o.altVal)
+}
+
+func (o orderedMap) MarshalJSON() ([]byte, error) {
+	if o.kv == nil {
+		return json.Marshal(o.altVal)
+	}
+	buf := new(bytes.Buffer)
+	enc := json.NewEncoder(buf)
+	buf.WriteByte('{')
+	for idx, el := range o.kv {
+		if err := enc.Encode(el.K); err != nil {
+			return nil, err
+		}
+		buf.WriteByte(':')
+		if err := enc.Encode(el.V); err != nil {
+			return nil, err
+		}
+		if idx != len(o.kv)-1 {
+			buf.WriteByte(',')
+		}
+	}
+	buf.WriteByte('}')
+	return buf.Bytes(), nil
+}
+
+func (o *orderedMap) UnmarshalYAML(node *yaml.Node) error {
+	switch node.Kind {
+	case yaml.DocumentNode:
+		if len(node.Content) == 0 {
+			return nil
+		}
+		return o.UnmarshalYAML(node.Content[0])
+	case yaml.AliasNode:
+		return o.UnmarshalYAML(node.Alias)
+	case yaml.ScalarNode:
+		return node.Decode(&o.altVal)
+	case yaml.MappingNode:
+		// set kv to non-nil
+		o.kv = []orderedMapKV{}
+		for i := 0; i < len(node.Content); i += 2 {
+			var key string
+			var val orderedMap
+			if err := node.Content[i].Decode(&key); err != nil {
+				return err
+			}
+			if err := node.Content[i+1].Decode(&val); err != nil {
+				return err
+			}
+			o.kv = append(o.kv, orderedMapKV{
+				K: key,
+				V: val,
+			})
+		}
+		return nil
+	case yaml.SequenceNode:
+		var res []orderedMap
+		if err := node.Decode(&res); err != nil {
+			return err
+		}
+		o.altVal = res
+		o.kv = nil
+		return nil
+	case 0:
+		// null
+		o.kv = nil
+		o.altVal = nil
+		return nil
+	default:
+		return fmt.Errorf("orderedMap: invalid yaml node")
+	}
+}
+
+func (o *orderedMap) MarshalYAML() (interface{}, error) {
+	// fast path: kv is nil, use altVal
+	if o.kv == nil {
+		return o.altVal, nil
+	}
+	content := make([]*yaml.Node, 0, len(o.kv)*2)
+	for _, val := range o.kv {
+		n := new(yaml.Node)
+		if err := n.Encode(val.V); err != nil {
+			return nil, err
+		}
+		content = append(content, &yaml.Node{
+			Kind:  yaml.ScalarNode,
+			Tag:   "!!str",
+			Value: val.K,
+		}, n)
+	}
+	return &yaml.Node{
+		Kind:    yaml.MappingNode,
+		Tag:     "!!map",
+		Content: content,
+	}, nil
+}

pkg/yqlib/lib.go

@@ -13,11 +13,13 @@ import (
 var log = logging.MustGetLogger("yq")
 
 type UpdateCommand struct {
-	Command             string
-	Path                string
-	Value               *yaml.Node
-	Overwrite           bool
-	DontUpdateNodeValue bool
+	Command               string
+	Path                  string
+	Value                 *yaml.Node
+	Overwrite             bool
+	DontUpdateNodeValue   bool
+	DontUpdateNodeContent bool
+	CommentsMergeStrategy CommentsMergeStrategy
 }
 
 func KindString(kind yaml.Kind) string {
@@ -49,20 +51,23 @@ func DebugNode(value *yaml.Node) {
 		}
 		encoder.Close()
 		log.Debug("Tag: %v, Kind: %v, Anchor: %v", value.Tag, KindString(value.Kind), value.Anchor)
-		log.Debug("%v", buf.String())
+		log.Debug("Head Comment: %v", value.HeadComment)
+		log.Debug("Line Comment: %v", value.LineComment)
+		log.Debug("FootComment Comment: %v", value.FootComment)
+		log.Debug("\n%v", buf.String())
 	}
 }
 
 func pathStackToString(pathStack []interface{}) string {
-	return mergePathStackToString(pathStack, false)
+	return mergePathStackToString(pathStack, UpdateArrayMergeStrategy)
 }
 
-func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
+func mergePathStackToString(pathStack []interface{}, arrayMergeStrategy ArrayMergeStrategy) string {
 	var sb strings.Builder
 	for index, path := range pathStack {
 		switch path.(type) {
 		case int, int64:
-			if appendArrays {
+			if arrayMergeStrategy == AppendArrayMergeStrategy {
 				sb.WriteString("[+]")
 			} else {
 				sb.WriteString(fmt.Sprintf("[%v]", path))
@@ -93,9 +98,7 @@ func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
 			sb.WriteString(".")
 		}
 	}
-	var pathString = sb.String()
-	log.Debug("got a path string: %v", pathString)
-	return pathString
+	return sb.String()
 }
 
 func guessKind(head interface{}, tail []interface{}, guess yaml.Kind) yaml.Kind {
@@ -133,12 +136,13 @@ func guessKind(head interface{}, tail []interface{}, guess yaml.Kind) yaml.Kind
 }
 
 type YqLib interface {
-	Get(rootNode *yaml.Node, path string, deeplyTraverseArrays bool) ([]*NodeContext, error)
+	Get(rootNode *yaml.Node, path string) ([]*NodeContext, error)
+	GetForMerge(rootNode *yaml.Node, path string, arrayMergeStrategy ArrayMergeStrategy) ([]*NodeContext, error)
 	Update(rootNode *yaml.Node, updateCommand UpdateCommand, autoCreate bool) error
 	New(path string) yaml.Node
 
 	PathStackToString(pathStack []interface{}) string
-	MergePathStackToString(pathStack []interface{}, appendArrays bool) string
+	MergePathStackToString(pathStack []interface{}, arrayMergeStrategy ArrayMergeStrategy) string
 }
 
 type lib struct {
@@ -151,21 +155,28 @@ func NewYqLib() YqLib {
 	}
 }
 
-func (l *lib) Get(rootNode *yaml.Node, path string, deeplyTraverseArrays bool) ([]*NodeContext, error) {
+func (l *lib) Get(rootNode *yaml.Node, path string) ([]*NodeContext, error) {
 	var paths = l.parser.ParsePath(path)
-	navigationStrategy := ReadNavigationStrategy(deeplyTraverseArrays)
+	navigationStrategy := ReadNavigationStrategy()
 	navigator := NewDataNavigator(navigationStrategy)
 	error := navigator.Traverse(rootNode, paths)
 	return navigationStrategy.GetVisitedNodes(), error
+}
 
+func (l *lib) GetForMerge(rootNode *yaml.Node, path string, arrayMergeStrategy ArrayMergeStrategy) ([]*NodeContext, error) {
+	var paths = l.parser.ParsePath(path)
+	navigationStrategy := ReadForMergeNavigationStrategy(arrayMergeStrategy)
+	navigator := NewDataNavigator(navigationStrategy)
+	error := navigator.Traverse(rootNode, paths)
+	return navigationStrategy.GetVisitedNodes(), error
 }
 
 func (l *lib) PathStackToString(pathStack []interface{}) string {
 	return pathStackToString(pathStack)
 }
 
-func (l *lib) MergePathStackToString(pathStack []interface{}, appendArrays bool) string {
-	return mergePathStackToString(pathStack, appendArrays)
+func (l *lib) MergePathStackToString(pathStack []interface{}, arrayMergeStrategy ArrayMergeStrategy) string {
+	return mergePathStackToString(pathStack, arrayMergeStrategy)
 }
 
 func (l *lib) New(path string) yaml.Node {
@@ -181,6 +192,10 @@ func (l *lib) Update(rootNode *yaml.Node, updateCommand UpdateCommand, autoCreat
 		var paths = l.parser.ParsePath(updateCommand.Path)
 		navigator := NewDataNavigator(UpdateNavigationStrategy(updateCommand, autoCreate))
 		return navigator.Traverse(rootNode, paths)
+	case "merge":
+		var paths = l.parser.ParsePath(updateCommand.Path)
+		navigator := NewDataNavigator(MergeNavigationStrategy(updateCommand, autoCreate))
+		return navigator.Traverse(rootNode, paths)
 	case "delete":
 		var paths = l.parser.ParsePath(updateCommand.Path)
 		lastBit, newTail := paths[len(paths)-1], paths[:len(paths)-1]

pkg/yqlib/lib_test.go

@@ -30,7 +30,7 @@ func TestLib(t *testing.T) {
 		array[0] = "a"
 		array[1] = 0
 		array[2] = "b"
-		got := subject.MergePathStackToString(array, true)
+		got := subject.MergePathStackToString(array, AppendArrayMergeStrategy)
 		test.AssertResult(t, `a.[+].b`, got)
 	})
 

pkg/yqlib/merge_navigation_strategy.go

@@ -0,0 +1,103 @@
+package yqlib
+
+import "gopkg.in/yaml.v3"
+
+type ArrayMergeStrategy uint32
+
+const (
+	UpdateArrayMergeStrategy ArrayMergeStrategy = 1 << iota
+	OverwriteArrayMergeStrategy
+	AppendArrayMergeStrategy
+)
+
+type CommentsMergeStrategy uint32
+
+const (
+	SetWhenBlankCommentsMergeStrategy CommentsMergeStrategy = 1 << iota
+	IgnoreCommentsMergeStrategy
+	OverwriteCommentsMergeStrategy
+	AppendCommentsMergeStrategy
+)
+
+func MergeNavigationStrategy(updateCommand UpdateCommand, autoCreate bool) NavigationStrategy {
+	return &NavigationStrategyImpl{
+		visitedNodes: []*NodeContext{},
+		pathParser:   NewPathParser(),
+		followAlias: func(nodeContext NodeContext) bool {
+			return false
+		},
+		autoCreateMap: func(nodeContext NodeContext) bool {
+			return autoCreate
+		},
+		visit: func(nodeContext NodeContext) error {
+			node := nodeContext.Node
+			changesToApply := updateCommand.Value
+
+			if node.Kind == yaml.DocumentNode && changesToApply.Kind != yaml.DocumentNode {
+				// when the path is empty, it matches both the top level pseudo document node
+				// and the actual top level node (e.g. map/sequence/whatever)
+				// so when we are updating with no path, make sure we update the right node.
+				node = node.Content[0]
+			}
+
+			log.Debug("going to update")
+			DebugNode(node)
+			log.Debug("with")
+			DebugNode(changesToApply)
+
+			if updateCommand.Overwrite || node.Value == "" {
+				node.Value = changesToApply.Value
+				node.Tag = changesToApply.Tag
+				node.Kind = changesToApply.Kind
+				node.Style = changesToApply.Style
+				node.Anchor = changesToApply.Anchor
+				node.Alias = changesToApply.Alias
+
+				if !updateCommand.DontUpdateNodeContent {
+					node.Content = changesToApply.Content
+				}
+			} else {
+				log.Debug("skipping update as node already has value %v and overwriteFlag is ", node.Value, updateCommand.Overwrite)
+			}
+
+			switch updateCommand.CommentsMergeStrategy {
+			case OverwriteCommentsMergeStrategy:
+				node.HeadComment = changesToApply.HeadComment
+				node.LineComment = changesToApply.LineComment
+				node.FootComment = changesToApply.FootComment
+			case SetWhenBlankCommentsMergeStrategy:
+				if node.HeadComment == "" {
+					node.HeadComment = changesToApply.HeadComment
+				}
+				if node.LineComment == "" {
+					node.LineComment = changesToApply.LineComment
+				}
+				if node.FootComment == "" {
+					node.FootComment = changesToApply.FootComment
+				}
+			case AppendCommentsMergeStrategy:
+				if node.HeadComment == "" {
+					node.HeadComment = changesToApply.HeadComment
+				} else {
+					node.HeadComment = node.HeadComment + "\n" + changesToApply.HeadComment
+				}
+				if node.LineComment == "" {
+					node.LineComment = changesToApply.LineComment
+				} else {
+					node.LineComment = node.LineComment + " " + changesToApply.LineComment
+				}
+				if node.FootComment == "" {
+					node.FootComment = changesToApply.FootComment
+				} else {
+					node.FootComment = node.FootComment + "\n" + changesToApply.FootComment
+				}
+			default:
+			}
+
+			log.Debug("result")
+			DebugNode(node)
+
+			return nil
+		},
+	}
+}

pkg/yqlib/navigation_strategy.go

@@ -11,6 +11,9 @@ type NodeContext struct {
 	Head      interface{}
 	Tail      []interface{}
 	PathStack []interface{}
+	// middle nodes are nodes that match along the original path, but not a
+	// target match of the path. This is only relevant when ShouldOnlyDeeplyVisitLeaves is false.
+	IsMiddleNode bool
 }
 
 func NewNodeContext(node *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) NodeContext {

pkg/yqlib/read_for_merge_navigation_strategy.go

@@ -0,0 +1,37 @@
+package yqlib
+
+import "gopkg.in/yaml.v3"
+
+func ReadForMergeNavigationStrategy(arrayMergeStrategy ArrayMergeStrategy) NavigationStrategy {
+	return &NavigationStrategyImpl{
+		visitedNodes: []*NodeContext{},
+		pathParser:   NewPathParser(),
+		followAlias: func(nodeContext NodeContext) bool {
+			return false
+		},
+		shouldOnlyDeeplyVisitLeaves: func(nodeContext NodeContext) bool {
+			return false
+		},
+		visit: func(nodeContext NodeContext) error {
+			return nil
+		},
+		shouldDeeplyTraverse: func(nodeContext NodeContext) bool {
+			if nodeContext.Node.Kind == yaml.SequenceNode && arrayMergeStrategy == OverwriteArrayMergeStrategy {
+				nodeContext.IsMiddleNode = false
+				return false
+			}
+
+			var isInArray = false
+			if len(nodeContext.PathStack) > 0 {
+				var lastElement = nodeContext.PathStack[len(nodeContext.PathStack)-1]
+				switch lastElement.(type) {
+				case int:
+					isInArray = true
+				default:
+					isInArray = false
+				}
+			}
+			return arrayMergeStrategy == UpdateArrayMergeStrategy || !isInArray
+		},
+	}
+}

pkg/yqlib/read_navigation_strategy.go

@@ -1,24 +1,11 @@
 package yqlib
 
-func ReadNavigationStrategy(deeplyTraverseArrays bool) NavigationStrategy {
+func ReadNavigationStrategy() NavigationStrategy {
 	return &NavigationStrategyImpl{
 		visitedNodes: []*NodeContext{},
 		pathParser:   NewPathParser(),
 		visit: func(nodeContext NodeContext) error {
 			return nil
 		},
-		shouldDeeplyTraverse: func(nodeContext NodeContext) bool {
-			var isInArray = false
-			if len(nodeContext.PathStack) > 0 {
-				var lastElement = nodeContext.PathStack[len(nodeContext.PathStack)-1]
-				switch lastElement.(type) {
-				case int:
-					isInArray = true
-				default:
-					isInArray = false
-				}
-			}
-			return deeplyTraverseArrays || !isInArray
-		},
 	}
 }

pkg/yqlib/update_navigation_strategy.go

@@ -24,12 +24,16 @@ func UpdateNavigationStrategy(updateCommand UpdateCommand, autoCreate bool) Navi
 				node.Tag = changesToApply.Tag
 				node.Kind = changesToApply.Kind
 				node.Style = changesToApply.Style
-				node.Content = changesToApply.Content
+				if !updateCommand.DontUpdateNodeContent {
+					node.Content = changesToApply.Content
+				}
 				node.Anchor = changesToApply.Anchor
 				node.Alias = changesToApply.Alias
-				node.HeadComment = changesToApply.HeadComment
-				node.LineComment = changesToApply.LineComment
-				node.FootComment = changesToApply.FootComment
+				if updateCommand.CommentsMergeStrategy != IgnoreCommentsMergeStrategy {
+					node.HeadComment = changesToApply.HeadComment
+					node.LineComment = changesToApply.LineComment
+					node.FootComment = changesToApply.FootComment
+				}
 			} else {
 				log.Debug("skipping update as node already has value %v and overwriteFlag is ", node.Value, updateCommand.Overwrite)
 			}

release_instructions.txt

@@ -4,6 +4,7 @@
 - tag git with same version number
 - make sure local build passes
 - push tag to git
+  - 3.4.0, v3
   - git push --tags
 - make local xcompile (builds binaries for all platforms)
 

scripts/doctools.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-
-brew install mkdocs libyaml
-pip3 install markdown-include
-pip3 install mkdocs-material
-

scripts/publish.sh

@@ -32,5 +32,5 @@ upload() {
     done < <(find ./build -mindepth 1 -maxdepth 1 -print0)
 }
 
-release
+# release
 upload

scripts/xcompile.sh

@@ -3,10 +3,12 @@
 # This assumes that gonative and gox is installed as per the 'one time setup' instructions
 # at https://github.com/inconshreveable/gonative
 
-gox -ldflags "${LDFLAGS}" -output="build/yq_{{.OS}}_{{.Arch}}"
+
+CGO_ENABLED=0 gox -ldflags "${LDFLAGS}" -output="build/yq_{{.OS}}_{{.Arch}}"
 # include non-default linux builds too
-gox -ldflags "${LDFLAGS}" -os=linux  -output="build/yq_{{.OS}}_{{.Arch}}"
+CGO_ENABLED=0 gox -ldflags "${LDFLAGS}" -os=linux  -output="build/yq_{{.OS}}_{{.Arch}}"
 
 cd build
 rhash -r -a . -P -o checksums
 
+rhash --list-hashes > checksums_hashes_order

snap/snapcraft.yaml

@@ -1,5 +1,5 @@
 name: yq
-version: '3.3.2'
+version: '3.4.1'
 summary: A lightweight and portable command-line YAML processor
 description: |
   The aim of the project is to be the jq or sed of yaml files.
@@ -16,7 +16,7 @@ apps:
 parts:
   yq:
     plugin: go
-    go-channel: 1.14/stable
+    go-channel: 1.15/stable
     source: .
     source-type: git
     go-importpath: github.com/mikefarah/yq

upgrading-from-v2.md

@@ -1,107 +0,0 @@
----
-description: New features and breaking changes
----
-
-# Upgrading from V2
-
-## New Features
-
-* Keeps yaml comments and formatting, can specify yaml [tags](usage/value-parsing.md#using-the-tag-field-to-override) when updating.
-* Handles anchors!
-* Can print out matching paths and values when splatting, more info [here](commands/read.md#printing-matching-paths).
-* JSON output works for all commands! Yaml files with multiple documents are printed out as one JSON document per line, more info [here](usage/convert.md)
-* Deep splat \(`**`\) to match arbitrary paths and match nodes by their children, more info [here](usage/path-expressions.md)
-
-## Breaking Changes
-
-### Parsing values from the CLI
-
-In V3 users are able to better control how values are treated when updating YAML by using a new `--tag` argument \(see more info [here](usage/value-parsing.md)\). A result of this however, is that quoting values, e.g. "true" will no longer have an effect on how the value is interpreted like it did in V2.
-
-For instance, to get the _string_ "true" into a yaml file:
-
-V2:
-
-```text
-yq n a.path '"true"'
-```
-
-V3
-
-```text
-yq n a.path --tag '!!str' true
-```
-
-### Reading paths that don't exist
-
-In V2 this would return null, V3 does not return anything.
-
-Similarly, reading null yaml values `null`, `~` and , V2 returns null whereas V3 returns the values as is.
-
-This is a result of taking effort not to format values coming in and out of the original YAML.
-
-
-
-### Update scripts file format has changed to be more powerful.
-
-Comments can be added, and delete commands have been introduced.
-
-V2
-
-```text
-b.e[+].name: Mike Farah
-```
-
-V3
-
-```yaml
-- command: update 
-  path: b.e[+].thing
-  value:
-    #great 
-    things: frog # wow!
-- command: delete
-  path: b.d
-```
-
-### Reading and splatting, matching results are printed once per line.
-
-e.g:
-
-```yaml
-parent:
-  childA: 
-    no: matches here
-  childB:
-    there: matches
-    hi: no match
-    there2: also matches
-```
-
-```text
-yq r sample.yaml 'parent.*.there*'
-```
-
-V2
-
-```text
-- null
-- - matches
-  - also matches
-```
-
-V3
-
-```text
-matches
-also matches
-```
-
-### Converting JSON to YAML
-
-As JSON is a subset of YAML, and `yq` now preserves the formatting of the passed in document, you will most likely need to use the `--prettyPrint` flag to format the JSON document as idiomatic YAML. See [Working with JSON](usage/convert.md#json-to-yaml) for more info.
-
-
-
-
-

usage/convert.md

@@ -1,94 +0,0 @@
-# Working with JSON
-
-## Yaml to Json
-
-To convert output to json, use the `--tojson` \(or `-j`\) flag. This is supported by all commands. You can change the json output format by using the [pretty print](output-format.md#pretty-print) or [indent](output-format.md#indent) flags. _Note that due to the implementation of the JSON marshaller in GO, object keys will be sorted on output \(_[_https://golang.org/pkg/encoding/json/\#Marshal_](https://golang.org/pkg/encoding/json/#Marshal)_\)._
-
-Given a sample.yaml file of:
-
-```yaml
-b:
-  c: 2
-```
-
-then
-
-```bash
-yq r -j sample.yaml
-```
-
-will output
-
-```javascript
-{"b":{"c":2}}
-```
-
-To format the json:
-
-```yaml
-yq r --prettyPrint -j sample.yaml
-```
-
-will yield
-
-```yaml
-{
-  "b": {
-    "c": 2
-  }
-}
-```
-
-### Multiple matches
-
-Each matching yaml node will be converted to json and printed out on a separate line. The [prettyPrint](output-format.md#pretty-print) and [indent](output-format.md#indent) flags will still work too. ****
-
-Given a sample.yaml file of:
-
-```yaml
-bob:
-  c: 2
-bab:
-  c: 5
-```
-
-then
-
-```bash
-yq r -j sample.yaml b*
-```
-
-will output
-
-```javascript
-{"c":2}
-{"c":5}
-```
-
-## Json to Yaml
-
-To read in json, just pass in a json file instead of yaml, it will just work - as json is a subset of yaml. However, you will probably want to [pretty print the output](output-format.md#pretty-print) to look more like an idiomatic yaml document.
-
-e.g given a json file
-
-```javascript
-{"a":"Easy! as one two three","b":{"c":2,"d":[3,4]}}
-```
-
-then
-
-```bash
-yq r --prettyPrint sample.json
-```
-
-will output
-
-```yaml
-a: Easy! as one two three
-b:
-  c: 2
-  d:
-  - 3
-  - 4
-```
-

usage/output-format.md

@@ -1,161 +0,0 @@
----
-description: Flags to control yaml and json output format
----
-
-# Output format
-
-These flags are available for all `yq` commands. 
-
-## Colorize Output
-
-Use the `--colors/-C`flag to print out yaml with colors. This does not work when outputing in JSON format.
-
-## Pretty Print
-
-Use the `--prettyPrint/-P` flag to enforce a formatting style for yaml documents. This is particularly useful when reading a json file \(which is a subset of yaml\) and wanting to format it in a more conventional yaml format.
-
-Given:
-
-```text
-{
-  "apples": [
-    {
-      "are": "great"
-    }
-  ]
-}
-```
-
-Then:
-
-```text
-yq r --prettyPrint sample.json
-```
-
-Will print out:
-
-```text
-apples:
-- are: great
-```
-
-This works in the same manner for yaml files:
-
-```text
-"apples": [are: great]
-```
-
-will format to:
-
-```text
-apples:
-- are: great
-```
-
-## Indent
-
-Use the indent flag `--indent/-I` to control the number of spaces used for indentation. This also works for JSON output. The default value is 2. 
-
-Note that lists are indented at the same level as the map key at indent level 2, but are more deeply indented at indent level 4 and greater. This is \(currently\) a quirk of the underlying [yaml parser](https://github.com/go-yaml/yaml/tree/v3).
-
-Given:
-
-```text
-apples:
-  collection:
-  - name: Green
-  - name: Blue
-  favourite: Pink Lady
-```
-
-Then:
-
-```text
-yq r -I4 sample.yaml
-```
-
-Will print out:
-
-```text
-apples:
-    collection:
-      - name: Green
-      - name: Blue
-    favourite: Pink Lady
-```
-
-With json, you must also specify the `--prettyPrint/-P` flag
-
-```text
-yq r -j -P -I4 sample.yaml
-```
-
-yields
-
-```text
-{
-    "apples": {
-        "collection": [
-            {
-                "name": "Green"
-            },
-            {
-                "name": "Blue"
-            }
-        ],
-        "favourite": "Pink Lady"
-    }
-}
-```
-
-## Unwrap scalars
-
-By default scalar values are 'unwrapped', that is only their value is printed \(except when outputting as JSON\). To print out the node as-is, with the original formatting an any comments pass in `--unwrapScalar=false`
-
-Given data.yml:
-
-```yaml
-a: "Things" # cool stuff
-```
-
-Then:
-
-`yq r --unwrapScalar=false data.yml a`
-
-Will yield:
-
-```yaml
-"Things" # cool stuff
-```
-
-where as without setting the flag to false you would get:
-
-```yaml
-Things
-```
-
-## Strip comments
-
-Use the `--stripComments` flag to print out the yaml file without any of the original comments.
-
-Given data.yml of:
-
-```yaml
-a:
-  b: # there is where the good stuff is
-    c: hi
-```
-
-Then
-
-```yaml
-yq r data.yml a --stripComments
-```
-
-Will yield:
-
-```yaml
-b:
-  c: hi
-```
-

usage/path-expressions.md

@@ -1,251 +0,0 @@
----
-description: Path expressions are used to deeply navigate and match particular yaml nodes.
----
-
-# Path Expressions
-
-_As a general rule, you should wrap paths in quotes to prevent your CLI from processing `*`, `[]` and other special characters._
-
-## Simple expressions
-
-### Maps
-
-`'a.b.c'`
-
-```yaml
-a:
-  b:
-    c: thing # MATCHES
-```
-
-### Arrays
-
-`'a.b[1].c'`
-
-```yaml
-a:
-  b:
-  - c: thing0 
-  - c: thing1 # MATCHES
-  - c: thing2
-```
-
-#### Appending to arrays
-
-\(e.g. when using the write command\)
-
-`'a.b[+].c'`
-
-```yaml
-a:
-  b:
-  - c: thing0
-```
-
-Will add a new entry:
-
-```yaml
-a:
-  b:
-  - c: thing0 
-  - c: thing1 # NEW entry from [+] on B array.
-```
-
-#### Negative Array indexes
-
-Negative array indexes can be used to traverse the array in reverse
-
-`'a.b[-1].c'`
-
-Will access the last element in the `b` array and yield:
-
-```yaml
-thing2
-```
-
-## Splat
-
-### Maps
-
-`'a.*.c'`
-
-```yaml
-a:
-  b1:
-    c: thing # MATCHES
-    d: whatever
-  b2:
-    c: thing # MATCHES
-    f: something irrelevant
-```
-
-#### Prefix splat
-
-`'bob.item*.cats'`
-
-```yaml
-bob:
-  item:
-    cats: bananas # MATCHES
-  something:
-    cats: lemons
-  itemThing:
-    cats: more bananas # MATCHES
-  item2:
-    cats: apples # MATCHES
-  thing:
-    cats: oranges
-```
-
-### Arrays
-
-`'a.b[*].c'`
-
-```yaml
-a:
-  b:
-  - c: thing0 # MATCHES
-    d: what..ever
-  - c: thing1 # MATCHES
-    d: blarh
-  - c: thing2 # MATCHES
-    f: thingamabob
-```
-
-## Deep Splat
-
-`**` will match arbitrary nodes for both maps and arrays:
-
-`'a.**.c'`
-
-```yaml
-a:
-  b1:
-    c: thing1 # MATCHES
-    d: cat cat
-  b2:
-    c: thing2 # MATCHES
-    d: dog dog
-  b3:
-    d:
-    - f:
-        c: thing3 # MATCHES
-        d: beep
-    - f:
-        g:
-          c: thing4 # MATCHES
-          d: boop
-    - d: mooo
-```
-
-## Search by children nodes
-
-You can search children by nodes - note that this will return the _parent_ of the matching expression, in the example below the parent\(s\) will be the matching indices of the 'a' array. We can then navigate down to get 'b.c' of each matching indice.
-
-`'a.(b.d==cat).b.c'`
-
-```yaml
-a:
-  - b:
-      c: thing0
-      d: leopard
-    ba: fast
-  - b:
-      c: thing1 # MATCHES
-      d: cat
-    ba: meowy
-  - b:
-      c: thing2
-      d: caterpillar
-    ba: icky
-  - b:
-      c: thing3 # MATCHES
-      d: cat
-    ba: also meowy
-```
-
-### With prefixes
-
-`'a.(b.d==cat*).c'`
-
-```yaml
-a:
-  - b:
-      c: thing0
-      d: leopard
-    ba: fast
-  - b:
-      c: thing1 # MATCHES
-      d: cat
-    ba: meowy
-  - b:
-      c: thing2 # MATCHES
-      d: caterpillar
-    ba: icky
-  - b:
-      c: thing3 # MATCHES
-      d: cat
-    ba: also meowy
-```
-
-### Matching children values
-
-`'animals(.==cat)'`
-
-```yaml
-animals:
-  - dog
-  - cat # MATCHES  
-  - rat
-```
-
-this also works in maps, and with prefixes
-
-`'animals(.==c*)'`
-
-```yaml
-animals:
-  friendliest: cow # MATCHES
-  favourite: cat # MATCHES
-  smallest: rat
-```
-
-## Special Characters
-
-When your yaml field has special characters that overlap with `yq` path expression characters, you will need to escape them in order for the command to work.
-
-### Keys with dots
-
-When specifying a key that has a dot use key lookup indicator.
-
-```yaml
-b:
-  foo.bar: 7
-```
-
-```bash
-yaml r sample.yaml 'b."foo.bar"'
-```
-
-```bash
-yaml w sample.yaml 'b."foo.bar"' 9
-```
-
-Any valid yaml key can be specified as part of a key lookup.
-
-Note that the path is in single quotes to avoid the double quotes being interpreted by your shell.
-
-### Keys \(and values\) with leading dashes
-
-The flag terminator needs to be used to stop the app from attempting to parse the subsequent arguments as flags, if they start if a dash.
-
-```bash
-yq n -j -- --key --value
-```
-
-Will result in
-
-```text
---key: --value
-```
-

usage/value-parsing.md

@@ -1,288 +0,0 @@
----
-description: >-
-  How values are parsed from the CLI to commands that create/update yaml (e.g.
-  new/write).
----
-
-# Value Parsing
-
-`yq` attempts to parse values intelligently, e.g. when a number is passed it - it will assume it's a number as opposed to a string. `yq` will not alter the representation of what you give. So if you pass '03.0' in, it will assume it's a number and keep the value formatted as it was passed in, that is '03.0'.
-
-The `--tag` flag can be used to override the tag type to force particular tags.
-
-## Default behavior
-
-### Integers
-
-_Given_
-
-```bash
-yq new key 3
-```
-
-results in
-
-```yaml
-key: 3
-```
-
-_Given a formatted number_
-
-```bash
-yq new key 03
-```
-
-results in
-
-```yaml
-key: 03
-```
-
-`yq` keeps the number formatted as it was passed in.
-
-### Float
-
-_Given_
-
-```bash
-yq new key "3.1"
-```
-
-results in
-
-```yaml
-key: 3.1
-```
-
-Note that quoting the number does not make a difference.
-
-_Given a formatted decimal number_
-
-```bash
-yq new key 03.0
-```
-
-results in
-
-```yaml
-key: 03.0
-```
-
-`yq` keeps the number formatted as it was passed in
-
-### Booleans
-
-```bash
-yq new key true
-```
-
-results in
-
-```yaml
-key: true
-```
-
-### Nulls
-
-```bash
-yq new key null
-```
-
-results in
-
-```yaml
-key: null
-```
-
-```bash
-yq new key '~'
-```
-
-results in
-
-```yaml
-key: ~
-```
-
-```bash
-yq new key ''
-```
-
-results in
-
-```yaml
-key:
-```
-
-### Strings
-
-```bash
-yq new key whatever
-```
-
-results in
-
-```yaml
-key: whatever
-```
-
-```bash
-yq new key ' whatever '
-```
-
-results in
-
-```yaml
-key: ' whatever '
-```
-
-## Using the tag flag to cast
-
-Previous versions of yq required double quoting to force values to be strings, this no longer works - instead use the --tag flag.
-
-### Casting booleans
-
-```bash
-yq new --tag '!!str' key true
-```
-
-results in
-
-```yaml
-key: "true"
-```
-
-### Casting nulls
-
-```bash
-yq new --tag '!!str' key null
-```
-
-results in
-
-```yaml
-key: "null"
-```
-
-### Custom types
-
-```bash
-yq new --tag '!!farah' key gold
-```
-
-results in
-
-```yaml
-key: !!farah gold
-```
-
-## The style flag
-
-The `--style` flag can be used to specify the quote or block style of the node value. Valid values are
-
-* single
-* double
-* folded
-* flow
-* literal
-* tagged
-
-For example, given:
-
-```bash
-MULTILINE=$(cat <<END
-    This is line one.
-    This is line two.
-END
-)
-
-SINGLE="only one line"
-```
-
-### Single
-
-```yaml
-yq n --style single things "$MULTILINE"
-```
-
-```yaml
-things: 'This is line one.
-
-  This is line two.'
-```
-
-### Double
-
-```yaml
-things: "This is line one.\nThis is line two."
-```
-
-### Folded:
-
-```yaml
-things: >-
-  This is line one.
-
-  This is line two.
-
-```
-
-#### Folded single line:
-
-```yaml
-things: >-
-  only one line
-```
-
-### Flow:
-
-```yaml
-things: |-
-  This is line one.
-  This is line two.
-
-```
-
-#### Flow single line:
-
-```yaml
-things: only one line
-```
-
-### Literal
-
-```yaml
-things: |-
-  This is line one.
-  This is line two.
-
-```
-
-#### Literal single line
-
-```yaml
-things: |-
-  only one line
-```
-
-### Tagged
-
-Always show the tag, note - you must also pass in `--tag='!!str'`
-
-```yaml
-things: !!str |-
-  This is line one.
-  This is line two.
-
-```
-
-#### Tagged single line
-
-```yaml
-things: !!str only one line
-```
-
-
-
-
-