3.4.1

This is to patch a high security vulnerability.
Closes #550

Signed-off-by: Peter Benjamin <petermbenjamin@gmail.com>

.gitbook/assets/convert.md

@@ -1,60 +0,0 @@
-## Yaml to Json
-To convert output to json, use the --tojson (or -j) flag. This is supported by all commands.
-
-Each matching yaml node will be converted to json and printed out on a separate line.
-
-Given a sample.yaml file of:
-```yaml
-b:
-  c: 2
-```
-then
-```bash
-yq r -j sample.yaml
-```
-
-will output
-```json
-{"b":{"c":2}}
-```
-
-Given a sample.yaml file of:
-```yaml
-bob:
-  c: 2
-bab:
-  c: 5
-```
-then
-```bash
-yq r -j sample.yaml b*
-```
-
-will output
-```json
-{"c":2}
-{"c":5}
-```
-
-## Json to Yaml
-To read in json, just pass in a json file instead of yaml, it will just work :)
-
-e.g given a json file
-
-```json
-{"a":"Easy! as one two three","b":{"c":2,"d":[3,4]}}
-```
-then
-```bash
-yq r sample.json
-```
-will output
-```yaml
-a: Easy! as one two three
-b:
-  c: 2
-  d:
-  - 3
-  - 4
-```
-

.github/workflows/go.yml

@@ -7,10 +7,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
 
-    - name: Set up Go 1.14
+    - name: Set up Go 1.15
       uses: actions/setup-go@v1
       with:
-        go-version: 1.14
+        go-version: 1.15
       id: go
 
     - name: Check out code into the Go module directory

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.14 as builder
+FROM golang:1.15 as builder
 
 WORKDIR /go/src/mikefarah/yq
 
@@ -12,7 +12,7 @@ RUN CGO_ENABLED=0 make local build
 
 # Choose alpine as a base image to make this useful for CI, as many
 # CI tools expect an interactive shell inside the container
-FROM alpine:3.8 as production
+FROM alpine:3.12 as production
 
 COPY --from=builder /go/src/mikefarah/yq/yq /usr/bin/yq
 RUN chmod +x /usr/bin/yq

Dockerfile.dev

@@ -1,4 +1,4 @@
-FROM golang:1.14
+FROM golang:1.15
 
 COPY scripts/devtools.sh /opt/devtools.sh
 

README.md

@@ -1,36 +1,166 @@
----
-description: yq is a lightweight and portable command-line YAML processor
----
+# yq
 
-# Yq
+![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq)
 
-### Install[¶]() <a id="install"></a>
 
-On MacOS:
+a lightweight and portable command-line YAML processor
 
-```text
+The aim of the project is to be the [jq](https://github.com/stedolan/jq) or sed of yaml files.
+
+## Install
+
+### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
+
+### MacOS:
+```
 brew install yq
 ```
 
-On Ubuntu and other Linux distros supporting `snap` packages:
-
-```text
+### Ubuntu and other Linux distros supporting `snap` packages:
+```
 snap install yq
 ```
 
-On Ubuntu 16.04 or higher from Debian package:
+#### Snap notes
+`yq` installs with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can:
 
-```text
+```
+sudo cat /etc/myfile | yq r - a.path
+```
+
+And to write to a root file you can either use [sponge](https://linux.die.net/man/1/sponge):
+```
+sudo cat /etc/myfile | yq w - a.path value | sudo sponge /etc/myfile
+```
+or write to a temporary file:
+```
+sudo cat /etc/myfile | yq w - a.path value | sudo tee /etc/myfile.tmp
+sudo mv /etc/myfile.tmp /etc/myfile
+rm /etc/myfile.tmp
+```
+
+### wget
+
+Use wget to download the pre-compiled binaries:
+
+```bash
+wget https://github.com/mikefarah/yq/releases/download/{VERSION}/{BINARY} -O /usr/bin/yq &&\
+    chmod +x /usr/bin/yq
+```
+
+For instance, VERSION=3.4.0 and BINARY=yq_linux_amd64
+
+
+### Run with Docker
+
+#### Oneshot use:
+
+```bash
+docker run --rm -v "${PWD}":/workdir mikefarah/yq yq [flags] <command> FILE...
+```
+
+#### Run commands interactively:
+
+```bash
+docker run --rm -it -v "${PWD}":/workdir mikefarah/yq sh
+```
+
+It can be useful to have a bash function to avoid typing the whole docker command:
+
+```bash
+yq() {
+  docker run --rm -i -v "${PWD}":/workdir mikefarah/yq yq "$@"
+}
+```
+
+### Go Get:
+```
+GO111MODULE=on go get github.com/mikefarah/yq/v3
+```
+
+## Community Supported Installation methods
+As these are supported by the community :heart: - however, they may be out of date with the officially supported releases.
+
+
+### Windows:
+```
+choco install yq
+```
+Supported by @chillum (https://chocolatey.org/packages/yq)
+
+### Alpine Linux
+- Enable edge/community repo by adding ```$MIRROR/alpine/edge/community``` to ```/etc/apk/repositories```
+- Update database index with ```apk update```
+- Install yq with ```apk add yq```
+
+Supported by Tuan Hoang
+https://pkgs.alpinelinux.org/package/edge/community/x86/yq
+
+
+### On Ubuntu 16.04 or higher from Debian package:
+```sh
+sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CC86BB64
 sudo add-apt-repository ppa:rmescandon/yq
 sudo apt update
 sudo apt install yq -y
 ```
+Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
 
-or, [Download latest binary](https://github.com/mikefarah/yq/releases/latest) or alternatively:
+## Features
+- Written in portable go, so you can download a lovely dependency free binary
+- [Colorize the output](https://mikefarah.gitbook.io/yq/usage/output-format#colorize-output)
+- [Deep read a yaml file with a given path expression](https://mikefarah.gitbook.io/yq/commands/read#basic)
+- [List matching paths of a given path expression](https://mikefarah.gitbook.io/yq/commands/read#path-only)
+- [Return the lengths of arrays/object/scalars](https://mikefarah.gitbook.io/yq/commands/read#printing-length-of-the-results)
+- Update a yaml file given a [path expression](https://mikefarah.gitbook.io/yq/commands/write-update#basic) or [script file](https://mikefarah.gitbook.io/yq/commands/write-update#basic)
+- Update creates any missing entries in the path on the fly
+- Deeply [compare](https://mikefarah.gitbook.io/yq/commands/compare) yaml files
+- Keeps yaml formatting and comments when updating
+- [Validate a yaml file](https://mikefarah.gitbook.io/yq/commands/validate)
+- Create a yaml file given a [deep path and value](https://mikefarah.gitbook.io/yq/commands/create#creating-a-simple-yaml-file) or a [script file](https://mikefarah.gitbook.io/yq/commands/create#creating-using-a-create-script)
+- [Prefix a path to a yaml file](https://mikefarah.gitbook.io/yq/commands/prefix)
+- [Convert to/from json to yaml](https://mikefarah.gitbook.io/yq/usage/convert)
+- [Pipe data in by using '-'](https://mikefarah.gitbook.io/yq/commands/read#from-stdin)
+- [Merge](https://mikefarah.gitbook.io/yq/commands/merge) multiple yaml files with various options for [overriding](https://mikefarah.gitbook.io/yq/commands/merge#overwrite-values) and [appending](https://mikefarah.gitbook.io/yq/commands/merge#append-values-with-arrays)
+- Supports multiple documents in a single yaml file for [reading](https://mikefarah.gitbook.io/yq/commands/read#multiple-documents), [writing](https://mikefarah.gitbook.io/yq/commands/write-update#multiple-documents) and [merging](https://mikefarah.gitbook.io/yq/commands/merge#multiple-documents)
+- General shell completion scripts (bash/zsh/fish/powershell) (https://mikefarah.gitbook.io/yq/commands/shell-completion)
 
-```text
-GO111MODULE=on go get github.com/mikefarah/yq/v2
+## [Usage](https://mikefarah.gitbook.io/yq/)
+
+Check out the [documentation](https://mikefarah.gitbook.io/yq/) for more detailed and advanced usage.
+
+```
+Usage:
+  yq [flags]
+  yq [command]
+
+Available Commands:
+  compare          yq x [--prettyPrint/-P] dataA.yaml dataB.yaml 'b.e(name==fr*).value'
+  delete           yq d [--inplace/-i] [--doc/-d index] sample.yaml 'b.e(name==fred)'
+  help             Help about any command
+  merge            yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--append/-a] sample.yaml sample2.yaml
+  new              yq n [--script/-s script_file] a.b.c newValue
+  prefix           yq p [--inplace/-i] [--doc/-d index] sample.yaml a.b.c
+  read             yq r [--printMode/-p pv] sample.yaml 'b.e(name==fr*).value'
+  shell-completion Generates shell completion scripts
+  validate         yq v sample.yaml
+  write            yq w [--inplace/-i] [--script/-s script_file] [--doc/-d index] sample.yaml 'b.e(name==fr*).value' newValue
+
+Flags:
+  -C, --colors        print with colors
+  -h, --help          help for yq
+  -I, --indent int    sets indent level for output (default 2)
+  -P, --prettyPrint   pretty print
+  -j, --tojson        output as json. By default it prints a json document in one line, use the prettyPrint flag to print a formatted doc.
+  -v, --verbose       verbose mode
+  -V, --version       Print version information and quit
+
+Use "yq [command] --help" for more information about a command.
 ```
 
-[View on GitHub](https://github.com/mikefarah/yq)
+## Upgrade from V2
+If you've been using v2 and want/need to upgrade, checkout the [upgrade guide](https://mikefarah.gitbook.io/yq/upgrading-from-v2).
 
+## Known Issues / Missing Features
+- `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
+- You cannot (yet) select multiple paths/keys from the yaml to be printed out (https://github.com/mikefarah/yq/issues/287)

SUMMARY.md

@@ -1,11 +0,0 @@
-# Table of contents
-
-* [Yq](README.md)
-* [Write](write-update.md)
-* [Read](read.md)
-* [Prefix](prefix.md)
-* [Delete](delete.md)
-* [Create](create.md)
-* [Convert](convert.md)
-* [Merge](merge.md)
-

cmd/constant.go

@@ -26,7 +26,8 @@ var defaultValue = ""
 var indent = 2
 var overwriteFlag = false
 var autoCreateFlag = true
-var appendFlag = false
+var arrayMergeStrategyFlag = "update"
+var commentsMergeStrategyFlag = "setWhenBlank"
 var verbose = false
 var version = false
 var docIndex = "0"

cmd/merge.go

@@ -11,14 +11,14 @@ func createMergeCmd() *cobra.Command {
 	var cmdMerge = &cobra.Command{
 		Use:     "merge [initial_yaml_file] [additional_yaml_file]...",
 		Aliases: []string{"m"},
-		Short:   "yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--append/-a] sample.yaml sample2.yaml",
+		Short:   "yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--arrayMerge/-a strategy] sample.yaml sample2.yaml",
 		Example: `
 yq merge things.yaml other.yaml
 yq merge --inplace things.yaml other.yaml
 yq m -i things.yaml other.yaml
 yq m --overwrite things.yaml other.yaml
 yq m -i -x things.yaml other.yaml
-yq m -i -a things.yaml other.yaml
+yq m -i -a=append things.yaml other.yaml
 yq m -i --autocreate=false things.yaml other.yaml
       `,
 		Long: `Updates the yaml file by adding/updating the path(s) and value(s) from additional yaml file(s).
@@ -32,7 +32,17 @@ If append flag is set then existing arrays will be merged with the arrays from e
 	cmdMerge.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the yaml file inplace")
 	cmdMerge.PersistentFlags().BoolVarP(&overwriteFlag, "overwrite", "x", false, "update the yaml file by overwriting existing values")
 	cmdMerge.PersistentFlags().BoolVarP(&autoCreateFlag, "autocreate", "c", true, "automatically create any missing entries")
-	cmdMerge.PersistentFlags().BoolVarP(&appendFlag, "append", "a", false, "update the yaml file by appending array values")
+	cmdMerge.PersistentFlags().StringVarP(&arrayMergeStrategyFlag, "arrays", "a", "update", `array merge strategy (update/append/overwrite)
+update: recursively update arrays by their index
+append: concatenate arrays together
+overwrite: replace arrays
+`)
+	cmdMerge.PersistentFlags().StringVarP(&commentsMergeStrategyFlag, "comments", "", "setWhenBlank", `comments merge strategy (setWhenBlank/ignore/append/overwrite)
+setWhenBlank: set comment if the original document has no comment at that node
+ignore: leave comments as-is in the original
+append: append comments together
+overwrite: overwrite comments completely
+`)
 	cmdMerge.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
 	return cmdMerge
 }
@@ -41,9 +51,9 @@ If append flag is set then existing arrays will be merged with the arrays from e
 * We don't deeply traverse arrays when appending a merge, instead we want to
 * append the entire array element.
  */
-func createReadFunctionForMerge() func(*yaml.Node) ([]*yqlib.NodeContext, error) {
+func createReadFunctionForMerge(arrayMergeStrategy yqlib.ArrayMergeStrategy) func(*yaml.Node) ([]*yqlib.NodeContext, error) {
 	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
-		return lib.Get(dataBucket, "**", !appendFlag)
+		return lib.GetForMerge(dataBucket, "**", arrayMergeStrategy)
 	}
 }
 
@@ -53,19 +63,59 @@ func mergeProperties(cmd *cobra.Command, args []string) error {
 	if len(args) < 1 {
 		return errors.New("Must provide at least 1 yaml file")
 	}
+	var arrayMergeStrategy yqlib.ArrayMergeStrategy
+
+	switch arrayMergeStrategyFlag {
+	case "update":
+		arrayMergeStrategy = yqlib.UpdateArrayMergeStrategy
+	case "append":
+		arrayMergeStrategy = yqlib.AppendArrayMergeStrategy
+	case "overwrite":
+		arrayMergeStrategy = yqlib.OverwriteArrayMergeStrategy
+	default:
+		return errors.New("Array merge strategy must be one of: update/append/overwrite")
+	}
+
+	var commentsMergeStrategy yqlib.CommentsMergeStrategy
+
+	switch commentsMergeStrategyFlag {
+	case "setWhenBlank":
+		commentsMergeStrategy = yqlib.SetWhenBlankCommentsMergeStrategy
+	case "ignore":
+		commentsMergeStrategy = yqlib.IgnoreCommentsMergeStrategy
+	case "append":
+		commentsMergeStrategy = yqlib.AppendCommentsMergeStrategy
+	case "overwrite":
+		commentsMergeStrategy = yqlib.OverwriteCommentsMergeStrategy
+	default:
+		return errors.New("Comments merge strategy must be one of: setWhenBlank/ignore/append/overwrite")
+	}
 
 	if len(args) > 1 {
 		// first generate update commands from the file
 		var filesToMerge = args[1:]
 
 		for _, fileToMerge := range filesToMerge {
-			matchingNodes, errorProcessingFile := doReadYamlFile(fileToMerge, createReadFunctionForMerge(), false, 0)
+			matchingNodes, errorProcessingFile := doReadYamlFile(fileToMerge, createReadFunctionForMerge(arrayMergeStrategy), false, 0)
 			if errorProcessingFile != nil {
 				return errorProcessingFile
 			}
+			log.Debugf("finished reading for merge!")
 			for _, matchingNode := range matchingNodes {
-				mergePath := lib.MergePathStackToString(matchingNode.PathStack, appendFlag)
-				updateCommands = append(updateCommands, yqlib.UpdateCommand{Command: "update", Path: mergePath, Value: matchingNode.Node, Overwrite: overwriteFlag})
+				log.Debugf("matched node %v", lib.PathStackToString(matchingNode.PathStack))
+				yqlib.DebugNode(matchingNode.Node)
+			}
+			for _, matchingNode := range matchingNodes {
+				mergePath := lib.MergePathStackToString(matchingNode.PathStack, arrayMergeStrategy)
+				updateCommands = append(updateCommands, yqlib.UpdateCommand{
+					Command:               "merge",
+					Path:                  mergePath,
+					Value:                 matchingNode.Node,
+					Overwrite:             overwriteFlag,
+					CommentsMergeStrategy: commentsMergeStrategy,
+					// dont update the content for nodes midway, only leaf nodes
+					DontUpdateNodeContent: matchingNode.IsMiddleNode && (arrayMergeStrategy != yqlib.OverwriteArrayMergeStrategy || matchingNode.Node.Kind != yaml.SequenceNode),
+				})
 			}
 		}
 	}

cmd/merge_test.go

@@ -60,7 +60,7 @@ func TestMergeOverwriteCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `a: other # better than the original
+	expectedOutput := `a: other # just the best
 b: [3, 4]
 c:
   test: 1
@@ -68,9 +68,36 @@ c:
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestMergeOverwriteDeepExampleCmd(t *testing.T) {
+	content := `c:
+  test: 1
+  thing: whatever
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `c:
+  test: 5
+`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --autocreate=false --overwrite %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `c:
+  test: 5
+  thing: whatever
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestMergeAppendCmd(t *testing.T) {
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --autocreate=false --append ../examples/data1.yaml ../examples/data2.yaml")
+	result := test.RunCmd(cmd, "merge --autocreate=false --arrays=append ../examples/data1.yaml ../examples/data2.yaml")
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
@@ -96,7 +123,7 @@ func TestMergeAppendArraysCmd(t *testing.T) {
 	defer test.RemoveTempYamlFile(mergeFilename)
 
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("merge --append -d* %s %s", filename, mergeFilename))
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --arrays=append -d* %s %s", filename, mergeFilename))
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
@@ -109,13 +136,56 @@ func TestMergeAppendArraysCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
-func TestMergeOverwriteAndAppendCmd(t *testing.T) {
+func TestMergeAliasArraysCmd(t *testing.T) {
+	content := `
+vars:
+  variable1: &var1 cat
+
+usage:
+  value1: *var1
+  valueAnother: *var1
+  valuePlain: thing
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `
+vars:
+  variable2: &var2 puppy
+
+usage:
+  value2: *var2
+  valueAnother: *var2
+  valuePlain: *var2
+`
+
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --autocreate=false --append --overwrite ../examples/data1.yaml ../examples/data2.yaml")
+	result := test.RunCmd(cmd, fmt.Sprintf("merge -x %s %s", filename, mergeFilename))
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `a: other # better than the original
+	expectedOutput := `vars:
+  variable1: &var1 cat
+  variable2: &var2 puppy
+usage:
+  value1: *var1
+  valueAnother: *var2
+  valuePlain: *var2
+  value2: *var2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOverwriteAndAppendCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --autocreate=false --arrays=append --overwrite ../examples/data1.yaml ../examples/data2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: other # just the best
 b: [1, 2, 3, 4]
 c:
   test: 1
@@ -123,13 +193,161 @@ c:
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
-func TestMergeArraysCmd(t *testing.T) {
+var commentContentA = `
+a: valueA1 # commentA1
+b: valueB1
+`
+
+var commentContentB = `
+a: valueA2 # commentA2
+b: valueB2 # commentB2
+c: valueC2 # commentC2
+`
+
+func TestMergeCommentsSetWhenBlankCmd(t *testing.T) {
+	filename := test.WriteTempYamlFile(commentContentA)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeFilename := test.WriteTempYamlFile(commentContentB)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "merge --append ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --comments=setWhenBlank %s %s", filename, mergeFilename))
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `[1, 2, 3, 4, 5]
+
+	expectedOutput := `a: valueA1 # commentA1
+b: valueB1 # commentB2
+c: valueC2 # commentC2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeCommentsIgnoreCmd(t *testing.T) {
+	filename := test.WriteTempYamlFile(commentContentA)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeFilename := test.WriteTempYamlFile(commentContentB)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --comments=ignore %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: valueA1 # commentA1
+b: valueB1
+c: valueC2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeCommentsAppendCmd(t *testing.T) {
+	filename := test.WriteTempYamlFile(commentContentA)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeFilename := test.WriteTempYamlFile(commentContentB)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --comments=append %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: valueA1 # commentA1 # commentA2
+b: valueB1 # commentB2
+c: valueC2 # commentC2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeCommentsOverwriteCmd(t *testing.T) {
+	filename := test.WriteTempYamlFile(commentContentA)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeFilename := test.WriteTempYamlFile(commentContentB)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --comments=overwrite %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: valueA1 # commentA2
+b: valueB1 # commentB2
+c: valueC2 # commentC2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOverwriteArraysTooCmd(t *testing.T) {
+	content := `a: simple # just the best
+b: [1, 2]
+c:
+  test: 1
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `a: things
+b: [6]`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --autocreate=false --arrays=overwrite --overwrite %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: things # just the best
+b: [6]
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeRootArraysCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --arrays=append ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- 1
+- 2
+- 3
+- 4
+- 5
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOverwriteArraysCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --arrays=overwrite ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- 4
+- 5
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeUpdateArraysCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge -x --arrays=update ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- 4
+- 5
+- 3
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
@@ -145,9 +363,7 @@ func TestMergeCmd_Multi(t *testing.T) {
 another:
   document: here
 a: simple # just the best
-b:
-  - 1
-  - 2
+b: [1, 2]
 c:
   test: 1
 ---
@@ -316,9 +532,7 @@ func TestMergeAllowEmptyTargetCmd(t *testing.T) {
 		t.Error(result.Error)
 	}
 	expectedOutput := `a: simple # just the best
-b:
-  - 1
-  - 2
+b: [1, 2]
 c:
   test: 1
 `

cmd/new.go

@@ -2,7 +2,6 @@ package cmd
 
 import (
 	"github.com/mikefarah/yq/v3/pkg/yqlib"
-	errors "github.com/pkg/errors"
 	"github.com/spf13/cobra"
 )
 
@@ -36,11 +35,7 @@ Note that you can give a create script to perform more sophisticated yaml. This
 
 func newProperty(cmd *cobra.Command, args []string) error {
 	var badArgsMessage = "Must provide <path_to_update> <value>"
-	if len(args) != 2 {
-		return errors.New(badArgsMessage)
-	}
-
-	var updateCommands, updateCommandsError = readUpdateCommands(args, 2, badArgsMessage)
+	var updateCommands, updateCommandsError = readUpdateCommands(args, 2, badArgsMessage, false)
 	if updateCommandsError != nil {
 		return updateCommandsError
 	}

cmd/new_test.go

@@ -1,6 +1,7 @@
 package cmd
 
 import (
+	"fmt"
 	"testing"
 
 	"github.com/mikefarah/yq/v3/test"
@@ -18,6 +19,24 @@ func TestNewCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestNewCmdScript(t *testing.T) {
+	updateScript := `- command: update
+  path: b.c
+  value: 7`
+	scriptFilename := test.WriteTempYamlFile(updateScript)
+	defer test.RemoveTempYamlFile(scriptFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("new --script %s", scriptFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 7
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestNewAnchorCmd(t *testing.T) {
 	cmd := getRootCommand()
 	result := test.RunCmd(cmd, "new b.c 3 --anchorName=fred")

cmd/read_test.go

@@ -94,6 +94,28 @@ func TestReadUnwrapJsonByDefaultCmd(t *testing.T) {
 	test.AssertResult(t, "\"frog\"\n", result.Output)
 }
 
+func TestReadOutputJsonNonStringKeysCmd(t *testing.T) {
+
+	content := `
+true: true
+5:
+  null:
+    0.1: deeply
+    false: things`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read %s -j", filename))
+
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `{"true":true,"5":{"null":{"0.1":"deeply","false":"things"}}}
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestReadWithAdvancedFilterCmd(t *testing.T) {
 	cmd := getRootCommand()
 	result := test.RunCmd(cmd, "read ../examples/sample.yaml b.e(name==sam).value")
@@ -192,7 +214,7 @@ func TestReadArrayLengthCmd(t *testing.T) {
 }
 
 func TestReadArrayLengthDeepCmd(t *testing.T) {
-	content := `holder: 
+	content := `holder:
 - things
 - whatever
 `
@@ -208,12 +230,12 @@ func TestReadArrayLengthDeepCmd(t *testing.T) {
 }
 
 func TestReadArrayLengthDeepMultipleCmd(t *testing.T) {
-	content := `holderA: 
+	content := `holderA:
 - things
 - whatever
 skipMe:
 - yep
-holderB: 
+holderB:
 - other things
 - cool
 `
@@ -272,10 +294,10 @@ func TestReadCollectArrayCmd(t *testing.T) {
 }
 
 func TestReadArrayLengthDeepMultipleWithPathCmd(t *testing.T) {
-	content := `holderA: 
+	content := `holderA:
 - things
 - whatever
-holderB: 
+holderB:
 - other things
 - cool
 `
@@ -306,7 +328,7 @@ dog: bark
 }
 
 func TestReadObjectLengthDeepCmd(t *testing.T) {
-	content := `holder: 
+	content := `holder:
   cat: meow
   dog: bark
 `
@@ -322,10 +344,10 @@ func TestReadObjectLengthDeepCmd(t *testing.T) {
 }
 
 func TestReadObjectLengthDeepMultipleCmd(t *testing.T) {
-	content := `holderA: 
+	content := `holderA:
   cat: meow
   dog: bark
-holderB: 
+holderB:
   elephant: meow
   zebra: bark
 `
@@ -341,10 +363,10 @@ holderB:
 }
 
 func TestReadObjectLengthDeepMultipleWithPathsCmd(t *testing.T) {
-	content := `holderA: 
+	content := `holderA:
   cat: meow
   dog: bark
-holderB: 
+holderB:
   elephant: meow
   zebra: bark
 `
@@ -400,7 +422,7 @@ func TestReadSingleQuotedStringCmd(t *testing.T) {
 
 func TestReadQuotedMultinlineStringCmd(t *testing.T) {
 	content := `test: |
-  abcdefg    
+  abcdefg
   hijklmno
 `
 	filename := test.WriteTempYamlFile(content)
@@ -411,7 +433,7 @@ func TestReadQuotedMultinlineStringCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `abcdefg    
+	expectedOutput := `abcdefg
 hijklmno
 
 `
@@ -420,7 +442,7 @@ hijklmno
 
 func TestReadQuotedMultinlineNoNewLineStringCmd(t *testing.T) {
 	content := `test: |-
-  abcdefg    
+  abcdefg
   hijklmno
 `
 	filename := test.WriteTempYamlFile(content)
@@ -431,7 +453,7 @@ func TestReadQuotedMultinlineNoNewLineStringCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `abcdefg    
+	expectedOutput := `abcdefg
 hijklmno
 `
 	test.AssertResult(t, expectedOutput, result.Output)
@@ -561,7 +583,7 @@ func TestReadMergeAnchorsExplodeJsonCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `{"bar":{"b":2,"c":"oldbar","thing":"coconut"},"foo":{"a":"original","thing":"coolasdf","thirsty":"yep"},"foobar":{"a":"original","c":3,"thing":"ice","thirsty":"yep","thirty":"well beyond"},"foobarList":{"a":"original","b":2,"c":"newbar","thing":"coconut","thirsty":"yep"}}
+	expectedOutput := `{"foo":{"a":"original","thing":"coolasdf","thirsty":"yep"},"bar":{"b":2,"thing":"coconut","c":"oldbar"},"foobarList":{"c":"newbar","b":2,"thing":"coconut","a":"original","thirsty":"yep"},"foobar":{"thirty":"well beyond","thing":"ice","c":3,"a":"original","thirsty":"yep"}}
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
@@ -595,6 +617,44 @@ pointer: *value-pointer`
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestReadMergeAnchorsExplodeMissingCmd(t *testing.T) {
+	content := `a:
+  <<: &anchor
+  c: d
+  e: f
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -X %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a:
+  c: d
+  e: f
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestReadMergeAnchorsExplodeKeyCmd(t *testing.T) {
+	content := `name: &nameField Mike
+*nameField: Great Guy`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -X %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `name: Mike
+Mike: Great Guy
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestReadMergeAnchorsExplodeSimpleArrayCmd(t *testing.T) {
 	content := `- things`
 	filename := test.WriteTempYamlFile(content)
@@ -850,9 +910,9 @@ func TestReadEmptyContentCmd(t *testing.T) {
 }
 
 func TestReadEmptyNodesPrintPathCmd(t *testing.T) {
-	content := `map: 
+	content := `map:
   that: {}
-array: 
+array:
   great: []
 null:
   indeed: ~`
@@ -902,6 +962,7 @@ b:
       value: 3
     - name: sam
       value: 4
+ab: must appear last
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
@@ -941,6 +1002,7 @@ b:
           value: 3
         - name: sam
           value: 4
+ab: must appear last
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
@@ -1209,7 +1271,7 @@ func TestReadBadDataCmd(t *testing.T) {
 
 func TestReadDeepFromRootCmd(t *testing.T) {
 	content := `state:
-  country: 
+  country:
     city: foo
 `
 	filename := test.WriteTempYamlFile(content)
@@ -1379,3 +1441,23 @@ func TestReadFindValueDeepObjectCmd(t *testing.T) {
 `
 	test.AssertResult(t, expectedOutput, result.Output)
 }
+
+func TestReadKeepsKeyOrderInJson(t *testing.T) {
+	const content = `{
+	"z": "One",
+	"a": 1,
+	"w": ["a", "r"],
+	"u": {"d": "o", "0": 11.5},
+}`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("r -j %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `{"z":"One","a":1,"w":["a","r"],"u":{"d":"o","0":11.5}}
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}

cmd/root.go

@@ -22,6 +22,7 @@ func New() *cobra.Command {
 			return nil
 		},
 		PersistentPreRun: func(cmd *cobra.Command, args []string) {
+			cmd.SetOut(cmd.OutOrStdout())
 			var format = logging.MustStringFormatter(
 				`%{color}%{time:15:04:05} %{shortfunc} [%{level:.4s}]%{color:reset} %{message}`,
 			)

cmd/utils.go

@@ -17,7 +17,7 @@ type readDataFn func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error)
 
 func createReadFunction(path string) func(*yaml.Node) ([]*yqlib.NodeContext, error) {
 	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
-		return lib.Get(dataBucket, path, true)
+		return lib.Get(dataBucket, path)
 	}
 }
 
@@ -163,6 +163,9 @@ func setIfNotThere(node *yaml.Node, key string, value *yaml.Node) {
 }
 
 func applyAlias(node *yaml.Node, alias *yaml.Node) {
+	if alias == nil {
+		return
+	}
 	for index := 0; index < len(alias.Content); index = index + 2 {
 		keyNode := alias.Content[index]
 		log.Debugf("applying alias key %v", keyNode.Value)
@@ -185,12 +188,14 @@ func explodeNode(node *yaml.Node) error {
 		return nil
 	case yaml.AliasNode:
 		log.Debugf("its an alias!")
-		node.Kind = node.Alias.Kind
-		node.Style = node.Alias.Style
-		node.Tag = node.Alias.Tag
-		node.Content = node.Alias.Content
-		node.Value = node.Alias.Value
-		node.Alias = nil
+		if node.Alias != nil {
+			node.Kind = node.Alias.Kind
+			node.Style = node.Alias.Style
+			node.Tag = node.Alias.Tag
+			node.Content = node.Alias.Content
+			node.Value = node.Alias.Value
+			node.Alias = nil
+		}
 		return nil
 	case yaml.MappingNode:
 		for index := 0; index < len(node.Content); index = index + 2 {
@@ -202,6 +207,10 @@ func explodeNode(node *yaml.Node) error {
 				if errorInContent != nil {
 					return errorInContent
 				}
+				errorInContent = explodeNode(keyNode)
+				if errorInContent != nil {
+					return errorInContent
+				}
 			} else {
 				if valueNode.Kind == yaml.SequenceNode {
 					log.Debugf("an alias merge list!")
@@ -477,7 +486,7 @@ type updateCommandParsed struct {
 	Value   yaml.Node
 }
 
-func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string) ([]yqlib.UpdateCommand, error) {
+func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string, allowNoValue bool) ([]yqlib.UpdateCommand, error) {
 	var updateCommands []yqlib.UpdateCommand = make([]yqlib.UpdateCommand, 0)
 	if writeScript != "" {
 		var parsedCommands = make([]updateCommandParsed, 0)
@@ -511,8 +520,9 @@ func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string)
 		log.Debug("args %v", args)
 		log.Debug("path %v", args[expectedArgs-2])
 		log.Debug("Value %v", args[expectedArgs-1])
-		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: valueParser.Parse(args[expectedArgs-1], customTag, customStyle, anchorName, makeAlias), Overwrite: true}
-	} else if len(args) == expectedArgs-1 {
+		value := valueParser.Parse(args[expectedArgs-1], customTag, customStyle, anchorName, makeAlias)
+		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: value, Overwrite: true, CommentsMergeStrategy: yqlib.IgnoreCommentsMergeStrategy}
+	} else if len(args) == expectedArgs-1 && allowNoValue {
 		// don't update the value
 		updateCommands = make([]yqlib.UpdateCommand, 1)
 		log.Debug("args %v", args)

cmd/validate.go

@@ -15,7 +15,7 @@ yq v - # reads from stdin
 `,
 		RunE:          validateProperty,
 		SilenceUsage:  true,
-		SilenceErrors: true,
+		SilenceErrors: false,
 	}
 	cmdRead.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
 	return cmdRead

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "3.3.2"
+	Version = "3.4.1"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release

cmd/write.go

@@ -53,7 +53,7 @@ format is list of update commands (update or delete) like so:
 }
 
 func writeProperty(cmd *cobra.Command, args []string) error {
-	var updateCommands, updateCommandsError = readUpdateCommands(args, 3, "Must provide <filename> <path_to_update> <value>")
+	var updateCommands, updateCommandsError = readUpdateCommands(args, 3, "Must provide <filename> <path_to_update> <value>", true)
 	if updateCommandsError != nil {
 		return updateCommandsError
 	}

cmd/write_test.go

@@ -27,6 +27,24 @@ func TestWriteCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestWriteKeepCommentsCmd(t *testing.T) {
+	content := `b:
+  c: 3 # comment
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 7 # comment
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestWriteWithTaggedStyleCmd(t *testing.T) {
 	content := `b:
   c: dog

convert.md

@@ -1,52 +0,0 @@
-# Convert
-
-### Yaml to Json[¶](convert.md#yaml-to-json) <a id="yaml-to-json"></a>
-
-To convert output to json, use the --tojson \(or -j\) flag. This can only be used with the read command.
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-```
-
-then
-
-```text
-yq r -j sample.yaml b.c
-```
-
-will output
-
-```text
-{"b":{"c":2}}
-```
-
-### Json to Yaml[¶](convert.md#json-to-yaml) <a id="json-to-yaml"></a>
-
-To read in json, just pass in a json file instead of yaml, it will just work :\)
-
-e.g given a json file
-
-```text
-{"a":"Easy! as one two three","b":{"c":2,"d":[3,4]}}
-```
-
-then
-
-```text
-yq r sample.json
-```
-
-will output
-
-```text
-a: Easy! as one two three
-b:
-  c: 2
-  d:
-  - 3
-  - 4
-```
-

create.md

@@ -1,88 +0,0 @@
-# Create
-
- Yaml files can be created using the 'new' command. This works in the same way as the write command, but you don't pass in an existing Yaml file. Currently this does not support creating multiple documents in a single yaml file.
-
-```text
-yq n <path> <new value>
-```
-
-### Creating a simple yaml file[¶](create.md#creating-a-simple-yaml-file) <a id="creating-a-simple-yaml-file"></a>
-
-```text
-yq n b.c cat
-```
-
-will output:
-
-```text
-b:
-  c: cat
-```
-
-### Creating using a create script[¶](create.md#creating-using-a-create-script) <a id="creating-using-a-create-script"></a>
-
-Create scripts follow the same format as the update scripts.
-
-Given a script create\_instructions.yaml of:
-
-```text
-b.c: 3
-b.e[+].name: Howdy Partner
-```
-
-then
-
-```text
-yq n -s create_instructions.yaml
-```
-
-will output:
-
-```text
-b:
-  c: 3
-  e:
-    - name: Howdy Partner
-```
-
-You can also pipe the instructions in:
-
-```text
-cat create_instructions.yaml | yq n -s -
-```
-
-### Keys with dots[¶](create.md#keys-with-dots) <a id="keys-with-dots"></a>
-
-When specifying a key that has a dot use key lookup indicator.
-
-```text
-b:
-  foo.bar: 7
-```
-
-```text
-yaml r sample.yaml 'b[foo.bar]'
-```
-
-```text
-yaml w sample.yaml 'b[foo.bar]' 9
-```
-
-Any valid yaml key can be specified as part of a key lookup.
-
-Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
-
-### Keys \(and values\) with leading dashes[¶](create.md#keys-and-values-with-leading-dashes) <a id="keys-and-values-with-leading-dashes"></a>
-
-If a key or value has leading dashes, yq won't know that you are passing a value as opposed to a flag \(and you will get a 'bad flag syntax' error\).
-
-To fix that, you will need to tell it to stop processing flags by adding '--' after the last flag like so:
-
-```text
-yq n -t -- --key --value
-```
-
-Will result in
-
-``` --key: --value``
-

debian/changelog

@@ -1,3 +1,10 @@
+yq (3.3.2) focal; urgency=medium
+
+  * Bug fix: existStatus bug (#459)
+  * Automatically makes a os temp directory if it does not exist (#461)
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Fri, 07 Aug 2020 18:53:01 +0200
+
 yq (3.3-0) focal; urgency=medium
 
   * You can control string styles (quotes) using the new --style flag

delete.md

@@ -1,280 +0,0 @@
-# Delete
-
-```text
-yq d <yaml_file> <path_to_delete>
-```
-
-### To Stdout[¶](delete.md#to-stdout) <a id="to-stdout"></a>
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-  apples: green
-```
-
-then
-
-```text
-yq d sample.yaml b.c
-```
-
-will output:
-
-```text
-b:
-  apples: green
-```
-
-### From STDIN[¶](delete.md#from-stdin) <a id="from-stdin"></a>
-
-```text
-cat sample.yaml | yq d - b.c
-```
-
-### Deleting array elements[¶](delete.md#deleting-array-elements) <a id="deleting-array-elements"></a>
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 
-    - 1
-    - 2
-    - 3
-```
-
-then
-
-```text
-yq d sample.yaml 'b.c[1]'
-```
-
-will output:
-
-```text
-b:
-  c:
-  - 1
-  - 3
-```
-
-### Deleting nodes in-place[¶](delete.md#deleting-nodes-in-place) <a id="deleting-nodes-in-place"></a>
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-  apples: green
-```
-
-then
-
-```text
-yq d -i sample.yaml b.c
-```
-
-will update the sample.yaml file so that the 'c' node is deleted
-
-### Splat[¶](delete.md#splat) <a id="splat"></a>
-
-Given a sample.yaml file of:
-
-```text
----
-bob:
-  item1:
-    cats: bananas
-    dogs: woof
-  item2:
-    cats: apples
-    dogs: woof2
-  thing:
-    cats: oranges
-    dogs: woof3
-```
-
-then
-
-```text
-yq d sample.yaml bob.*.cats
-```
-
-will output:
-
-```text
----
-bob:
-  item1:
-    dogs: woof
-  item2:
-    dogs: woof2
-  thing:
-    dogs: woof3
-```
-
-### Prefix Splat[¶](delete.md#prefix-splat) <a id="prefix-splat"></a>
-
-Given a sample.yaml file of:
-
-```text
----
-bob:
-  item1:
-    cats: bananas
-    dogs: woof
-  item2:
-    cats: apples
-    dogs: woof2
-  thing:
-    cats: oranges
-    dogs: woof3
-```
-
-then
-
-```text
-yq d sample.yaml bob.item*.cats
-```
-
-will output:
-
-```text
----
-bob:
-  item1:
-    dogs: woof
-  item2:
-    dogs: woof2
-  thing:
-    cats: oranges
-    dogs: woof3
-```
-
-### Array Splat[¶](delete.md#array-splat) <a id="array-splat"></a>
-
-Given a sample.yaml file of:
-
-```text
----
-bob:
-- cats: bananas
-  dogs: woof
-- cats: apples
-  dogs: woof2
-- cats: oranges
-  dogs: woof3
-```
-
-then
-
-```text
-yq d sample.yaml bob.[*].cats
-```
-
-will output:
-
-```text
----
-bob:
-- dogs: woof
-- dogs: woof2
-- dogs: woof3
-```
-
-### Multiple Documents - delete from single document[¶](delete.md#multiple-documents-delete-from-single-document) <a id="multiple-documents-delete-from-single-document"></a>
-
-Given a sample.yaml file of:
-
-```text
-something: else
-field: leaveMe
----
-b:
-  c: 2
-field: deleteMe
-```
-
-then
-
-```text
-yq w -d1 sample.yaml field
-```
-
-will output:
-
-```text
-something: else
-field: leaveMe
----
-b:
-  c: 2
-```
-
-### Multiple Documents - delete from all documents[¶](delete.md#multiple-documents-delete-from-all-documents) <a id="multiple-documents-delete-from-all-documents"></a>
-
-Given a sample.yaml file of:
-
-```text
-something: else
-field: deleteMe
----
-b:
-  c: 2
-field: deleteMeToo
-```
-
-then
-
-```text
-yq w -d'*' sample.yaml field
-```
-
-will output:
-
-```text
-something: else
----
-b:
-  c: 2
-```
-
-Note that '\*' is in quotes to avoid being interpreted by your shell.
-
-### Keys with dots[¶](delete.md#keys-with-dots) <a id="keys-with-dots"></a>
-
-When specifying a key that has a dot use key lookup indicator.
-
-```text
-b:
-  foo.bar: 7
-```
-
-```text
-yaml r sample.yaml 'b[foo.bar]'
-```
-
-```text
-yaml w sample.yaml 'b[foo.bar]' 9
-```
-
-Any valid yaml key can be specified as part of a key lookup.
-
-Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
-
-### Keys \(and values\) with leading dashes[¶](delete.md#keys-and-values-with-leading-dashes) <a id="keys-and-values-with-leading-dashes"></a>
-
-If a key or value has leading dashes, yq won't know that you are passing a value as opposed to a flag \(and you will get a 'bad flag syntax' error\).
-
-To fix that, you will need to tell it to stop processing flags by adding '--' after the last flag like so:
-
-```text
-yq n -t -- --key --value
-```
-
-Will result in
-
-``` --key: --value``
-

examples/sample.json

@@ -1 +1 @@
-{"a":"Easy! as one two three","b":{"c":2,"d":[3,4],"e":[{"name":"fred","value":3},{"name":"sam","value":4}]}}
+{"a":"Easy! as one two three","b":{"c":2,"d":[3,4],"e":[{"name":"fred","value":3},{"name":"sam","value":4}]},"ab":"must appear last"}

go.mod

@@ -2,16 +2,16 @@ module github.com/mikefarah/yq/v3
 
 require (
 	github.com/fatih/color v1.9.0
-	github.com/goccy/go-yaml v1.7.5
+	github.com/goccy/go-yaml v1.8.1
 	github.com/kylelemons/godebug v1.1.0
-	github.com/mattn/go-colorable v0.1.6 // indirect
+	github.com/mattn/go-colorable v0.1.7 // indirect
 	github.com/pkg/errors v0.9.1
 	github.com/spf13/cobra v1.0.0
 	github.com/spf13/pflag v1.0.5 // indirect
-	golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980 // indirect
-	golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 // indirect
+	golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
-	gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c
+	gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
 )
 
-go 1.14
+go 1.15

go.sum

@@ -29,8 +29,8 @@ github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V
 github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
 github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
-github.com/goccy/go-yaml v1.7.5 h1:dWvj+p3BG11S/GlUzwzt1WZz0lhBTzTIDtmXT/ZOaPY=
-github.com/goccy/go-yaml v1.7.5/go.mod h1:wS4gNoLalDSJxo/SpngzPQ2BN4uuZVLCmbM4S3vd4+Y=
+github.com/goccy/go-yaml v1.8.1 h1:JuZRFlqLM5cWF6A+waL8AKVuCcqvKOuhJtUQI+L3ez0=
+github.com/goccy/go-yaml v1.8.1/go.mod h1:wS4gNoLalDSJxo/SpngzPQ2BN4uuZVLCmbM4S3vd4+Y=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
 github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
@@ -64,8 +64,8 @@ github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgx
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
 github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
-github.com/mattn/go-colorable v0.1.6 h1:6Su7aK7lXmJ/U79bYtBjLNaha4Fs1Rg9plHpcH+vvnE=
-github.com/mattn/go-colorable v0.1.6/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
+github.com/mattn/go-colorable v0.1.7 h1:bQGKb3vps/j0E9GfJQ03JyhRuxsvdAanXlT9BTw3mdw=
+github.com/mattn/go-colorable v0.1.7/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
 github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
 github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
 github.com/mattn/go-isatty v0.0.11 h1:FxPOTFNqGkuDUGi3H/qkUbQO4ZiBa2brKq5r0l8TGeM=
@@ -144,19 +144,20 @@ golang.org/x/sys v0.0.0-20191026070338-33540a1f6037 h1:YyJpGZS1sBuBCzLAR1VEpK193
 golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980 h1:OjiUf46hAmXblsZdnoSXsEUSKU8r1UEzcL5RVZ4gO9Y=
-golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f h1:Fqb3ao1hUmOR3GkUOg/Y+BadLwykBIzs5q8Ez2SbHyc=
+golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
@@ -175,6 +176,6 @@ gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bl
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c h1:grhR+C34yXImVGp7EzNk+DTIk+323eIUWOmEevy6bDo=
-gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ=
+gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

merge.md

@@ -1,245 +0,0 @@
-# Merge
-
- Yaml files can be merged using the 'merge' command. Each additional file merged with the first file will set values for any key not existing already or where the key has no value.
-
-```text
-yq m <yaml_file> <path>...
-```
-
-### To Stdout[¶](merge.md#to-stdout) <a id="to-stdout"></a>
-
-Given a data1.yaml file of:
-
-```text
-a: simple
-b: [1, 2]
-```
-
-and data2.yaml file of:
-
-```text
-a: other
-c:
-  test: 1
-```
-
-then
-
-```text
-yq m data1.yaml data2.yaml
-```
-
-will output:
-
-```text
-a: simple
-b: [1, 2]
-c:
-  test: 1
-```
-
-### Updating files in-place[¶](merge.md#updating-files-in-place) <a id="updating-files-in-place"></a>
-
-Given a data1.yaml file of:
-
-```text
-a: simple
-b: [1, 2]
-```
-
-and data2.yaml file of:
-
-```text
-a: other
-c:
-  test: 1
-```
-
-then
-
-```text
-yq m -i data1.yaml data2.yaml
-```
-
-will update the data1.yaml file so that the value of 'c' is 'test: 1'.
-
-### Overwrite values[¶](merge.md#overwrite-values) <a id="overwrite-values"></a>
-
-Given a data1.yaml file of:
-
-```text
-a: simple
-b: [1, 2]
-```
-
-and data2.yaml file of:
-
-```text
-a: other
-c:
-  test: 1
-```
-
-then
-
-```text
-yq m -x data1.yaml data2.yaml
-```
-
-will output:
-
-```text
-a: other
-b: [1, 2]
-c:
-  test: 1
-```
-
-### Overwrite values with arrays[¶](merge.md#overwrite-values-with-arrays) <a id="overwrite-values-with-arrays"></a>
-
-Given a data1.yaml file of:
-
-```text
-a: simple
-b: [1, 2]
-```
-
-and data3.yaml file of:
-
-```text
-b: [3, 4]
-c:
-  test: 2
-  other: true
-d: false
-```
-
-then
-
-```text
-yq m -x data1.yaml data3.yaml
-```
-
-will output:
-
-```text
-a: simple
-b: [3, 4]
-c:
-  test: 2
-  other: true
-d: false
-```
-
-Notice that 'b' does not result in the merging of the values within an array.
-
-### Append values with arrays[¶](merge.md#append-values-with-arrays) <a id="append-values-with-arrays"></a>
-
-Given a data1.yaml file of:
-
-```text
-a: simple
-b: [1, 2]
-d: hi
-```
-
-and data3.yaml file of:
-
-```text
-a: something
-b: [3, 4]
-c:
-  test: 2
-  other: true
-```
-
-then
-
-```text
-yq m -a data1.yaml data3.yaml
-```
-
-will output:
-
-```text
-a: simple
-b: [1, 2, 3, 4]
-c:
-  test: 2
-  other: true
-d: hi
-```
-
-Note that the 'b' array has concatenated the values from the second data file. Also note that other map keys are not overridden \(field a\).
-
-Append cannot be used with overwrite, if both flags are given then append is ignored.
-
-### Multiple Documents - merge into single document[¶](merge.md#multiple-documents-merge-into-single-document) <a id="multiple-documents-merge-into-single-document"></a>
-
-Currently yq only has multi-document support for the _first_ document being merged into. The remaining yaml files will have their first document selected.
-
-Given a data1.yaml file of:
-
-```text
-something: else
----
-a: simple
-b: cat
-```
-
-and data3.yaml file of:
-
-```text
-b: dog
-```
-
-then
-
-```text
-yq m -x -d1 data1.yaml data3.yaml
-```
-
-will output:
-
-```text
-something: else
----
-a: simple
-b: dog
-```
-
-### Multiple Documents - merge into all documents[¶](merge.md#multiple-documents-merge-into-all-documents) <a id="multiple-documents-merge-into-all-documents"></a>
-
-Currently yq only has multi-document support for the _first_ document being merged into. The remaining yaml files will have their first document selected.
-
-Given a data1.yaml file of:
-
-```text
-something: else
----
-a: simple
-b: cat
-```
-
-and data3.yaml file of:
-
-```text
-b: dog
-```
-
-then
-
-```text
-yq m -x -d'*' data1.yaml data3.yaml
-```
-
-will output:
-
-```text
-b: dog
-something: else
----
-a: simple
-b: dog
-```
-

pkg/yqlib/data_navigator.go

@@ -22,13 +22,8 @@ func NewDataNavigator(NavigationStrategy NavigationStrategy) DataNavigator {
 }
 
 func (n *navigator) Traverse(value *yaml.Node, path []interface{}) error {
-	realValue := value
 	emptyArray := make([]interface{}, 0)
 	log.Debugf("Traversing path %v", pathStackToString(path))
-	if realValue.Kind == yaml.DocumentNode {
-		log.Debugf("its a document! returning the first child")
-		return n.doTraverse(value.Content[0], "", path, emptyArray)
-	}
 	return n.doTraverse(value, "", path, emptyArray)
 }
 
@@ -39,7 +34,8 @@ func (n *navigator) doTraverse(value *yaml.Node, head interface{}, tail []interf
 	var nodeContext = NewNodeContext(value, head, tail, pathStack)
 
 	var errorDeepSplatting error
-	if head == "**" && value.Kind != yaml.ScalarNode && n.navigationStrategy.ShouldDeeplyTraverse(nodeContext) {
+	// no need to deeply traverse the DocumentNode, as it's already covered by its first child.
+	if head == "**" && value.Kind != yaml.DocumentNode && value.Kind != yaml.ScalarNode && n.navigationStrategy.ShouldDeeplyTraverse(nodeContext) {
 		if len(pathStack) == 0 || pathStack[len(pathStack)-1] != "<<" {
 			errorDeepSplatting = n.recurse(value, head, tail, pathStack)
 		}
@@ -51,7 +47,11 @@ func (n *navigator) doTraverse(value *yaml.Node, head interface{}, tail []interf
 		return errorDeepSplatting
 	}
 
-	if len(tail) > 0 && value.Kind != yaml.ScalarNode {
+	if value.Kind == yaml.DocumentNode {
+		log.Debugf("its a document, diving into %v", head)
+		DebugNode(value)
+		return n.recurse(value, head, tail, pathStack)
+	} else if len(tail) > 0 && value.Kind != yaml.ScalarNode {
 		log.Debugf("diving into %v", tail[0])
 		DebugNode(value)
 		return n.recurse(value, tail[0], tail[1:], pathStack)
@@ -73,6 +73,7 @@ func (n *navigator) recurse(value *yaml.Node, head interface{}, tail []interface
 	nodeContext := NewNodeContext(value, head, tail, pathStack)
 
 	if head == "**" && !n.navigationStrategy.ShouldOnlyDeeplyVisitLeaves(nodeContext) {
+		nodeContext.IsMiddleNode = true
 		errorVisitingDeeply := n.navigationStrategy.Visit(nodeContext)
 		if errorVisitingDeeply != nil {
 			return errorVisitingDeeply
@@ -108,6 +109,8 @@ func (n *navigator) recurse(value *yaml.Node, head interface{}, tail []interface
 			return n.recurse(value.Alias, head, tail, pathStack)
 		}
 		return nil
+	case yaml.DocumentNode:
+		return n.doTraverse(value.Content[0], head, tail, pathStack)
 	default:
 		return n.navigationStrategy.Visit(nodeContext)
 	}

pkg/yqlib/encoder.go

@@ -3,6 +3,7 @@ package yqlib
 import (
 	"bytes"
 	"encoding/json"
+	"fmt"
 	"io"
 
 	yaml "gopkg.in/yaml.v3"
@@ -58,6 +59,21 @@ type jsonEncoder struct {
 	encoder *json.Encoder
 }
 
+func mapKeysToStrings(node *yaml.Node) {
+
+	if node.Kind == yaml.MappingNode {
+		for index, child := range node.Content {
+			if index%2 == 0 { // its a map key
+				child.Tag = "!!str"
+			}
+		}
+	}
+
+	for _, child := range node.Content {
+		mapKeysToStrings(child)
+	}
+}
+
 func NewJsonEncoder(destination io.Writer, prettyPrint bool, indent int) Encoder {
 	var encoder = json.NewEncoder(destination)
 	var indentString = ""
@@ -72,10 +88,162 @@ func NewJsonEncoder(destination io.Writer, prettyPrint bool, indent int) Encoder
 }
 
 func (je *jsonEncoder) Encode(node *yaml.Node) error {
-	var dataBucket interface{}
+	var dataBucket orderedMap
+	// firstly, convert all map keys to strings
+	mapKeysToStrings(node)
 	errorDecoding := node.Decode(&dataBucket)
 	if errorDecoding != nil {
 		return errorDecoding
 	}
 	return je.encoder.Encode(dataBucket)
 }
+
+// orderedMap allows to marshal and unmarshal JSON and YAML values keeping the
+// order of keys and values in a map or an object.
+type orderedMap struct {
+	// if this is an object, kv != nil. If this is not an object, kv == nil.
+	kv     []orderedMapKV
+	altVal interface{}
+}
+
+type orderedMapKV struct {
+	K string
+	V orderedMap
+}
+
+func (o *orderedMap) UnmarshalJSON(data []byte) error {
+	switch data[0] {
+	case '{':
+		// initialise so that even if the object is empty it is not nil
+		o.kv = []orderedMapKV{}
+
+		// create decoder
+		dec := json.NewDecoder(bytes.NewReader(data))
+		_, err := dec.Token() // open object
+		if err != nil {
+			return err
+		}
+
+		// cycle through k/v
+		var tok json.Token
+		for tok, err = dec.Token(); err != io.EOF; tok, err = dec.Token() {
+			// we can expect two types: string or Delim. Delim automatically means
+			// that it is the closing bracket of the object, whereas string means
+			// that there is another key.
+			if _, ok := tok.(json.Delim); ok {
+				break
+			}
+			kv := orderedMapKV{
+				K: tok.(string),
+			}
+			if err := dec.Decode(&kv.V); err != nil {
+				return err
+			}
+			o.kv = append(o.kv, kv)
+		}
+		// unexpected error
+		if err != nil && err != io.EOF {
+			return err
+		}
+		return nil
+	case '[':
+		var arr []orderedMap
+		return json.Unmarshal(data, &arr)
+	}
+
+	return json.Unmarshal(data, &o.altVal)
+}
+
+func (o orderedMap) MarshalJSON() ([]byte, error) {
+	if o.kv == nil {
+		return json.Marshal(o.altVal)
+	}
+	buf := new(bytes.Buffer)
+	enc := json.NewEncoder(buf)
+	buf.WriteByte('{')
+	for idx, el := range o.kv {
+		if err := enc.Encode(el.K); err != nil {
+			return nil, err
+		}
+		buf.WriteByte(':')
+		if err := enc.Encode(el.V); err != nil {
+			return nil, err
+		}
+		if idx != len(o.kv)-1 {
+			buf.WriteByte(',')
+		}
+	}
+	buf.WriteByte('}')
+	return buf.Bytes(), nil
+}
+
+func (o *orderedMap) UnmarshalYAML(node *yaml.Node) error {
+	switch node.Kind {
+	case yaml.DocumentNode:
+		if len(node.Content) == 0 {
+			return nil
+		}
+		return o.UnmarshalYAML(node.Content[0])
+	case yaml.AliasNode:
+		return o.UnmarshalYAML(node.Alias)
+	case yaml.ScalarNode:
+		return node.Decode(&o.altVal)
+	case yaml.MappingNode:
+		// set kv to non-nil
+		o.kv = []orderedMapKV{}
+		for i := 0; i < len(node.Content); i += 2 {
+			var key string
+			var val orderedMap
+			if err := node.Content[i].Decode(&key); err != nil {
+				return err
+			}
+			if err := node.Content[i+1].Decode(&val); err != nil {
+				return err
+			}
+			o.kv = append(o.kv, orderedMapKV{
+				K: key,
+				V: val,
+			})
+		}
+		return nil
+	case yaml.SequenceNode:
+		var res []orderedMap
+		if err := node.Decode(&res); err != nil {
+			return err
+		}
+		o.altVal = res
+		o.kv = nil
+		return nil
+	case 0:
+		// null
+		o.kv = nil
+		o.altVal = nil
+		return nil
+	default:
+		return fmt.Errorf("orderedMap: invalid yaml node")
+	}
+}
+
+func (o *orderedMap) MarshalYAML() (interface{}, error) {
+	// fast path: kv is nil, use altVal
+	if o.kv == nil {
+		return o.altVal, nil
+	}
+	content := make([]*yaml.Node, 0, len(o.kv)*2)
+	for _, val := range o.kv {
+		n := new(yaml.Node)
+		if err := n.Encode(val.V); err != nil {
+			return nil, err
+		}
+		content = append(content, &yaml.Node{
+			Kind:  yaml.ScalarNode,
+			Tag:   "!!str",
+			Value: val.K,
+		}, n)
+	}
+	return &yaml.Node{
+		Kind:    yaml.MappingNode,
+		Tag:     "!!map",
+		Content: content,
+	}, nil
+}

pkg/yqlib/lib.go

@@ -13,11 +13,13 @@ import (
 var log = logging.MustGetLogger("yq")
 
 type UpdateCommand struct {
-	Command             string
-	Path                string
-	Value               *yaml.Node
-	Overwrite           bool
-	DontUpdateNodeValue bool
+	Command               string
+	Path                  string
+	Value                 *yaml.Node
+	Overwrite             bool
+	DontUpdateNodeValue   bool
+	DontUpdateNodeContent bool
+	CommentsMergeStrategy CommentsMergeStrategy
 }
 
 func KindString(kind yaml.Kind) string {
@@ -49,20 +51,23 @@ func DebugNode(value *yaml.Node) {
 		}
 		encoder.Close()
 		log.Debug("Tag: %v, Kind: %v, Anchor: %v", value.Tag, KindString(value.Kind), value.Anchor)
-		log.Debug("%v", buf.String())
+		log.Debug("Head Comment: %v", value.HeadComment)
+		log.Debug("Line Comment: %v", value.LineComment)
+		log.Debug("FootComment Comment: %v", value.FootComment)
+		log.Debug("\n%v", buf.String())
 	}
 }
 
 func pathStackToString(pathStack []interface{}) string {
-	return mergePathStackToString(pathStack, false)
+	return mergePathStackToString(pathStack, UpdateArrayMergeStrategy)
 }
 
-func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
+func mergePathStackToString(pathStack []interface{}, arrayMergeStrategy ArrayMergeStrategy) string {
 	var sb strings.Builder
 	for index, path := range pathStack {
 		switch path.(type) {
 		case int, int64:
-			if appendArrays {
+			if arrayMergeStrategy == AppendArrayMergeStrategy {
 				sb.WriteString("[+]")
 			} else {
 				sb.WriteString(fmt.Sprintf("[%v]", path))
@@ -93,9 +98,7 @@ func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
 			sb.WriteString(".")
 		}
 	}
-	var pathString = sb.String()
-	log.Debug("got a path string: %v", pathString)
-	return pathString
+	return sb.String()
 }
 
 func guessKind(head interface{}, tail []interface{}, guess yaml.Kind) yaml.Kind {
@@ -133,12 +136,13 @@ func guessKind(head interface{}, tail []interface{}, guess yaml.Kind) yaml.Kind
 }
 
 type YqLib interface {
-	Get(rootNode *yaml.Node, path string, deeplyTraverseArrays bool) ([]*NodeContext, error)
+	Get(rootNode *yaml.Node, path string) ([]*NodeContext, error)
+	GetForMerge(rootNode *yaml.Node, path string, arrayMergeStrategy ArrayMergeStrategy) ([]*NodeContext, error)
 	Update(rootNode *yaml.Node, updateCommand UpdateCommand, autoCreate bool) error
 	New(path string) yaml.Node
 
 	PathStackToString(pathStack []interface{}) string
-	MergePathStackToString(pathStack []interface{}, appendArrays bool) string
+	MergePathStackToString(pathStack []interface{}, arrayMergeStrategy ArrayMergeStrategy) string
 }
 
 type lib struct {
@@ -151,21 +155,28 @@ func NewYqLib() YqLib {
 	}
 }
 
-func (l *lib) Get(rootNode *yaml.Node, path string, deeplyTraverseArrays bool) ([]*NodeContext, error) {
+func (l *lib) Get(rootNode *yaml.Node, path string) ([]*NodeContext, error) {
 	var paths = l.parser.ParsePath(path)
-	navigationStrategy := ReadNavigationStrategy(deeplyTraverseArrays)
+	navigationStrategy := ReadNavigationStrategy()
 	navigator := NewDataNavigator(navigationStrategy)
 	error := navigator.Traverse(rootNode, paths)
 	return navigationStrategy.GetVisitedNodes(), error
+}
 
+func (l *lib) GetForMerge(rootNode *yaml.Node, path string, arrayMergeStrategy ArrayMergeStrategy) ([]*NodeContext, error) {
+	var paths = l.parser.ParsePath(path)
+	navigationStrategy := ReadForMergeNavigationStrategy(arrayMergeStrategy)
+	navigator := NewDataNavigator(navigationStrategy)
+	error := navigator.Traverse(rootNode, paths)
+	return navigationStrategy.GetVisitedNodes(), error
 }
 
 func (l *lib) PathStackToString(pathStack []interface{}) string {
 	return pathStackToString(pathStack)
 }
 
-func (l *lib) MergePathStackToString(pathStack []interface{}, appendArrays bool) string {
-	return mergePathStackToString(pathStack, appendArrays)
+func (l *lib) MergePathStackToString(pathStack []interface{}, arrayMergeStrategy ArrayMergeStrategy) string {
+	return mergePathStackToString(pathStack, arrayMergeStrategy)
 }
 
 func (l *lib) New(path string) yaml.Node {
@@ -181,6 +192,10 @@ func (l *lib) Update(rootNode *yaml.Node, updateCommand UpdateCommand, autoCreat
 		var paths = l.parser.ParsePath(updateCommand.Path)
 		navigator := NewDataNavigator(UpdateNavigationStrategy(updateCommand, autoCreate))
 		return navigator.Traverse(rootNode, paths)
+	case "merge":
+		var paths = l.parser.ParsePath(updateCommand.Path)
+		navigator := NewDataNavigator(MergeNavigationStrategy(updateCommand, autoCreate))
+		return navigator.Traverse(rootNode, paths)
 	case "delete":
 		var paths = l.parser.ParsePath(updateCommand.Path)
 		lastBit, newTail := paths[len(paths)-1], paths[:len(paths)-1]

pkg/yqlib/lib_test.go

@@ -30,7 +30,7 @@ func TestLib(t *testing.T) {
 		array[0] = "a"
 		array[1] = 0
 		array[2] = "b"
-		got := subject.MergePathStackToString(array, true)
+		got := subject.MergePathStackToString(array, AppendArrayMergeStrategy)
 		test.AssertResult(t, `a.[+].b`, got)
 	})
 

pkg/yqlib/merge_navigation_strategy.go

@@ -0,0 +1,103 @@
+package yqlib
+
+import "gopkg.in/yaml.v3"
+
+type ArrayMergeStrategy uint32
+
+const (
+	UpdateArrayMergeStrategy ArrayMergeStrategy = 1 << iota
+	OverwriteArrayMergeStrategy
+	AppendArrayMergeStrategy
+)
+
+type CommentsMergeStrategy uint32
+
+const (
+	SetWhenBlankCommentsMergeStrategy CommentsMergeStrategy = 1 << iota
+	IgnoreCommentsMergeStrategy
+	OverwriteCommentsMergeStrategy
+	AppendCommentsMergeStrategy
+)
+
+func MergeNavigationStrategy(updateCommand UpdateCommand, autoCreate bool) NavigationStrategy {
+	return &NavigationStrategyImpl{
+		visitedNodes: []*NodeContext{},
+		pathParser:   NewPathParser(),
+		followAlias: func(nodeContext NodeContext) bool {
+			return false
+		},
+		autoCreateMap: func(nodeContext NodeContext) bool {
+			return autoCreate
+		},
+		visit: func(nodeContext NodeContext) error {
+			node := nodeContext.Node
+			changesToApply := updateCommand.Value
+
+			if node.Kind == yaml.DocumentNode && changesToApply.Kind != yaml.DocumentNode {
+				// when the path is empty, it matches both the top level pseudo document node
+				// and the actual top level node (e.g. map/sequence/whatever)
+				// so when we are updating with no path, make sure we update the right node.
+				node = node.Content[0]
+			}
+
+			log.Debug("going to update")
+			DebugNode(node)
+			log.Debug("with")
+			DebugNode(changesToApply)
+
+			if updateCommand.Overwrite || node.Value == "" {
+				node.Value = changesToApply.Value
+				node.Tag = changesToApply.Tag
+				node.Kind = changesToApply.Kind
+				node.Style = changesToApply.Style
+				node.Anchor = changesToApply.Anchor
+				node.Alias = changesToApply.Alias
+
+				if !updateCommand.DontUpdateNodeContent {
+					node.Content = changesToApply.Content
+				}
+			} else {
+				log.Debug("skipping update as node already has value %v and overwriteFlag is ", node.Value, updateCommand.Overwrite)
+			}
+
+			switch updateCommand.CommentsMergeStrategy {
+			case OverwriteCommentsMergeStrategy:
+				node.HeadComment = changesToApply.HeadComment
+				node.LineComment = changesToApply.LineComment
+				node.FootComment = changesToApply.FootComment
+			case SetWhenBlankCommentsMergeStrategy:
+				if node.HeadComment == "" {
+					node.HeadComment = changesToApply.HeadComment
+				}
+				if node.LineComment == "" {
+					node.LineComment = changesToApply.LineComment
+				}
+				if node.FootComment == "" {
+					node.FootComment = changesToApply.FootComment
+				}
+			case AppendCommentsMergeStrategy:
+				if node.HeadComment == "" {
+					node.HeadComment = changesToApply.HeadComment
+				} else {
+					node.HeadComment = node.HeadComment + "\n" + changesToApply.HeadComment
+				}
+				if node.LineComment == "" {
+					node.LineComment = changesToApply.LineComment
+				} else {
+					node.LineComment = node.LineComment + " " + changesToApply.LineComment
+				}
+				if node.FootComment == "" {
+					node.FootComment = changesToApply.FootComment
+				} else {
+					node.FootComment = node.FootComment + "\n" + changesToApply.FootComment
+				}
+			default:
+			}
+
+			log.Debug("result")
+			DebugNode(node)
+
+			return nil
+		},
+	}
+}

pkg/yqlib/navigation_strategy.go

@@ -11,6 +11,9 @@ type NodeContext struct {
 	Head      interface{}
 	Tail      []interface{}
 	PathStack []interface{}
+	// middle nodes are nodes that match along the original path, but not a
+	// target match of the path. This is only relevant when ShouldOnlyDeeplyVisitLeaves is false.
+	IsMiddleNode bool
 }
 
 func NewNodeContext(node *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) NodeContext {

pkg/yqlib/read_for_merge_navigation_strategy.go

@@ -0,0 +1,37 @@
+package yqlib
+
+import "gopkg.in/yaml.v3"
+
+func ReadForMergeNavigationStrategy(arrayMergeStrategy ArrayMergeStrategy) NavigationStrategy {
+	return &NavigationStrategyImpl{
+		visitedNodes: []*NodeContext{},
+		pathParser:   NewPathParser(),
+		followAlias: func(nodeContext NodeContext) bool {
+			return false
+		},
+		shouldOnlyDeeplyVisitLeaves: func(nodeContext NodeContext) bool {
+			return false
+		},
+		visit: func(nodeContext NodeContext) error {
+			return nil
+		},
+		shouldDeeplyTraverse: func(nodeContext NodeContext) bool {
+			if nodeContext.Node.Kind == yaml.SequenceNode && arrayMergeStrategy == OverwriteArrayMergeStrategy {
+				nodeContext.IsMiddleNode = false
+				return false
+			}
+
+			var isInArray = false
+			if len(nodeContext.PathStack) > 0 {
+				var lastElement = nodeContext.PathStack[len(nodeContext.PathStack)-1]
+				switch lastElement.(type) {
+				case int:
+					isInArray = true
+				default:
+					isInArray = false
+				}
+			}
+			return arrayMergeStrategy == UpdateArrayMergeStrategy || !isInArray
+		},
+	}
+}

pkg/yqlib/read_navigation_strategy.go

@@ -1,24 +1,11 @@
 package yqlib
 
-func ReadNavigationStrategy(deeplyTraverseArrays bool) NavigationStrategy {
+func ReadNavigationStrategy() NavigationStrategy {
 	return &NavigationStrategyImpl{
 		visitedNodes: []*NodeContext{},
 		pathParser:   NewPathParser(),
 		visit: func(nodeContext NodeContext) error {
 			return nil
 		},
-		shouldDeeplyTraverse: func(nodeContext NodeContext) bool {
-			var isInArray = false
-			if len(nodeContext.PathStack) > 0 {
-				var lastElement = nodeContext.PathStack[len(nodeContext.PathStack)-1]
-				switch lastElement.(type) {
-				case int:
-					isInArray = true
-				default:
-					isInArray = false
-				}
-			}
-			return deeplyTraverseArrays || !isInArray
-		},
 	}
 }

pkg/yqlib/update_navigation_strategy.go

@@ -24,12 +24,16 @@ func UpdateNavigationStrategy(updateCommand UpdateCommand, autoCreate bool) Navi
 				node.Tag = changesToApply.Tag
 				node.Kind = changesToApply.Kind
 				node.Style = changesToApply.Style
-				node.Content = changesToApply.Content
+				if !updateCommand.DontUpdateNodeContent {
+					node.Content = changesToApply.Content
+				}
 				node.Anchor = changesToApply.Anchor
 				node.Alias = changesToApply.Alias
-				node.HeadComment = changesToApply.HeadComment
-				node.LineComment = changesToApply.LineComment
-				node.FootComment = changesToApply.FootComment
+				if updateCommand.CommentsMergeStrategy != IgnoreCommentsMergeStrategy {
+					node.HeadComment = changesToApply.HeadComment
+					node.LineComment = changesToApply.LineComment
+					node.FootComment = changesToApply.FootComment
+				}
 			} else {
 				log.Debug("skipping update as node already has value %v and overwriteFlag is ", node.Value, updateCommand.Overwrite)
 			}

prefix.md

@@ -1,127 +0,0 @@
-# Prefix
-
- Paths can be prefixed using the 'prefix' command. The complete yaml content will be nested inside the new prefix path.
-
-```text
-yq p <yaml_file> <path>
-```
-
-### To Stdout[¶](prefix.md#to-stdout) <a id="to-stdout"></a>
-
-Given a data1.yaml file of:
-
-```text
-a: simple
-b: [1, 2]
-```
-
-then
-
-```text
-yq p data1.yaml c
-```
-
-will output:
-
-```text
-c:
-  a: simple
-  b: [1, 2]
-```
-
-### Arbitrary depth[¶](prefix.md#arbitrary-depth) <a id="arbitrary-depth"></a>
-
-Given a data1.yaml file of:
-
-```text
-a:
-  b: [1, 2]
-```
-
-then
-
-```text
-yq p data1.yaml c.d
-```
-
-will output:
-
-```text
-c:
-  d:
-    a:
-      b: [1, 2]
-```
-
-### Updating files in-place[¶](prefix.md#updating-files-in-place) <a id="updating-files-in-place"></a>
-
-Given a data1.yaml file of:
-
-```text
-a: simple
-b: [1, 2]
-```
-
-then
-
-```text
-yq p -i data1.yaml c
-```
-
-will update the data1.yaml file so that the path 'c' is prefixed to all other paths.
-
-### Multiple Documents - prefix a single document[¶](prefix.md#multiple-documents-prefix-a-single-document) <a id="multiple-documents-prefix-a-single-document"></a>
-
-Given a data1.yaml file of:
-
-```text
-something: else
----
-a: simple
-b: cat
-```
-
-then
-
-```text
-yq p -d1 data1.yaml c
-```
-
-will output:
-
-```text
-something: else
----
-c:
-  a: simple
-  b: cat
-```
-
-### Multiple Documents - prefix all documents[¶](prefix.md#multiple-documents-prefix-all-documents) <a id="multiple-documents-prefix-all-documents"></a>
-
-Given a data1.yaml file of:
-
-```text
-something: else
----
-a: simple
-b: cat
-```
-
-then
-
-```text
-yq p -d'*' data1.yaml c
-```
-
-will output:
-
-```text
-c:
-  something: else
----
-c:
-  a: simple
-  b: cat
-```
-

read.md

@@ -1,228 +0,0 @@
-# Read
-
-```text
-yq r <yaml_file|json_file> <path>
-```
-
-This command can take a json file as input too, and will output yaml unless specified to export as json \(-j\)
-
-### Basic[¶](read.md#basic) <a id="basic"></a>
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-```
-
-then
-
-```text
-yq r sample.yaml b.c
-```
-
-will output the value of '2'.
-
-### From Stdin[¶](read.md#from-stdin) <a id="from-stdin"></a>
-
-Given a sample.yaml file of:
-
-```text
-cat sample.yaml | yq r - b.c
-```
-
-will output the value of '2'.
-
-### Splat[¶](read.md#splat) <a id="splat"></a>
-
-Given a sample.yaml file of:
-
-```text
----
-bob:
-  item1:
-    cats: bananas
-  item2:
-    cats: apples
-  thing:
-    cats: oranges
-```
-
-then
-
-```text
-yq r sample.yaml bob.*.cats
-```
-
-will output
-
-```text
-- bananas
-- apples
-- oranges
-```
-
-### Prefix Splat[¶](read.md#prefix-splat) <a id="prefix-splat"></a>
-
-Given a sample.yaml file of:
-
-```text
----
-bob:
-  item1:
-    cats: bananas
-  item2:
-    cats: apples
-  thing:
-    cats: oranges
-```
-
-then
-
-```text
-yq r sample.yaml bob.item*.cats
-```
-
-will output
-
-```text
-- bananas
-- apples
-```
-
-### Multiple Documents - specify a single document[¶](read.md#multiple-documents-specify-a-single-document) <a id="multiple-documents-specify-a-single-document"></a>
-
-Given a sample.yaml file of:
-
-```text
-something: else
----
-b:
-  c: 2
-```
-
-then
-
-```text
-yq r -d1 sample.yaml b.c
-```
-
-will output the value of '2'.
-
-### Multiple Documents - read all documents[¶](read.md#multiple-documents-read-all-documents) <a id="multiple-documents-read-all-documents"></a>
-
-Reading all documents will return the result as an array. This can be converted to json using the '-j' flag if desired.
-
-Given a sample.yaml file of:
-
-```text
-name: Fred
-age: 22
----
-name: Stella
-age: 23
----
-name: Android
-age: 232
-```
-
-then
-
-```text
-yq r -d'*' sample.yaml name
-```
-
-will output:
-
-```text
-- Fred
-- Stella
-- Android
-```
-
-### Arrays[¶](read.md#arrays) <a id="arrays"></a>
-
-You can give an index to access a specific element: e.g.: given a sample file of
-
-```text
-b:
-  e:
-    - name: fred
-      value: 3
-    - name: sam
-      value: 4
-```
-
-then
-
-```text
-yq r sample.yaml 'b.e[1].name'
-```
-
-will output 'sam'
-
-Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
-
-### Array Splat[¶](read.md#array-splat) <a id="array-splat"></a>
-
-e.g.: given a sample file of
-
-```text
-b:
-  e:
-    - name: fred
-      value: 3
-    - name: sam
-      value: 4
-```
-
-then
-
-```text
-yq r sample.yaml 'b.e[*].name'
-```
-
-will output:
-
-```text
-- fred
-- sam
-```
-
-Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
-
-### Keys with dots[¶](read.md#keys-with-dots) <a id="keys-with-dots"></a>
-
-When specifying a key that has a dot use key lookup indicator.
-
-```text
-b:
-  foo.bar: 7
-```
-
-```text
-yaml r sample.yaml 'b[foo.bar]'
-```
-
-```text
-yaml w sample.yaml 'b[foo.bar]' 9
-```
-
-Any valid yaml key can be specified as part of a key lookup.
-
-Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
-
-### Keys \(and values\) with leading dashes[¶](read.md#keys-and-values-with-leading-dashes) <a id="keys-and-values-with-leading-dashes"></a>
-
-If a key or value has leading dashes, yq won't know that you are passing a value as opposed to a flag \(and you will get a 'bad flag syntax' error\).
-
-To fix that, you will need to tell it to stop processing flags by adding '--' after the last flag like so:
-
-```text
-yq n -t -- --key --value
-```
-
-Will result in
-
-``` --key: --value``
-

release_instructions.txt

@@ -4,6 +4,7 @@
 - tag git with same version number
 - make sure local build passes
 - push tag to git
+  - 3.4.0, v3
   - git push --tags
 - make local xcompile (builds binaries for all platforms)
 

scripts/doctools.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-
-brew install mkdocs libyaml
-pip3 install markdown-include
-pip3 install mkdocs-material
-

scripts/publish.sh

@@ -32,5 +32,5 @@ upload() {
     done < <(find ./build -mindepth 1 -maxdepth 1 -print0)
 }
 
-release
+# release
 upload

scripts/xcompile.sh

@@ -3,10 +3,12 @@
 # This assumes that gonative and gox is installed as per the 'one time setup' instructions
 # at https://github.com/inconshreveable/gonative
 
-gox -ldflags "${LDFLAGS}" -output="build/yq_{{.OS}}_{{.Arch}}"
+
+CGO_ENABLED=0 gox -ldflags "${LDFLAGS}" -output="build/yq_{{.OS}}_{{.Arch}}"
 # include non-default linux builds too
-gox -ldflags "${LDFLAGS}" -os=linux  -output="build/yq_{{.OS}}_{{.Arch}}"
+CGO_ENABLED=0 gox -ldflags "${LDFLAGS}" -os=linux  -output="build/yq_{{.OS}}_{{.Arch}}"
 
 cd build
 rhash -r -a . -P -o checksums
 
+rhash --list-hashes > checksums_hashes_order

snap/snapcraft.yaml

@@ -1,5 +1,5 @@
 name: yq
-version: '3.3.2'
+version: '3.4.1'
 summary: A lightweight and portable command-line YAML processor
 description: |
   The aim of the project is to be the jq or sed of yaml files.
@@ -16,7 +16,7 @@ apps:
 parts:
   yq:
     plugin: go
-    go-channel: 1.14/stable
+    go-channel: 1.15/stable
     source: .
     source-type: git
     go-importpath: github.com/mikefarah/yq

write-update.md

@@ -1,349 +0,0 @@
-# Write
-
-```text
-yq w <yaml_file> <path> <new value>
-```
-
-### To Stdout[¶](write-update.md#to-stdout) <a id="to-stdout"></a>
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-```
-
-then
-
-```text
-yq w sample.yaml b.c cat
-```
-
-will output:
-
-```text
-b:
-  c: cat
-```
-
-### From STDIN[¶](write-update.md#from-stdin) <a id="from-stdin"></a>
-
-```text
-cat sample.yaml | yq w - b.c blah
-```
-
-### Adding new fields[¶](write-update.md#adding-new-fields) <a id="adding-new-fields"></a>
-
-Any missing fields in the path will be created on the fly.
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-```
-
-then
-
-```text
-yq w sample.yaml b.d[+] "new thing"
-```
-
-will output:
-
-```text
-b:
-  c: cat
-  d:
-    - new thing
-```
-
-### Splat[¶](write-update.md#splat) <a id="splat"></a>
-
-Given a sample.yaml file of:
-
-```text
----
-bob:
-  item1:
-    cats: bananas
-  item2:
-    cats: apples
-  thing:
-    cats: oranges
-```
-
-then
-
-```text
-yq w sample.yaml bob.*.cats meow
-```
-
-will output:
-
-```text
----
-bob:
-  item1:
-    cats: meow
-  item2:
-    cats: meow
-  thing:
-    cats: meow
-```
-
-### Prefix Splat[¶](write-update.md#prefix-splat) <a id="prefix-splat"></a>
-
-Given a sample.yaml file of:
-
-```text
----
-bob:
-  item1:
-    cats: bananas
-  item2:
-    cats: apples
-  thing:
-    cats: oranges
-```
-
-then
-
-```text
-yq w sample.yaml bob.item*.cats meow
-```
-
-will output:
-
-```text
----
-bob:
-  item1:
-    cats: meow
-  item2:
-    cats: meow
-  thing:
-    cats: oranges
-```
-
-### Array Splat[¶](write-update.md#array-splat) <a id="array-splat"></a>
-
-Given a sample.yaml file of:
-
-```text
----
-bob:
-- cats: bananas
-- cats: apples
-- cats: oranges
-```
-
-then
-
-```text
-yq w sample.yaml bob[*].cats meow
-```
-
-will output:
-
-```text
----
-bob:
-- cats: meow
-- cats: meow
-- cats: meow
-```
-
-### Appending value to an array field[¶](write-update.md#appending-value-to-an-array-field) <a id="appending-value-to-an-array-field"></a>
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-  d:
-    - new thing
-    - foo thing
-```
-
-then
-
-```text
-yq w sample.yaml "b.d[+]" "bar thing"
-```
-
-will output:
-
-```text
-b:
-  c: cat
-  d:
-    - new thing
-    - foo thing
-    - bar thing
-```
-
-Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
-
-### Multiple Documents - update a single document[¶](write-update.md#multiple-documents-update-a-single-document) <a id="multiple-documents-update-a-single-document"></a>
-
-Given a sample.yaml file of:
-
-```text
-something: else
----
-b:
-  c: 2
-```
-
-then
-
-```text
-yq w -d1 sample.yaml b.c 5
-```
-
-will output:
-
-```text
-something: else
----
-b:
-  c: 5
-```
-
-### Multiple Documents - update all documents[¶](write-update.md#multiple-documents-update-all-documents) <a id="multiple-documents-update-all-documents"></a>
-
-Given a sample.yaml file of:
-
-```text
-something: else
----
-b:
-  c: 2
-```
-
-then
-
-```text
-yq w -d'*' sample.yaml b.c 5
-```
-
-will output:
-
-```text
-something: else
-b:
-  c: 5
----
-b:
-  c: 5
-```
-
-Note that '\*' is in quotes to avoid being interpreted by your shell.
-
-### Updating files in-place[¶](write-update.md#updating-files-in-place) <a id="updating-files-in-place"></a>
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-```
-
-then
-
-```text
-yq w -i sample.yaml b.c cat
-```
-
-will update the sample.yaml file so that the value of 'c' is cat.
-
-### Updating multiple values with a script[¶](write-update.md#updating-multiple-values-with-a-script) <a id="updating-multiple-values-with-a-script"></a>
-
-Given a sample.yaml file of:
-
-```text
-b:
-  c: 2
-  e:
-    - name: Billy Bob
-```
-
-and a script update\_instructions.yaml of:
-
-```text
-b.c: 3
-b.e[+].name: Howdy Partner
-```
-
-then
-
-```text
-yq w -s update_instructions.yaml sample.yaml
-```
-
-will output:
-
-```text
-b:
-  c: 3
-  e:
-    - name: Howdy Partner
-```
-
-And, of course, you can pipe the instructions in using '-':
-
-```text
-cat update_instructions.yaml | yq w -s - sample.yaml
-```
-
-### Values starting with a hyphen \(or dash\)[¶](write-update.md#values-starting-with-a-hyphen-or-dash) <a id="values-starting-with-a-hyphen-or-dash"></a>
-
-The flag terminator needs to be used to stop the app from attempting to parse the subsequent arguments as flags:
-
-```text
-yq w -- my.path -3
-```
-
-will output
-
-```text
-my:
-  path: -3
-```
-
-### Keys with dots[¶](write-update.md#keys-with-dots) <a id="keys-with-dots"></a>
-
-When specifying a key that has a dot use key lookup indicator.
-
-```text
-b:
-  foo.bar: 7
-```
-
-```text
-yaml r sample.yaml 'b[foo.bar]'
-```
-
-```text
-yaml w sample.yaml 'b[foo.bar]' 9
-```
-
-Any valid yaml key can be specified as part of a key lookup.
-
-Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
-
-### Keys \(and values\) with leading dashes[¶](write-update.md#keys-and-values-with-leading-dashes) <a id="keys-and-values-with-leading-dashes"></a>
-
-If a key or value has leading dashes, yq won't know that you are passing a value as opposed to a flag \(and you will get a 'bad flag syntax' error\).
-
-To fix that, you will need to tell it to stop processing flags by adding '--' after the last flag like so:
-
-```text
-yq n -t -- --key --value
-```
-
-Will result in
-
-``` --key: --value``
-